[gentoo-dev] Re: Packages up for grabs

[Duncan]

Tom Wijsman posted on Sun, 16 Jun 2013 23:24:27 +0200 as excerpted:

> On Sun, 16 Jun 2013 19:33:53 +0000 (UTC)
> Duncan <1i5t5.duncan@×××.net> wrote:
>
>> TL;DR: SSDs help. =:^)
>
> TL;DR: SSDs help, but they don't solve the underlying problem. =:-(

Well, there's the long-term fix to the underlying problem, and there's 
coping strategies to help with where things are at now.  I was simply 
saying that an SSD helps a LOT in dealing with the inefficiencies of the 
current code.  See the "quite apart... practical question of ... dealing 
with the problem /now/" bit quoted below.

> I have one; it's great to help make my boot short, but it isn't really a
> great improvement for the Portage tree. Better I/O isn't a solution to
> computational complexity; it doesn't deal with the CPU bottleneck.

But here, agreed with ciaranm, the cpu's not the bottleneck, at least not 
from cold-cache.  It doesn't even up the cpu clocking from minimum as 
it's mostly filesystem access.  Once the cache is warm, then yes, it ups 
the CPU speed and I see the single-core behavior you mention, but cold-
cache, no way; it's I/O bound.

And with an ssd, the portage tree update (the syncs both of gentoo and 
the overlays) went from a /crawling/ console scroll, to scrolling so fast 
I can't read it.

>> Quite apart from the theory and question of making the existing code
>> faster vs. a new from-scratch implementation, there's the practical
>> question of what options one can actually use to deal with the problem
>> /now/.
>
> Don't rush it: Do you know the problem well? Does the solution properly
> deal with it? Is it still usable some months / years from now?

Not necessarily.  But first we must /get/ to some months / years from 
now, and that's a lot easier if the best is made of the current 
situation, while a long term fix is being developed.

>> FWIW, one solution (particularly for folks who don't claim to have
>> reasonable coding skills and thus have limited options in that regard)
>> is to throw hardware at the problem.
>
> Improvements in algorithmic complexity (exponential) are much bigger
> than improvements you can achieve by buying new hardware (linear).

Same song different verse.  Fixing the algorithmic complexity is fine and 
certainly a good idea longer term, but it's not something I can use at my 
next update.  Throwing hardware at the problem is usable now.

>> ---
>> [1] I'm running ntp and the initial ntp-client connection and time sync
>> takes ~12 seconds a lot of the time, just over the initial 10 seconds
>> down, 50 to go, trigger on openrc's 1-minute timeout.
>
> Why do you make your boot wait for NTP to sync its time?

Well, ntpd is waiting for the initial step so it doesn't have to slew so 
hard for so long if the clock's multiple seconds off.

And ntpd is in my default runlevel, with a few local service tasks that 
are after * and need a good clock time anyway, so...

> How could hardware make this time sync go any faster?

Which is what I said, that as a practical matter, my boot didn't speed up 
much /because/ I'm running (and waiting for) the ntp-client time-
stepper.  Thus, I'd not /expect/ a hardware update (unless it's to a more 
direct net connection) to help much.

>> [2] ... SNIP ... runs ~1 hour ... SNIP ...
>
> Sounds great, but the same thing could run in much less time. I have
> worse hardware, and it doesn't take much longer than yours do; so, I
> don't really see the benefits new hardware bring to the table. And that
> HDD to SSD change, that's really a once in a lifetime flood.

I expect I'm more particular than most about checking changelogs.  I 
certainly don't read them all, but if there's a revision-bump for 
instance, I like to see what the gentoo devs considered important enough 
to do a revision bump.  And I religiously check portage logs, selecting 
mentioned bug numbers probably about half the time, which pops up a menu 
with a gentoo bug search on the number, from which I check the bug 
details and sometimes the actual git commit code.  For all my overlays I 
check the git whatchanged logs, and I have a helper script that lets me 
fetch and then check git whatchanged for a number of my live packages, 
including openrc (where I switched to live-git precisely /because/ I was 
following it closely enough to find the git whatchanged logs useful, both 
for general information and for troubleshooting when something went wrong 
-- release versions simply didn't have enough resolution, too many things 
changing in each openrc release to easily track down problems and file 
bugs as appropriate), as well.

And you're probably not rebuilding well over a hundred live-packages 
(thank $DEITY and the devs in question for ccache!) at every update, in 
addition to the usual (deep) @world version-bump and newuse updates, are 
you?

Of course maybe you are, but I did specify that, and I didn't see 
anything in your comments indicating anything like an apples to apples 
comparision.

>> [3] Also relevant, 16 gigs RAM, PORTAGETMPDIR on tmpfs.
>
> Sounds all cool, but think about your CPU again; saturate it...
>
> Building the Linux kernel with `make -j32 -l8` versus `make -j8` is a
> huge difference; most people follow the latter instructions, without
> really thinking through what actually happens with the underlying data.
> The former queues up jobs for your processor; so the moment a job is
> done a new job will be ready, so, you don't need to wait on the disk.

Truth is, I used to run a plain make -j (no number and no -l at all) on 
my kernel builds, just to watch the system stress and then so elegantly 
recover.  It's an amazing thing to watch, this Linux kernel thing and how 
it deals with cpu oversaturation.  =:^)

But I suppose I've gotten more conservative in my old age. =:^P  
Needlessly oversaturating the CPU (and RAM) only slows things down and 
forces cache dump and swappage.  These days according to my kernel-build-
script configuration I only run -j24, which seems a reasonable balance as 
it keeps the CPUs busy but stays safely enough within a few gigs of RAM 
so I don't dump-cache or hit swap.  Timing a kernel build from make clean 
suggests it's the same sub-seconds range from -j10 or so, up to (from 
memory) -j50 or so, after which build time starts to go up, not down.

> Something completely different; look at the history of data mining,
> today's algorithms are much much faster than those of years ago.
>
> Just to point out that different implementations and configurations have
> much more power in cutting time than the typical hardware change does.

I agree and am not arguing that.  All I'm saying is that there are 
measures that a sysadmin can take today to at least help work around the 
problem, today, while all those faster algorithms are being developed, 
implemented, tested and deployed. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman