| 1 |
On 01/17/2015 04:46 PM, Patrick Lauer wrote: |
| 2 |
> On Saturday 17 January 2015 13:12:56 Zac Medico wrote: |
| 3 |
>> On 01/17/2015 03:35 AM, Patrick Lauer wrote: |
| 4 |
>>> * Portage is too slow |
| 5 |
>>> |
| 6 |
>>> On 'small' hardware emerge -upNDv @world can take enough time |
| 7 |
>>> to make updates prohibitive - on an 800Mhz machine it took me |
| 8 |
>>> about 3 days to figure out a solution to some silly blockers due to |
| 9 |
>>> the |
| 10 |
>>> very slow change test cycle |
| 11 |
>>> Fix: Do some profiling and un-refactoring to remove useless code |
| 12 |
>>> Fix: Set up a reference system (CI) to catch future regressions |
| 13 |
>> |
| 14 |
>> I'm certainly in favor of improving portage performance. However, for |
| 15 |
>> "small" hardware (which includes many ARM and MIPS systems), we really |
| 16 |
>> need to focus on binary package support. Note that dependency resolution |
| 17 |
>> for installing binary packages tends to be much simpler than for |
| 18 |
>> building ebuilds from source, and this translates into much shorter |
| 19 |
>> dependency resolution time. |
| 20 |
>> |
| 21 |
> That's an orthogonal problem. And that's coming from someone who extensively |
| 22 |
> uses binpkgs already ... |
| 23 |
|
| 24 |
Sure, but building from source on "small" hardware is not necessarily |
| 25 |
the best practice. If our binary package support was better, then people |
| 26 |
might be more likely to use "big" hardware to build packages for |
| 27 |
distribution to "small" hardware. |
| 28 |
|
| 29 |
> The problem with (dependency resolution) performance is that in some |
| 30 |
> scenarios, even on fast machines, it takes "too long" - especially if there's |
| 31 |
> some silly useflag mismatch and two packages that have ~arch keywords, and now |
| 32 |
> you need 12 attempts to figure out a solution that works for you ... |
| 33 |
> At 30 seconds for each resolution cycle that's 6 minutes waiting for portage. |
| 34 |
> |
| 35 |
> If it were faster it'd almost be interactive ... |
| 36 |
|
| 37 |
Yeah, that would be great. On a related note, I think that the default |
| 38 |
emerge --backtrack=10 setting is too high, causing emerge to waste lots |
| 39 |
of cpu time before it ultimately fails to find a valid solution. I've |
| 40 |
filed a bug to make it default to --backtrack=3 [1]. |
| 41 |
|
| 42 |
> Also - just for comparison: |
| 43 |
> On my currently fastest box "emerge -ep @world" takes about 3 seconds since |
| 44 |
> there's almost no packages installed. |
| 45 |
> On my currently slowest box same takes about 15 minutes, because (1) lots of |
| 46 |
> packages installed and (2) slow CPU and (3) brutally slow IO |
| 47 |
> |
| 48 |
> Binpkgs only help so much - if any dependencies change I need to sync the |
| 49 |
> configuration between client and server, and to see if it resolves I need to do |
| 50 |
> a dryrun on the client (or be very certain that the configuration really |
| 51 |
> matches) - or risk that it's going to fail because of mismatches. |
| 52 |
|
| 53 |
For some, maybe a nice way to sync configuration would be to create a |
| 54 |
customized profile. Then, emerge --sync would pull in your configuration |
| 55 |
updates from the server. |
| 56 |
|
| 57 |
With "profile-formats = portage-2" in metadata/layout.conf of your |
| 58 |
profiles repository, you can put things like |
| 59 |
"gentoo:default/linux/amd64/13.0/desktop" in the parent file of your |
| 60 |
profile, so it inherits from a profile in the "gentoo" repository. |
| 61 |
|
| 62 |
> I haven't quite figured out why portage needs such humongous amounts of memory |
| 63 |
> (>300MB ?!) |
| 64 |
|
| 65 |
Yes, I would like to work on reducing the memory consumption. |
| 66 |
|
| 67 |
> and why it needs to spend a minute to figure out how to install a |
| 68 |
> simple almost-no-dependencies tool like htop or iotop. |
| 69 |
|
| 70 |
Note that the emerge --complete-graph-if-new-ver and |
| 71 |
--complete-graph-if-new-use options are enabled by default. These |
| 72 |
options will force emerge to create a complete dependency graph, in |
| 73 |
order to ensure that all reverse-dependencies are respected. |
| 74 |
|
| 75 |
> There's some obvious |
| 76 |
> badness, and just saying "well let's use binpkgs" won't fix it (and, well, on |
| 77 |
> the binpkg server if you have 3k packages installed you get the same |
| 78 |
> performance issues, so ignoring the issue is kinda not a good idea) |
| 79 |
|
| 80 |
Of course not. |
| 81 |
|
| 82 |
> There's been some good progress, I remember you reducing memory usage already |
| 83 |
> (>500MB -> 350MB or so for merging kernel sources) and there's some speedups |
| 84 |
> from the last round of performance work. Still I think we can do a lot better |
| 85 |
> :) |
| 86 |
|
| 87 |
Sure we can. |
| 88 |
|
| 89 |
> Thanks for your efforts, |
| 90 |
> |
| 91 |
> Patrick |
| 92 |
|
| 93 |
[1] https://bugs.gentoo.org/show_bug.cgi?id=536926 |
| 94 |
-- |
| 95 |
Thanks, |
| 96 |
Zac |
Archives