| 1 |
On 11/17/2014 07:07 PM, Andrew Savchenko wrote: |
| 2 |
> Hello, |
| 3 |
> |
| 4 |
> On Sun, 09 Nov 2014 00:15:56 -0800 Zac Medico wrote: |
| 5 |
>> On 11/08/2014 10:53 PM, Andrew Savchenko wrote: |
| 6 |
> [...] |
| 7 |
>>> On old hardware it may take dozens of minutes of CPU time. I have |
| 8 |
>>> that *.pickle files, I have sqlite metadata cache, I have 100% CPU |
| 9 |
>>> usage. It's not an I/O problem. Just take into account that due to |
| 10 |
>>> instruction set, Lx cache, frequency and memory speed difference |
| 11 |
>>> old CPU-based system may be 20x times slower than recent one. |
| 12 |
>> |
| 13 |
>> It could be useful for us to collect profiling data generated on old |
| 14 |
>> hardware. If you'd like to help, you can use python's cProfile module to |
| 15 |
>> generate statistics for us to analyze. The statistics can be nicely |
| 16 |
>> visualized as a shaded call graph by using gprof2dot and graphviz [1]. |
| 17 |
> |
| 18 |
> Sorry for delay, I generated samples on two old hosts. |
| 19 |
> |
| 20 |
> Tarball contains per host directories. Each one contains: |
| 21 |
> - pstats file; |
| 22 |
> - generated pdf with call graphs and timing; |
| 23 |
> - host-related information: |
| 24 |
> * emerge --info |
| 25 |
> * /proc/cpuinfo |
| 26 |
> * /proc/memnifo |
| 27 |
> |
| 28 |
> *.pstats and *.pdf filename should describe command unambiguously, |
| 29 |
> e.g. emerge-pv_python:2.7_python:3.3-python-3.3.5-r1 means: |
| 30 |
> emerge -pv python:2.7 python:3.3 while using python-3.3.5-r1 as |
| 31 |
> interpreter. |
| 32 |
|
| 33 |
Thank you for this data. I will see what I can to do optimize the |
| 34 |
problem areas that your data highlights. |
| 35 |
|
| 36 |
> hitomi system was not fully updated for a bit more than a year, |
| 37 |
> while another one for about half a year. So dependency calculations |
| 38 |
> may be of different intencities. Sets of packages installed are |
| 39 |
> similar but not the same: |
| 40 |
> 2502 on hitomi |
| 41 |
|
| 42 |
For hitomi, _slot_operator_update_probe/use_reduce is an obvious thing |
| 43 |
to optimize. It called use_reduce 53763 times there, so it seems to |
| 44 |
repeat use_reduce multiple times for the same packages. That means we |
| 45 |
should see a benefit from memoization. |
| 46 |
|
| 47 |
> 2953 on desktop |
| 48 |
|
| 49 |
For desktop, _dynamic_deps_preload is an obvious thing to optimize. You |
| 50 |
can avoid this function entirely if you use --dynamic-deps=n. You may |
| 51 |
need to run 'emerge @changed-deps' in order to ensure that emerge will |
| 52 |
be able to cope with --dynamic-deps=n. IIRC you need at least |
| 53 |
sys-apps/portage-2.2.14 for @changed-deps support. |
| 54 |
|
| 55 |
-- |
| 56 |
Thanks, |
| 57 |
Zac |
Archives