| 1 |
Hmm, that's strange... it seems to have made it to the list archives: |
| 2 |
https://archives.gentoo.org/gentoo-portage-dev/message/a4db905a64e3c1f6d88c4876e8291a65 |
| 3 |
|
| 4 |
(but it is entirely possible that I used "git send-email" incorrectly) |
| 5 |
|
| 6 |
On Thu, Jul 9, 2020 at 2:04 PM Alec Warner <antarus@g.o> wrote: |
| 7 |
|
| 8 |
> |
| 9 |
> |
| 10 |
> On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei <cshei@××××××.com> wrote: |
| 11 |
> |
| 12 |
>> Awesome! Here's a patch that adds @lru_cache to use_reduce, vercmp, and |
| 13 |
>> catpkgsplit. use_reduce was split into 2 functions, with the outer one |
| 14 |
>> converting lists/sets to tuples so they can be hashed and creating a |
| 15 |
>> copy of the returned list (since the caller seems to modify it |
| 16 |
>> sometimes). I tried to select cache sizes that minimized memory use |
| 17 |
>> increase, |
| 18 |
>> while still providing about the same speedup compared to a cache with |
| 19 |
>> unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases |
| 20 |
>> from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the |
| 21 |
>> RES column in htop increases from 280 MB -> 290 MB. |
| 22 |
>> |
| 23 |
>> "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while |
| 24 |
>> max observed RES value actually decreases from 228 MB -> 214 MB (similar |
| 25 |
>> values observed across a few before/after runs). |
| 26 |
>> |
| 27 |
>> Here are the cache hit stats, max observed RES memory, and runtime in |
| 28 |
>> seconds for various sizes in the update case. Caching for each |
| 29 |
>> function was tested independently (only 1 function with caching enabled |
| 30 |
>> at a time): |
| 31 |
>> |
| 32 |
>> catpkgsplit: |
| 33 |
>> CacheInfo(hits=1222233, misses=21419, maxsize=None, currsize=21419) |
| 34 |
>> 270 MB |
| 35 |
>> 39.217 |
| 36 |
>> |
| 37 |
>> CacheInfo(hits=1218900, misses=24905, maxsize=10000, currsize=10000) |
| 38 |
>> 271 MB |
| 39 |
>> 39.112 |
| 40 |
>> |
| 41 |
>> CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000) |
| 42 |
>> 271 MB |
| 43 |
>> 39.217 |
| 44 |
>> |
| 45 |
>> CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500) |
| 46 |
>> 269 MB |
| 47 |
>> 39.438 |
| 48 |
>> |
| 49 |
>> CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000) |
| 50 |
>> 271 MB |
| 51 |
>> 39.348 |
| 52 |
>> |
| 53 |
>> CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100) |
| 54 |
>> 271 MB |
| 55 |
>> 39.487 |
| 56 |
>> |
| 57 |
>> |
| 58 |
>> use_reduce: |
| 59 |
>> CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561) |
| 60 |
>> 407 MB |
| 61 |
>> 35.77 |
| 62 |
>> |
| 63 |
>> CacheInfo(hits=45186, misses=18800, maxsize=10000, currsize=10000) |
| 64 |
>> 353 MB |
| 65 |
>> 35.52 |
| 66 |
>> |
| 67 |
>> CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000) |
| 68 |
>> 335 MB |
| 69 |
>> 35.31 |
| 70 |
>> |
| 71 |
>> CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500) |
| 72 |
>> 318 MB |
| 73 |
>> 35.85 |
| 74 |
>> |
| 75 |
>> CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000) |
| 76 |
>> 301 MB |
| 77 |
>> 36.39 |
| 78 |
>> |
| 79 |
>> CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100) |
| 80 |
>> 299 MB |
| 81 |
>> 37.175 |
| 82 |
>> |
| 83 |
>> |
| 84 |
>> I didn't bother collecting detailed stats for vercmp, since the |
| 85 |
>> inputs/outputs are quite small and don't cause much memory increase. |
| 86 |
>> Please let me know if there are any other suggestions/improvements (and |
| 87 |
>> thanks Sid for the lru_cache suggestion!). |
| 88 |
>> |
| 89 |
> |
| 90 |
> I don't see a patch attached; can you link to it? |
| 91 |
> |
| 92 |
> -A |
| 93 |
> |
| 94 |
> |
| 95 |
>> |
| 96 |
>> Thanks, |
| 97 |
>> Chun-Yu |
| 98 |
>> |
| 99 |
>> |
| 100 |
>> |
| 101 |
>> |
Archives