Re: [gentoo-portage-dev] [PATCH 2/2] Add iter_completed convenience function (bug 648790)

[Zac Medico]

On 02/25/2018 07:17 PM, Alec Warner wrote:
>
>
> On Sun, Feb 25, 2018 at 8:50 PM, Zac Medico <zmedico@g.o
> <mailto:zmedico@g.o>> wrote:
>
>     The iter_completed function is similar to asyncio.as_completed, but
>     takes an iterator of futures as input, and includes support for
>     max_jobs and max_load parameters. The default values for max_jobs
>     and max_load correspond to multiprocessing.cpu_count().
>
>     Example usage for async_aux_get:
>
>       import portage
>       from portage.util.futures.iter_completed import iter_completed
>
>       portdb = portage.portdb
>       future_cpv = {}
>
>
> I'm not sure I grasp the purpose of this dict, can't we just modify the
> async aux get to return the cpv from the future?

If we do that then we should probably return all of the other aux_get
inputs too, including mylist, mytree, and myrepo. Pretty soon it feels
like there's a lot of clutter here. If we leave the burden to the
caller, then the API is simpler, and it's not much of a burden to the
caller anyway.

>       def future_generator():
>         for cpv in portdb.cp_list('sys-apps/portage'):
>           future = portdb.async_aux_get(cpv, portage.auxdbkeys)
>           future_cpv[id(future)] = cpv
>           yield future
>
>
> for cpv in portdb.cp_list('...'):
>    yield portdb.async_aux_get(cpv, portage.auxdbkeys)
>  
>
>       for future in iter_completed(future_generator()):
>         cpv = future_cpv.pop(id(future))
>         try:
>           result = future.result()
>         except KeyError as e:
>           # aux_get failed
>           print('error:', cpv, e)
>         else:
>           print(cpv, result)
>
>
> for future in iter_completed(future_generator()):
>   try:
>     cpv, result = future.result() 
>   except KeyError as e:
>     print('error', cpv, e)
>  
>
> Or do we expect callers to need other things to key off of in this API?

Yeah it's complicated because of the number of input arguments to
aux_get. You can have the same cpv existing in multiple repos. It's so
much simpler to let the caller manage the mapping from input arguments
to future instance.
-- 
Thanks,
Zac