Re: [gentoo-project] For next council meeting: moving Portage to python3.6+

[Michał Górny]

W dniu śro, 18.04.2018 o godzinie 08∶58 -0700, użytkownik Zac Medico
napisał:
> On 02/23/2018 06:58 PM, Daniel Robbins wrote:
> > Hi there,
> >
> > Zac Medico and I would like to discuss the following at the next council
> > meeting: moving Portage to support only python3.6+. One of the primary
> > benefits of this change is that it will allow us to use python's native
> > async implementation as we look to make Portage faster at a critical
> > time when we are looking to improve Portage performance. As we approach
> > the stabilization of python3.6 in Gentoo, it seems like a good time to
> > discuss the process of jettisoning 2.7 support and standardizing on 3.6
> > for Portage.
> >
> > Right now, a custom 'asyncio-like' set of classes is used for async in
> > Portage. This limits interoperability with other async-friendly python
> > modules and produces additional maintenance overhead for the project.
> >
> > Maybe even more importantly, 3.6+ compatibility also gives us the
> > ability to use many other new language features and reduce the
> > maintenance overhead of having 2.7 compatibility for the extensive
> > Portage code-base. We expect that this will improve the velocity of
> > Portage development, provide opportunities to improve Portage
> > functionality and also allow us clean up a lot of unnecessarily complex
> > code (and eliminate code) that is there only to ensure compatibility
> > with 2.7.
> >
> > Best,
> >
> > Daniel
>
> In portage-2.3.30, portage's python API includes an asyncio event loop
> policy that is compatible with PEP 492 coroutines using async and await
> syntax [1]. This is possible while still maintaining backward
> compatibility with python2.7, though obviously you need to use a newer
> python if you want to use async and await syntax.
>
> I've posted a code example that is compatible with python2.7 and uses
> portage's event loop policy to do the same thing as emerge --regen,
> complete with --jobs and --load-average options [2].
>
> [1] https://www.python.org/dev/peps/pep-0492/
> [2]
> https://blogs.gentoo.org/zmedico/2018/04/17/portage-api-asyncio-event-loop/

<Council hat>
For the record, I still believe this is within Portage team's domain
to decide.  Worst case, someone forks Portage and we switch to the fork
to provide better user experience.

If we really are supposed to vote on this, I would suggest a non-binding 
wording alike:

  Keeping Python<3.6 supported by Portage is recommended.

i.e. where 'abstain' votes would not result in implicit 'no' to your
request.
</Council hat>


As a developer, I am opposed to this.  I am yet to see any evidence
of claimed performance improvements, improved interoperability (with
what?) and the reduced maintenance overhead.  So far, the effort built
around asyncio seems to do the exact opposite of that, with
the refactoring adding a lot of extra code and a number of new bugs.

So please understand that I have serious doubts whether removing support
for older versions of Python would suddenly significantly reduce
the code size or complexity below the state before the refactoring
started.  This really feels like 'switching to the new cool API' more
than actually working for any real gain.

Removing support for Python 2.7 is going to be a major drawback for our
users.  I am not even going to consider those who intentionally reduce
their systems to 2.7 only for one reason or another.  However, 2.7 has
been the Gentoo's 'LTS' so far -- different versions of Python 3 come
and go but 2.7 was there for a very long time to ease the transition. 
Only recently one of the developers had to solve hard upgrade conflicts
and the simplest way of doing that was to unmerge Python 3 and use
Python 2.7 to install a new version.

Going up to 3.6 is even worse than that.  I would like to remind you
that 3.5 is the *current stable* (i.e. the target set by make.defaults).
 3.6 has stable keywords only as an effort to make it possible to test
and stabilize 3.6 support in packages sooner -- it does not imply that
we are ready to switch.  Not to mention the whole backwards
compatibility deal.

With EAPI 7 approaching in days (and the implementation going to
probably have some inevitable bugs), switching supported Python
implementation (or doing any major refactorings, to say) is a very bad
idea.  This is going to once again cause a horrible backwards
compatibility problem loop where people wouldn't be able to use EAPI 7
because they would need to provide means to upgrade Python and its
dependencies first, in order to be able to upgrade Portage.

-- 
Best regards,
Michał Górny