Re: [gentoo-dev] Re: [RFC] dev-python/namespace-* retirement plan

[Michał Górny]

On Sat, 2022-04-09 at 19:08 +0300, Arthur Zamarin wrote:
> On 09/04/2022 18.20, Michał Górny wrote:
> > Hi, everyone.
> >
> > TL;DR I think I came up with a feasible plan towards cleanly removing
> > dev-python/namespace-* packages that aren't technically needed anymore
> > with modern versions of Python.
> >
> >
> > What are namespace packages?  Let's take "zope" as an example.
> > Normally, various subpackages like "zope.interface" and
> > "zope.configuration" would have all to be present in a single directory,
> > that is inside the "zope" package.  However, if "zope" is made a
> > namespace package, its subpackages can be loaded from different
> > locations.  Notably, in order to test zope.configuration prior to
> > installing it, we load it from build tree but its dependency
> > zope.interface from system site-packages.
> >
> > Historically, for this to work, the "zope/__init__.py" had to specify
> > some magic incantations.  On Gentoo, we've added dev-python/namespace-*
> > that installed these magical "__init__.py" files and made all
> > subpackages (e.g. dev-python/zope-*) depend on it.
> >
> > However, Python 3.3 introduced PEP 420 implicit namespaces that made
> > magical incantations unnecessary -- if "zope" didn't include
> > "__init__.py", it was implicitly treated as a namespace.  Unfortunately,
> > there were some interoperability issues between the two kinds of magical
> > incantations and implicit namespaces.  So while we were able to retire
> > pkgutil-style namespaces, setuptools-style namespaces remained.
> >
> > I think we can change that now.
> >
> >
> > I've done some experiments, particularly with dev-python/zope-interface
> > and dev-python/zope-configuration.  If nobody finds a problem with this
> > solution, I think we can aim to bump all packages relying on namespaces
> > and retire them in 1-2 months.
> >
> > The rough idea is to remove RDEP on dev-python/namespace-* from
> > the package, and updating python_test to use a temporary pkgutil-style
> > incantation (yes, for setuptools-style packages), e.g.:
> >
> > ```
> > python_compile() {
> > 	distutils-r1_python_compile
> > 	find "${BUILD_DIR}" -name '*.pth' -delete || die
> > }
> >
> > python_test() {
> > 	cd "${BUILD_DIR}/install$(python_get_sitedir)" || die
> > 	# this is needed to keep the tests working while
> > 	# dev-python/namespace-zope is still installed
> > 	cat > zope/__init__.py <<-EOF || die
> > 		__path__ = __import__('pkgutil').extend_path(__path__,
> > __name__)
> > 	EOF
> > 	eunittest
> > 	rm zope/__init__.py || die
> > }
> > ```
>
> I quite like that we make the "hack" more local, just in test phase. But
> I think this code is quite error prone, and an easy mistake can be made
> if someone copies it incorrectly.
>
> Is it possible to add an eclass function to auto create and cleanup this
> file, and all I need to do is to pass it the correct namespace ("zope"
> in above example)?
>
> Also, by using a common function name, it would be easy to fix it if we
> did a mistake and find all consumers when we feel safe to cleanup this code.
>
> One point of failure for this function is how to do the auto cleanup? I
> was thinking using environment variables to hold our added magic names,
> which is created before `python_test` call and cleans after it is done.
> I really prefer if we could call it during prepare phase (so we can
> still use auto generated d_e_t calls).
>

That's pretty much the big problem -- I don't really know.  I feel like
adding two functions is pretty much error prone itself, especially given
that this 'cd' logic is not necessarily correct for other packages.

In the end, we're talking about a few ebuilds for period of a few
months, so I'm not sure if there's really a point in putting more effort
into it.

-- 
Best regards,
Michał Górny