[gentoo-dev] Slotting PyPy

[gentoo-dev] Slotting PyPy

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 4715 bytes --]

Hello,

Given that you've expressed your preference for dev-lang/python
remaining slotted, I'd like to open another can of worms: should we slot
PyPy consistently with it?  Some history and then my ideas below.


PyPy versioning
===============

PyPy has basically two versions:

>>>> sys.version_info
sys.version_info(major=3, minor=10, micro=14, releaselevel='final', serial=0)
>>>> sys.pypy_version_info
sys.pypy_version_info(major=7, minor=3, micro=17, releaselevel='final', serial=0)

The former indicates the CPython version it is compatible with,
and the stdlib version from CPython it used.  The latter is the actual
PyPy release.

PyPy is doing synchronous releases for all Python versions it supports
at the time, with PyPy release matching between them.  For example,
7.3.16 release involved the following tarballs:

  pypy3.10-v7.3.16-src.tar.bz2
  pypy3.9-v7.3.16-src.tar.bz2
  pypy2.7-v7.3.16-src.tar.bz2

Nowadays PyPy2.7 and PyPy3.10 are supported and released, and there
should be a PyPy3.11 release soon.


History and overview of Gentoo packages
=======================================

Initially, we had just PyPy 2.x packaged, as dev-python/pypy, e.g.:

  dev-python/pypy-7.3.17

corresponds to:

  pypy2.7-v7.3.17-src.tar.bz2

Then we added PyPy 3.x as a dev-python/pypy3 package, e.g.:

  dev-python/pypy3-7.3.16

would correspond to:

  pypy3.10-v7.3.16-src.tar.bz2

We have decided to support only a single PyPy 3.x slot, for two reasons:
1) PyPy is quite experimental, and 2) upstream is usually far behind
CPython on releases (PyPy is at 3.10 still, while CPython just released
3.13).

For the relatively short transitions when two PyPy 3.x versions were
released, we combined revisions and subslots, e.g.:

  dev-python/pypy3-7.3.16:0/pypy39-...
  dev-python/pypy3-7.3.16-r100:0/pypy310-...

would correspond to, respectively:

  pypy3.9-v7.3.16-src.tar.bz2
  pypy3.10-v7.3.16-src.tar.bz2

The eclasses would depend on 'dev-python/pypy3:=' to bind to a specific
subslot, and soon afterwards we'd just be providing the next PyPy 3.x
versions without the older.

Then, as we decided to keep older CPython versions around without eclass
support (3.8 and 3.9), it started to make sense to also keep old PyPy
3.x versions around -- also without eclass support.  So I've split the
packages even further, into:

  dev-python/pypy3_9-7.3.16
  dev-python/pypy3_10-7.3.16

that correspond to, respectively:

  pypy3.9-v7.3.16-src.tar.bz2
  pypy3.10-v7.3.16-src.tar.bz2

And dev-python/pypy3 remained as a subslotted virtual that would pull
whichever PyPy 3.x version we support at the time.


Slotting again
==============

A possible goal for the future would be to recombine all these packages
into one, e.g.:

  dev-lang/pypy-2.7.7.3.16:pypy27/...
  dev-lang/pypy-3.9.7.3.16:pypy39/...
  dev-lang/pypy-3.10.7.3.16:pypy310/...

corresponding to:

  pypy2.7-v7.3.16-src.tar.bz2
  pypy3.9-v7.3.16-src.tar.bz2
  pypy3.10-v7.3.16-src.tar.bz2

So the PV would combine slot and release version, slot would indicate
PyPy 3.x slot and subslot would indicate the ABI (changes rarely).

The ebuilds could now depend e.g. on:

  >=dev-lang/pypy-3.10:=

This would ensure that only slots newer than 3.10 are acceptable,
and that packages are rebuilt (as they are right now) once we introduce
3.11 slot.  Then, after the transitional period we'd update it to:

  >=dev-lang/pypy-3.11:=

and so on.


Backwards compatibility and transition
======================================

For the time being, we'd have to maintain dev-python/pypy3
as a "virtual" for compatibility.  The eclasses would be updated, so
that newly built packages depend on and bind to the slots of the new
package.

Once PyPy3.11 is released, dev-python/pypy3 would gain a new subslot,
and the := deps will be triggering the rebuild.  At the same time,
the rebuild will result in the packages updating to depend on the new
package.  Some time after this transition, we'd be able to last rite
dev-python/pypy3.


Summary
=======

So the rough idea is that we'd be replacing dev-python/pypy, dev-
python/pypy3, dev-python/pypy3_9, dev-python/pypy3_10 with a single
slotted package.  At least the second package would have to preserved
for some time, for backwards compatibility; the others are less
significant since they are not used in eclass-generated dependencies.

I suppose we'd also be then aiming to combine dev-python/pypy*-exe
dev-python/pypy*-exe-bin to some degree, but that's a minor point, since
they are implementation details.

WDYT?

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Slotting PyPy

[Jérôme Carretero]

[-- Attachment #1: Type: text/plain, Size: 705 bytes --]

On Thu, 2024-10-17 at 14:12 +0200, Michał Górny wrote:
> Hello,
> 
> Given that you've expressed your preference for dev-lang/python
> remaining slotted, I'd like to open another can of worms: should we
> slot
> PyPy consistently with it?  Some history and then my ideas below.
> 

FWIW I'm all for it.

One reason is that I have current /usr/lib/pypy3.9 but still residues
of some dev-python packages that somehow haven't been updated in
/usr/lib/pypy3.8/site-packages and older folders, when pypy itself of
the corresponding version isn't even there.
It would be nice for the USE magic to work with pypy so such things
can't happen.


Thank you and best regards,

-- 
Jérôme

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 854 bytes --]

Re: [gentoo-dev] Slotting PyPy

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]

On Thu, 2024-10-17 at 08:47 -0400, Jérôme Carretero wrote:
> On Thu, 2024-10-17 at 14:12 +0200, Michał Górny wrote:
> > Hello,
> > 
> > Given that you've expressed your preference for dev-lang/python
> > remaining slotted, I'd like to open another can of worms: should we
> > slot
> > PyPy consistently with it?  Some history and then my ideas below.
> > 
> 
> FWIW I'm all for it.
> 
> One reason is that I have current /usr/lib/pypy3.9 but still residues
> of some dev-python packages that somehow haven't been updated in
> /usr/lib/pypy3.8/site-packages and older folders, when pypy itself of
> the corresponding version isn't even there.
> It would be nice for the USE magic to work with pypy so such things
> can't happen.

Unless I'm missing something, I don't think that's going to help with
your case.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Slotting PyPy

[Sam James]

Michał Górny <mgorny@gentoo.org> writes:

> Hello,
>
> Given that you've expressed your preference for dev-lang/python
> remaining slotted, I'd like to open another can of worms: should we slot
> PyPy consistently with it?  Some history and then my ideas below.

I'm on board with this. It'd been on my mind for a while but I went with
the existing scheme as you do most of the pypy work and I didn't want to
make your life hard by objecting on purity ;)

>
>
> PyPy versioning
> ===============
>
> PyPy has basically two versions:
>
>>>>> sys.version_info
> sys.version_info(major=3, minor=10, micro=14, releaselevel='final', serial=0)
>>>>> sys.pypy_version_info
> sys.pypy_version_info(major=7, minor=3, micro=17, releaselevel='final', serial=0)
>
> The former indicates the CPython version it is compatible with,
> and the stdlib version from CPython it used.  The latter is the actual
> PyPy release.
>
> PyPy is doing synchronous releases for all Python versions it supports
> at the time, with PyPy release matching between them.  For example,
> 7.3.16 release involved the following tarballs:
>
>   pypy3.10-v7.3.16-src.tar.bz2
>   pypy3.9-v7.3.16-src.tar.bz2
>   pypy2.7-v7.3.16-src.tar.bz2
>
> Nowadays PyPy2.7 and PyPy3.10 are supported and released, and there
> should be a PyPy3.11 release soon.
>
>
> History and overview of Gentoo packages
> =======================================
>
> Initially, we had just PyPy 2.x packaged, as dev-python/pypy, e.g.:
>
>   dev-python/pypy-7.3.17
>
> corresponds to:
>
>   pypy2.7-v7.3.17-src.tar.bz2
>
> Then we added PyPy 3.x as a dev-python/pypy3 package, e.g.:
>
>   dev-python/pypy3-7.3.16
>
> would correspond to:
>
>   pypy3.10-v7.3.16-src.tar.bz2
>
> We have decided to support only a single PyPy 3.x slot, for two reasons:
> 1) PyPy is quite experimental, and 2) upstream is usually far behind
> CPython on releases (PyPy is at 3.10 still, while CPython just released
> 3.13).
>
> For the relatively short transitions when two PyPy 3.x versions were
> released, we combined revisions and subslots, e.g.:
>
>   dev-python/pypy3-7.3.16:0/pypy39-...
>   dev-python/pypy3-7.3.16-r100:0/pypy310-...
>
> would correspond to, respectively:
>
>   pypy3.9-v7.3.16-src.tar.bz2
>   pypy3.10-v7.3.16-src.tar.bz2
>
> The eclasses would depend on 'dev-python/pypy3:=' to bind to a specific
> subslot, and soon afterwards we'd just be providing the next PyPy 3.x
> versions without the older.
>
> Then, as we decided to keep older CPython versions around without eclass
> support (3.8 and 3.9), it started to make sense to also keep old PyPy
> 3.x versions around -- also without eclass support.  So I've split the
> packages even further, into:
>
>   dev-python/pypy3_9-7.3.16
>   dev-python/pypy3_10-7.3.16
>
> that correspond to, respectively:
>
>   pypy3.9-v7.3.16-src.tar.bz2
>   pypy3.10-v7.3.16-src.tar.bz2
>
> And dev-python/pypy3 remained as a subslotted virtual that would pull
> whichever PyPy 3.x version we support at the time.
>
>
> Slotting again
> ==============
>
> A possible goal for the future would be to recombine all these packages
> into one, e.g.:
>
>   dev-lang/pypy-2.7.7.3.16:pypy27/...
>   dev-lang/pypy-3.9.7.3.16:pypy39/...
>   dev-lang/pypy-3.10.7.3.16:pypy310/...
>
> corresponding to:
>
>   pypy2.7-v7.3.16-src.tar.bz2
>   pypy3.9-v7.3.16-src.tar.bz2
>   pypy3.10-v7.3.16-src.tar.bz2
>
> So the PV would combine slot and release version, slot would indicate
> PyPy 3.x slot and subslot would indicate the ABI (changes rarely).
>
> The ebuilds could now depend e.g. on:
>
>   >=dev-lang/pypy-3.10:=
>
> This would ensure that only slots newer than 3.10 are acceptable,
> and that packages are rebuilt (as they are right now) once we introduce
> 3.11 slot.  Then, after the transitional period we'd update it to:
>
>   >=dev-lang/pypy-3.11:=
>
> and so on.
>
>
> Backwards compatibility and transition
> ======================================
>
> For the time being, we'd have to maintain dev-python/pypy3
> as a "virtual" for compatibility.  The eclasses would be updated, so
> that newly built packages depend on and bind to the slots of the new
> package.
>
> Once PyPy3.11 is released, dev-python/pypy3 would gain a new subslot,
> and the := deps will be triggering the rebuild.  At the same time,
> the rebuild will result in the packages updating to depend on the new
> package.  Some time after this transition, we'd be able to last rite
> dev-python/pypy3.
>
>
> Summary
> =======
>
> So the rough idea is that we'd be replacing dev-python/pypy, dev-
> python/pypy3, dev-python/pypy3_9, dev-python/pypy3_10 with a single
> slotted package.  At least the second package would have to preserved
> for some time, for backwards compatibility; the others are less
> significant since they are not used in eclass-generated dependencies.
>
> I suppose we'd also be then aiming to combine dev-python/pypy*-exe
> dev-python/pypy*-exe-bin to some degree, but that's a minor point, since
> they are implementation details.
>
> WDYT?

I think we should do it, but I have one question: I thought (I might be
misremembering though) that one reason for the existing scheme was to
allow easy testing of pypy versions for the same Python version outside
of packages (e.g. test two pypy3.10 versions).

But that's kind of limited, we can do that with binpkgs, and we already
can't do that with CPython. IMO the consistency is more worth it.

Re: [gentoo-dev] Slotting PyPy

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

On Sat, 2024-10-19 at 11:42 +0100, Sam James wrote:
> I think we should do it, but I have one question: I thought (I might be
> misremembering though) that one reason for the existing scheme was to
> allow easy testing of pypy versions for the same Python version outside
> of packages (e.g. test two pypy3.10 versions).

Not sure what you mean by that.  You can't have two pypy3.10 versions
installed simultaneously.  Unless you mean switching between versions
without having to rebuild the whole interpreter -- that part will remain
as-is, since pypy*-exe* will remain separate, though I'm not sure yet if
I'll merge them into slots or leave completely separate.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Slotting PyPy

[Mitchell Dorrell]

[-- Attachment #1: Type: text/plain, Size: 3404 bytes --]

On Thu, Oct 17, 2024, 08:12 Michał Górny <mgorny@gentoo.org> wrote:

> PyPy has basically two versions:
>
> >>>> sys.version_info
> sys.version_info(major=3, minor=10, micro=14, releaselevel='final',
> serial=0)
> >>>> sys.pypy_version_info
> sys.pypy_version_info(major=7, minor=3, micro=17, releaselevel='final',
> serial=0)
>
> The former indicates the CPython version it is compatible with,
> and the stdlib version from CPython it used.  The latter is the actual
> PyPy release.
>
> PyPy is doing synchronous releases for all Python versions it supports at
> the time, with PyPy release matching between them.  For example, 7.3.16
> release involved the following tarballs:
>
>   pypy3.10-v7.3.16-src.tar.bz2
>   pypy3.9-v7.3.16-src.tar.bz2
>   pypy2.7-v7.3.16-src.tar.bz2
>

This sounds like another potential application of the "VARIANTS" concept I
suggested last week:
https://marc.info/?l=gentoo-dev&m=172877401607588
In short, some software has multiple variants of each version which may
need to be installed simultaneously, without the older/newer relationship
that slots imply. (There could be other benefits, too, irrelevant to this
discussion.) Perhaps I should implement this as a proof-of-concept and
start a separate thread to discuss it.

For the relatively short transitions when two PyPy 3.x versions were
> released, we combined revisions and subslots, e.g.:
>
>   dev-python/pypy3-7.3.16:0/pypy39-...
>   dev-python/pypy3-7.3.16-r100:0/pypy310-...
>
> would correspond to, respectively:
>
>   pypy3.9-v7.3.16-src.tar.bz2
>   pypy3.10-v7.3.16-src.tar.bz2
>
> The eclasses would depend on 'dev-python/pypy3:=' to bind to a specific
> subslot, and soon afterwards we'd just be providing the next PyPy 3.x
> versions without the older.
>
> Then, as we decided to keep older CPython versions around without eclass
> support (3.8 and 3.9), it started to make sense to also keep old PyPy 3.x
> versions around -- also without eclass support.  So I've split the packages
> even further, into:
>
>   dev-python/pypy3_9-7.3.16
>   dev-python/pypy3_10-7.3.16
>
> that correspond to, respectively:
>
>   pypy3.9-v7.3.16-src.tar.bz2
>   pypy3.10-v7.3.16-src.tar.bz2
>
> And dev-python/pypy3 remained as a subslotted virtual that would pull
> whichever PyPy 3.x version we support at the time.
>
>
> Slotting again
> ==============
>
> A possible goal for the future would be to recombine all these packages
> into one, e.g.:
>
>   dev-lang/pypy-2.7.7.3.16:pypy27/...
>   dev-lang/pypy-3.9.7.3.16:pypy39/...
>   dev-lang/pypy-3.10.7.3.16:pypy310/...
>
> corresponding to:
>
>   pypy2.7-v7.3.16-src.tar.bz2
>   pypy3.9-v7.3.16-src.tar.bz2
>   pypy3.10-v7.3.16-src.tar.bz2
>
> So the PV would combine slot and release version, slot would indicate PyPy
> 3.x slot and subslot would indicate the ABI (changes rarely).
>

Previously, you "split the packages even further", but now you're
suggesting to "recombine all these packages into one". What was the
original reason for splitting the packages, and what has changed since then?

I don't understand enough of the Python and PyPy infrastructure to support
or object to either approach. I generally think it's more elegant to have a
single package for a single codebase/project, but the practical concerns
are more important.

- Mitchell Dorrell

>

[-- Attachment #2: Type: text/html, Size: 4673 bytes --]

Re: [gentoo-dev] Slotting PyPy

[Sam James]

Michał Górny <mgorny@gentoo.org> writes:

> On Sat, 2024-10-19 at 11:42 +0100, Sam James wrote:
>> I think we should do it, but I have one question: I thought (I might be
>> misremembering though) that one reason for the existing scheme was to
>> allow easy testing of pypy versions for the same Python version outside
>> of packages (e.g. test two pypy3.10 versions).
>
> Not sure what you mean by that.  You can't have two pypy3.10 versions
> installed simultaneously.  Unless you mean switching between versions
> without having to rebuild the whole interpreter -- that part will remain
> as-is, since pypy*-exe* will remain separate, though I'm not sure yet if
> I'll merge them into slots or leave completely separate.

Yeah, that's what I was getting at - but I don't think it's a big dela anyway.

Re: [gentoo-dev] Slotting PyPy

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1569 bytes --]

On Thu, 2024-10-17 at 14:12 +0200, Michał Górny wrote:
> The ebuilds could now depend e.g. on:
> 
>   >=dev-lang/pypy-3.10:=
> 
> This would ensure that only slots newer than 3.10 are acceptable,
> and that packages are rebuilt (as they are right now) once we introduce
> 3.11 slot.  Then, after the transitional period we'd update it to:
> 
>   >=dev-lang/pypy-3.11:=
> 
> and so on.
> 

Ok, I've missed something significant here.

Along all the reshuffling, dev-python/pypy3 is not really a virtual
package but supplies /usr/bin/pypy3 (vs. pypy3.x supplied by dev-
python/pypy3_x).  Switching to one-package model would require us to
install "pypy3" as part of the baseline package, and that is going to
make transitions less friendly.

In other words, with the old approach, you could have pypy3.10
and pypy3.11 installed simultaneously, and switch to 3.11 via upgrading
dev-python/pypy3 metapackage.  With the new approach, we won't be able
to have fully coexisting pypy3.10 and pypy3.11 packages without them
conflicting over /usr/bin/pypy3.  During the transition period, we'd
have to have something like USE=symlink to clearly distinguish the PyPy
version that satisfies PYTHON_TARGETS from the other one.

Not saying it's a fatal flaw but certainly makes things less nice.
I don't see any "obviously nice" solutions, other than having a separate
metapackage for the purpose of keeping /usr/bin/pypy3.  But then,
merging dev-python/pypy3_x becomes a meaningless implementation detail.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]