[gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1937 bytes --]

Hello,

Historically, all versions of CPython were slotted in a single package,
i.e.:

  dev-lang/python:3.N

This approach has been causing a major annoyance for users -- due to
Portage "greedy" upgrade behavior, any time a new Python version was
keyworded, Portage insisted on installing it, even though user's
selected targets did not request the specific version.  The potentially
worst consequence of that would be random user scripts stopping to work,
as they suddenly start using new Python, while all their dependencies
are still installed per PYTHON_TARGETS.

Upstream has recently added freethreading support to CPython.  Since
this support is not ABI compatible with the regular build, we need to
introduce a separate target for it, and to package it separately.
In the planned patchset, I've already put it as a separate package (dev-
lang/python-freethreading), because otherwise Portage would insist
on upgrading to it!

However, I think the cleanest way forward would be to stop slotting
CPython like this, and instead have a separate package for each version,
just like the vast majority of distributions do, i.e.:

  dev-lang/python3_N

This naturally means that only the specific version requested (e.g. via
targets) would be installed, and no cross-slot autoupgrades would
happen.  Ideally, I'd like to start doing that with Python 3.14 whose
first alpha is expected next week.  Depending on how they handle
freethreading, we'd end up having the first or both of:

  dev-lang/python3_14
  dev-lang/python3_14t

(Alternatives: python-3_14, python-freethreading-3_14? Though I think
following PYTHON_TARGETS is cleaner here.)

As a side notice, the existing versions would probably remain as-is
until removal, since there's really no gain in splitting them, given
we'd have to retain compatibility with existing depstrings.

Comments?

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

[gentoo-dev] Re: [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Luca Barbato]

On 12/10/24 10:12, Michał Górny wrote:
> Comments?
> 
I'm afraid it would lead to way too many packages and I'm not sure the 
overall experience would be an improvement.

With your proposed solution, if an user wants to have any version of 
python what should ask to emerge?

An alternative for freethreading support wouldn't be to install both 
from the same package python-3.14 and have the two PYTHON_TARGETS ?

lu

Re: [gentoo-dev] Re: [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

On Sat, 2024-10-12 at 10:50 +0200, Luca Barbato wrote:
> On 12/10/24 10:12, Michał Górny wrote:
> > Comments?
> > 
> I'm afraid it would lead to way too many packages and I'm not sure the 
> overall experience would be an improvement.

5 are too many?

> With your proposed solution, if an user wants to have any version of 
> python what should ask to emerge?

Can you actually imagine having a Gentoo system with no Python
preinstalled, with an user actually needing to emerge one?

In fact, even today "emerge dev-lang/python" is probably a bad solution,
as it will lead to a beta/rc version on an ~arch system most
of the time.

> An alternative for freethreading support wouldn't be to install both 
> from the same package python-3.14 and have the two PYTHON_TARGETS ?

I don't understand.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Re: [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Anna (cybertailor) Vyalkova]

On 2024-10-12 11:13, Michał Górny wrote:
>On Sat, 2024-10-12 at 10:50 +0200, Luca Barbato wrote:
>> On 12/10/24 10:12, Michał Górny wrote:
>> > Comments?
>> >
>> I'm afraid it would lead to way too many packages and I'm not sure the
>> overall experience would be an improvement.
>
>5 are too many?

Absolutely no, and your proposal seems to solve the shortcomings of 
slotting given the limitations. Just Python is not the only language 
with multiple simultaneous versions supported in Gentoo (Ruby comes to 
mind first), and it opens a slippery slope.

Is there anything that makes Python unique enough to stand out from 
conventions estabilished for packaging other languages? Or would it 
essentially lead to a new policy?


(I'm satisfied with using 'package.mask' and 'package.accept_keywords' 
to prevent "greedy" upgrades, if that matters)

Re: [gentoo-dev] Re: [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On Sat, 2024-10-12 at 17:30 +0500, Anna (cybertailor) Vyalkova wrote:
> On 2024-10-12 11:13, Michał Górny wrote:
> > On Sat, 2024-10-12 at 10:50 +0200, Luca Barbato wrote:
> > > On 12/10/24 10:12, Michał Górny wrote:
> > > > Comments?
> > > > 
> > > I'm afraid it would lead to way too many packages and I'm not sure the
> > > overall experience would be an improvement.
> > 
> > 5 are too many?
> 
> Absolutely no, and your proposal seems to solve the shortcomings of 
> slotting given the limitations. Just Python is not the only language 
> with multiple simultaneous versions supported in Gentoo (Ruby comes to 
> mind first), and it opens a slippery slope.

I still find this weak, compared to ~20k packages in Gentoo right now.

> Is there anything that makes Python unique enough to stand out from 
> conventions estabilished for packaging other languages? Or would it 
> essentially lead to a new policy?

Python pretty much sets the quality standards for Gentoo these days.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Re: [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Luca Barbato]

On 12/10/24 11:13, Michał Górny wrote:
> On Sat, 2024-10-12 at 10:50 +0200, Luca Barbato wrote:
>> On 12/10/24 10:12, Michał Górny wrote:
>>> Comments?
>>>
>> I'm afraid it would lead to way too many packages and I'm not sure the
>> overall experience would be an improvement.
> 
> 5 are too many?

potentially it is python_{version}_{variants} so at least 10*2 assuming 
we keep around 3.{N..N+5} and we have two worthy variants.

Plus the chore of treecleaning older packages. Not sure if it is worth it.

>> With your proposed solution, if an user wants to have any version of
>> python what should ask to emerge?
> 
> Can you actually imagine having a Gentoo system with no Python
> preinstalled, with an user actually needing to emerge one?

right now if I need a specific version of python I have to do emerge 
=python-N* and usually it works as intended.

people would have to get used to do emerge python_{that specific N}


> In fact, even today "emerge dev-lang/python" is probably a bad solution,
> as it will lead to a beta/rc version on an ~arch system most
> of the time.
> 
>> An alternative for freethreading support wouldn't be to install both
>> from the same package python-3.14 and have the two PYTHON_TARGETS ?
> 
> I don't understand.

emerge =python-3.14 would install both a non-freethreading and a 
freethreading version and it would satisfy 3_14 and 3_14t at the same.

lu

Re: [gentoo-dev] Re: [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]

On Sat, 2024-10-12 at 15:00 +0200, Luca Barbato wrote:
> On 12/10/24 11:13, Michał Górny wrote:
> > On Sat, 2024-10-12 at 10:50 +0200, Luca Barbato wrote:
> > > On 12/10/24 10:12, Michał Górny wrote:
> > > > Comments?
> > > > 
> > > I'm afraid it would lead to way too many packages and I'm not sure the
> > > overall experience would be an improvement.
> > 
> > 5 are too many?
> 
> potentially it is python_{version}_{variants} so at least 10*2 assuming 
> we keep around 3.{N..N+5} and we have two worthy variants.
> 
> Plus the chore of treecleaning older packages. Not sure if it is worth it.

That sounds like an advantage, actually.  Currently, we keep older
versions for a long time, then they suddenly disappear.  All the things
considered, perhaps we ought to verbosely last rite them.

> > In fact, even today "emerge dev-lang/python" is probably a bad solution,
> > as it will lead to a beta/rc version on an ~arch system most
> > of the time.
> > 
> > > An alternative for freethreading support wouldn't be to install both
> > > from the same package python-3.14 and have the two PYTHON_TARGETS ?
> > 
> > I don't understand.
> 
> emerge =python-3.14 would install both a non-freethreading and a 
> freethreading version and it would satisfy 3_14 and 3_14t at the same.
> 

But that's not how PMs work?  It would install only the "newer" version,
which would probably mean freethreading, which is not what people
usually want.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Re: [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Luca Barbato]

On 12/10/24 15:03, Michał Górny wrote:

>> emerge =python-3.14 would install both a non-freethreading and a
>> freethreading version and it would satisfy 3_14 and 3_14t at the same.
>>
> 
> But that's not how PMs work?  It would install only the "newer" version,
> which would probably mean freethreading, which is not what people
> usually want.

I mean that the python-3.14 ebuild would install both versions for the 
time being. the PM does not see anything special.

lu

Re: [gentoo-dev] Re: [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Sam James]

Michał Górny <mgorny@gentoo.org> writes:

> On Sat, 2024-10-12 at 10:50 +0200, Luca Barbato wrote:
>> On 12/10/24 10:12, Michał Górny wrote:
>> > Comments?
>> > 
>> I'm afraid it would lead to way too many packages and I'm not sure the 
>> overall experience would be an improvement.
>
> 5 are too many?
>
>> With your proposed solution, if an user wants to have any version of 
>> python what should ask to emerge?
>
> Can you actually imagine having a Gentoo system with no Python
> preinstalled, with an user actually needing to emerge one?

I'll note that this happens readily with crossdev.

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Eray Aslan]

On Sat, Oct 12, 2024 at 10:12:56AM +0200, Michał Górny wrote:
> Historically, all versions of CPython were slotted in a single package,
> i.e.:
> 
>   dev-lang/python:3.N
> 
> This approach has been causing a major annoyance for users -- due to
> Portage "greedy" upgrade behavior, any time a new Python version was
> keyworded, Portage insisted on installing it, even though user's
> selected targets did not request the specific version.  The potentially
> worst consequence of that would be random user scripts stopping to work,
> as they suddenly start using new Python, while all their dependencies
> are still installed per PYTHON_TARGETS.
> 
> Upstream has recently added freethreading support to CPython.  Since
> this support is not ABI compatible with the regular build, we need to
> introduce a separate target for it, and to package it separately.
> In the planned patchset, I've already put it as a separate package (dev-
> lang/python-freethreading), because otherwise Portage would insist
> on upgrading to it!
> 
> However, I think the cleanest way forward would be to stop slotting
> CPython like this, and instead have a separate package for each version,
> just like the vast majority of distributions do, i.e.:
> 
>   dev-lang/python3_N

FWIW, if I were to start anew, my ideal env would probably be:

1/ keep N versions of python installed all the time
2/ give the option to define N with sane defaults and limits
3/ upgrade greedily
4/ give the option to define upgrade behaviour (switch default python
interpreter to newest, no change, no change unless no longer in the tree
etc)
5/ do not change from free-threaded to regular GIL versions or vice
versa automatically. keep to one lane (this might even apply to jits so
perhaps make it easy to define lanes)
6/ make it easy to create virtualenvs with any python interpreter
7/ make it easy to change between any python interpreter and set a
default

Your proposal is close. It would require some (minimal) manual work but
that is reasonable afaic

> This naturally means that only the specific version requested (e.g. via
> targets) would be installed, and no cross-slot autoupgrades would
> happen.  Ideally, I'd like to start doing that with Python 3.14 whose
> first alpha is expected next week.  Depending on how they handle
> freethreading, we'd end up having the first or both of:
> 
>   dev-lang/python3_14
>   dev-lang/python3_14t
> 
> (Alternatives: python-3_14, python-freethreading-3_14? Though I think
> following PYTHON_TARGETS is cleaner here.)

dev-lang/python3_14t is a better looking option imho

> As a side notice, the existing versions would probably remain as-is
> until removal, since there's really no gain in splitting them, given
> we'd have to retain compatibility with existing depstrings.

so no dev-lang/python3_13t? nothing crucial but would have been nice

> Comments?

Thanks for doing this

-- 
Eray

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

On Sat, 2024-10-12 at 11:49 +0200, Eray Aslan wrote:
> On Sat, Oct 12, 2024 at 10:12:56AM +0200, Michał Górny wrote:
> 
> > As a side notice, the existing versions would probably remain as-is
> > until removal, since there's really no gain in splitting them, given
> > we'd have to retain compatibility with existing depstrings.
> 
> so no dev-lang/python3_13t? nothing crucial but would have been nice

Actually, I've put the whole freethreading work on hold to determine if
we want to go for dev-lang/python3_13t.  What I've meant, regular 3.13
and older will remain as dev-lang/python:3.N.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

>>>>> On Sat, 12 Oct 2024, Michał Górny wrote:

> However, I think the cleanest way forward would be to stop slotting
> CPython like this, and instead have a separate package for each version,
> just like the vast majority of distributions do, i.e.:

>   dev-lang/python3_N

That other distributions do it that way is not an argument because most
other distributions don't have slots.

> This naturally means that only the specific version requested (e.g. via
> targets) would be installed, and no cross-slot autoupgrades would
> happen.  Ideally, I'd like to start doing that with Python 3.14 whose
> first alpha is expected next week.  Depending on how they handle
> freethreading, we'd end up having the first or both of:

>   dev-lang/python3_14
>   dev-lang/python3_14t

IMHO this would abuse the package name for information that absolutely
doesn't belong there. It belongs in PV or SLOT.

To me it seems that you try to work around a problem (greedy upgrade
behaviour) that should really be solved in the package manager.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

>>>>> On Sat, 12 Oct 2024, Ulrich Mueller wrote:

> IMHO this would abuse the package name for information that absolutely
> doesn't belong there. It belongs in PV or SLOT.

> To me it seems that you try to work around a problem (greedy upgrade
> behaviour) that should really be solved in the package manager.

How about introducing something like PROPERTIES="separate-slot" instead,
which would instruct the PM to treat the slot like a separate package?

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

On Sat, 2024-10-12 at 12:03 +0200, Ulrich Mueller wrote:
> > > > > > On Sat, 12 Oct 2024, Ulrich Mueller wrote:
> 
> > IMHO this would abuse the package name for information that absolutely
> > doesn't belong there. It belongs in PV or SLOT.
> 
> > To me it seems that you try to work around a problem (greedy upgrade
> > behaviour) that should really be solved in the package manager.
> 
> How about introducing something like PROPERTIES="separate-slot" instead,
> which would instruct the PM to treat the slot like a separate package?
> 

Isn't it ironic that I didn't manage to finish sending my reply about
slots being abused and things bent so hard to make them fit slots,
and you've already managed to propose adding more complexity in order to
bend things even more into the wrong solution?

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2049 bytes --]

On Sat, 2024-10-12 at 11:59 +0200, Ulrich Mueller wrote:
> > > > > > On Sat, 12 Oct 2024, Michał Górny wrote:
> 
> > However, I think the cleanest way forward would be to stop slotting
> > CPython like this, and instead have a separate package for each version,
> > just like the vast majority of distributions do, i.e.:
> 
> >   dev-lang/python3_N
> 
> That other distributions do it that way is not an argument because most
> other distributions don't have slots.
> 
> > This naturally means that only the specific version requested (e.g. via
> > targets) would be installed, and no cross-slot autoupgrades would
> > happen.  Ideally, I'd like to start doing that with Python 3.14 whose
> > first alpha is expected next week.  Depending on how they handle
> > freethreading, we'd end up having the first or both of:
> 
> >   dev-lang/python3_14
> >   dev-lang/python3_14t
> 
> IMHO this would abuse the package name for information that absolutely
> doesn't belong there. It belongs in PV or SLOT.
> 
> To me it seems that you try to work around a problem (greedy upgrade
> behaviour) that should really be solved in the package manager.

In my opinion, it's the other way around.  We have slots, that are a fit
solution for packages that are roughly compatible between every major
release, and we keep abusing them for every single thing we can bend
enough to make it fit.

Greedy upgrade behavior makes sense when you have packages where :=
and :* operators are applicable.  It doesn't make sense when every
single dependency is expected to specify an explicit slot.  In fact,
when you are never supposed to depend on the package without specifying
a slot, why would you think slots are the right solution?

In fact, the "freethreading" variation already implies that there could
be more than one "slot" per package version.

It's as meaningless as having sqlite3 packaged as sqlite:3, or gtk that
is now randomly split between gtk+:2, gtk+:3 and gtk:4.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

>>>>> On Sat, 12 Oct 2024, Michał Górny wrote:

>> IMHO this would abuse the package name for information that absolutely
>> doesn't belong there. It belongs in PV or SLOT.
>> 
>> To me it seems that you try to work around a problem (greedy upgrade
>> behaviour) that should really be solved in the package manager.

> In my opinion, it's the other way around.  We have slots, that are a fit
> solution for packages that are roughly compatible between every major
> release, and we keep abusing them for every single thing we can bend
> enough to make it fit.

So are you saying that Python versions aren't "roughly compatible"
between releases? Like, they're a completely different language?

Also, we're not talking about major releases here (that would have been
Python 2 vs 3).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

On Sat, 2024-10-12 at 12:13 +0200, Ulrich Mueller wrote:
> > > > > > On Sat, 12 Oct 2024, Michał Górny wrote:
> 
> > > IMHO this would abuse the package name for information that absolutely
> > > doesn't belong there. It belongs in PV or SLOT.
> > > 
> > > To me it seems that you try to work around a problem (greedy upgrade
> > > behaviour) that should really be solved in the package manager.
> 
> > In my opinion, it's the other way around.  We have slots, that are a fit
> > solution for packages that are roughly compatible between every major
> > release, and we keep abusing them for every single thing we can bend
> > enough to make it fit.
> 
> So are you saying that Python versions aren't "roughly compatible"
> between releases? Like, they're a completely different language?

They're not compatible in the sense "if I install package for
python3.12, it will work in python3.13", because every version uses
a separate package tree.  Upstream has effectively made every version
separate.  On top of that, the same "minor" version can have a PyPy
variation and a "freethreading" variation now.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Mitchell Dorrell]

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

On Sat, Oct 12, 2024, 06:23 Michał Górny <mgorny@gentoo.org> wrote:

> On Sat, 2024-10-12 at 12:13 +0200, Ulrich Mueller wrote:
> > > > > > > On Sat, 12 Oct 2024, Michał Górny wrote:
> >
> > > > IMHO this would abuse the package name for information that
> absolutely
> > > > doesn't belong there. It belongs in PV or SLOT.
> > > >
> > > > To me it seems that you try to work around a problem (greedy upgrade
> > > > behaviour) that should really be solved in the package manager.
> >
> > > In my opinion, it's the other way around.  We have slots, that are a
> fit
> > > solution for packages that are roughly compatible between every major
> > > release, and we keep abusing them for every single thing we can bend
> > > enough to make it fit.
> >
> > So are you saying that Python versions aren't "roughly compatible"
> > between releases? Like, they're a completely different language?
>
> They're not compatible in the sense "if I install package for
> python3.12, it will work in python3.13", because every version uses
> a separate package tree.  Upstream has effectively made every version
> separate.  On top of that, the same "minor" version can have a PyPy
> variation and a "freethreading" variation now.
>
> --
> Best regards,
> Michał Górny
>

How will this affect ebuilds that can use any version of Python 3? Will it
need a giant or-block in DEPEND?

>

[-- Attachment #2: Type: text/html, Size: 2098 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]

On Sat, 2024-10-12 at 07:23 -0400, Mitchell Dorrell wrote:
> On Sat, Oct 12, 2024, 06:23 Michał Górny <mgorny@gentoo.org> wrote:
> 
> > On Sat, 2024-10-12 at 12:13 +0200, Ulrich Mueller wrote:
> > > > > > > > On Sat, 12 Oct 2024, Michał Górny wrote:
> > > 
> > > > > IMHO this would abuse the package name for information that
> > absolutely
> > > > > doesn't belong there. It belongs in PV or SLOT.
> > > > > 
> > > > > To me it seems that you try to work around a problem (greedy upgrade
> > > > > behaviour) that should really be solved in the package manager.
> > > 
> > > > In my opinion, it's the other way around.  We have slots, that are a
> > fit
> > > > solution for packages that are roughly compatible between every major
> > > > release, and we keep abusing them for every single thing we can bend
> > > > enough to make it fit.
> > > 
> > > So are you saying that Python versions aren't "roughly compatible"
> > > between releases? Like, they're a completely different language?
> > 
> > They're not compatible in the sense "if I install package for
> > python3.12, it will work in python3.13", because every version uses
> > a separate package tree.  Upstream has effectively made every version
> > separate.  On top of that, the same "minor" version can have a PyPy
> > variation and a "freethreading" variation now.
> > 
> > --
> > Best regards,
> > Michał Górny
> > 
> 
> How will this affect ebuilds that can use any version of Python 3? Will it
> need a giant or-block in DEPEND?
> 

Ebuilds are forbidden from depending on dev-lang/python directly. 
Eclasses take care of generating correct dependency strings.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Mitchell Dorrell]

[-- Attachment #1: Type: text/plain, Size: 2484 bytes --]

How will the upgrade path look, from e.g. 3.16 to 3.17? What will the
typical Gentoo user experience be? What will the experience be for a Python
developer with a local overlay containing custom Python packages?

I didn't like this idea at first, but I'm warming up to it.

On Sat, Oct 12, 2024, 06:05 Michał Górny <mgorny@gentoo.org> wrote:

> On Sat, 2024-10-12 at 11:59 +0200, Ulrich Mueller wrote:
> > > > > > > On Sat, 12 Oct 2024, Michał Górny wrote:
> >
> > > However, I think the cleanest way forward would be to stop slotting
> > > CPython like this, and instead have a separate package for each
> version,
> > > just like the vast majority of distributions do, i.e.:
> >
> > >   dev-lang/python3_N
> >
> > That other distributions do it that way is not an argument because most
> > other distributions don't have slots.
> >
> > > This naturally means that only the specific version requested (e.g. via
> > > targets) would be installed, and no cross-slot autoupgrades would
> > > happen.  Ideally, I'd like to start doing that with Python 3.14 whose
> > > first alpha is expected next week.  Depending on how they handle
> > > freethreading, we'd end up having the first or both of:
> >
> > >   dev-lang/python3_14
> > >   dev-lang/python3_14t
> >
> > IMHO this would abuse the package name for information that absolutely
> > doesn't belong there. It belongs in PV or SLOT.
> >
> > To me it seems that you try to work around a problem (greedy upgrade
> > behaviour) that should really be solved in the package manager.
>
> In my opinion, it's the other way around.  We have slots, that are a fit
> solution for packages that are roughly compatible between every major
> release, and we keep abusing them for every single thing we can bend
> enough to make it fit.
>
> Greedy upgrade behavior makes sense when you have packages where :=
> and :* operators are applicable.  It doesn't make sense when every
> single dependency is expected to specify an explicit slot.  In fact,
> when you are never supposed to depend on the package without specifying
> a slot, why would you think slots are the right solution?
>
> In fact, the "freethreading" variation already implies that there could
> be more than one "slot" per package version.
>
> It's as meaningless as having sqlite3 packaged as sqlite:3, or gtk that
> is now randomly split between gtk+:2, gtk+:3 and gtk:4.
>
> --
> Best regards,
> Michał Górny
>
>

[-- Attachment #2: Type: text/html, Size: 3101 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Sam James]

Mitchell Dorrell <mwd@psc.edu> writes:

> How will the upgrade path look, from e.g. 3.16 to 3.17? What will the typical Gentoo user experience be? What will the
> experience be for a Python developer with a local overlay containing custom Python packages?

It should be transparent because PYTHON_COMPAT will handle that as usual
for a local repository.

The upgrade thing will need testing just to make sure nothing odd
happens though.

>
> I didn't like this idea at first, but I'm warming up to it.
>
> On Sat, Oct 12, 2024, 06:05 Michał Górny <mgorny@gentoo.org> wrote:
>
>  On Sat, 2024-10-12 at 11:59 +0200, Ulrich Mueller wrote:
>  > > > > > > On Sat, 12 Oct 2024, Michał Górny wrote:
>  > 
>  > > However, I think the cleanest way forward would be to stop slotting
>  > > CPython like this, and instead have a separate package for each version,
>  > > just like the vast majority of distributions do, i.e.:
>  > 
>  > >   dev-lang/python3_N
>  > 
>  > That other distributions do it that way is not an argument because most
>  > other distributions don't have slots.
>  > 
>  > > This naturally means that only the specific version requested (e.g. via
>  > > targets) would be installed, and no cross-slot autoupgrades would
>  > > happen.  Ideally, I'd like to start doing that with Python 3.14 whose
>  > > first alpha is expected next week.  Depending on how they handle
>  > > freethreading, we'd end up having the first or both of:
>  > 
>  > >   dev-lang/python3_14
>  > >   dev-lang/python3_14t
>  > 
>  > IMHO this would abuse the package name for information that absolutely
>  > doesn't belong there. It belongs in PV or SLOT.
>  > 
>  > To me it seems that you try to work around a problem (greedy upgrade
>  > behaviour) that should really be solved in the package manager.
>
>  In my opinion, it's the other way around.  We have slots, that are a fit
>  solution for packages that are roughly compatible between every major
>  release, and we keep abusing them for every single thing we can bend
>  enough to make it fit.
>
>  Greedy upgrade behavior makes sense when you have packages where :=
>  and :* operators are applicable.  It doesn't make sense when every
>  single dependency is expected to specify an explicit slot.  In fact,
>  when you are never supposed to depend on the package without specifying
>  a slot, why would you think slots are the right solution?
>
>  In fact, the "freethreading" variation already implies that there could
>  be more than one "slot" per package version.
>
>  It's as meaningless as having sqlite3 packaged as sqlite:3, or gtk that
>  is now randomly split between gtk+:2, gtk+:3 and gtk:4.
>
>  -- 
>  Best regards,
>  Michał Górny

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Anna (navi) Figueiredo Gomes]

[-- Attachment #1.1.1: Type: text/plain, Size: 1709 bytes --]

> > IMHO this would abuse the package name for information that absolutely
> > doesn't belong there. It belongs in PV or SLOT.
> >
> > To me it seems that you try to work around a problem (greedy upgrade
> > behaviour) that should really be solved in the package manager.
>
> In my opinion, it's the other way around.  We have slots, that are a fit
> solution for packages that are roughly compatible between every major
> release, and we keep abusing them for every single thing we can bend
> enough to make it fit.
>
> It's as meaningless as having sqlite3 packaged as sqlite:3, or gtk that
> is now randomly split between gtk+:2, gtk+:3 and gtk:4.

SLOTs in my view, and in documentation descriptions, `ebuild(5)`, are
listed as "a package that may need to have multiple versions co-exist".
seemingly giving more semantics to package versions.

such semantics don't exist in other distros, so they represent them with
different packages. doing the same here feels like a hack, going around
a limitation of the SLOT system, and not the ideal solution at all.

i'd think instead looking for improving the SLOT system so that
dependencies can be even more structurally declared would be the
ideal solution, making different packages just breaks those semantics
to work around it's limitations.

> In fact, when you are never supposed to depend on the package without
> specifying a slot, why would you think slots are the right solution?

so yes i do think they're the right solution still, as they communicate
"this is the same package, the same project, just a different version
and this package needs *this* version". doing that is no different than
DEPEND="=foo/nya-4.2"

[-- Attachment #1.2: 68990292A7A98C5E.asc --]
[-- Type: application/pgp-keys, Size: 6230 bytes --]

-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBGN1ek4BEADRdFZqH33rYk/8yHpnS4SnTQIFfsS7xMS/lHoIYPTquT+TG/Gp
QNBvqextt8RsWnCY6XA5tGMYX7cutTPjF86Y2pYbLo7y7Mt/xgdhCyF3yFlHgFYr
/DyxQemRh8qHI0eMt5cUXGSjzCDSf+S8YFRbUJe8Q7rXM40xEXoqLukN+3UdmD1e
nY9UVN6Mv/mRLARfKiBQZDhBXRRNt7A2ZjQ7nnJDfv7oqYQwaKZtAskqW98owNp7
QXO6cukn8UwVRU5EvAmDsR9aVdb446y+0HuBNzHrjn14tCYpQ0S6EbAf/QAOOTzk
hCZekcQ5OT/hXfwbDfwStfc6Eb7B/2zcfVyKhdvzeuy0v2RQsIOGCujtbui4POQp
Lrm0MhxHmYM9DZVP1mB3ugwrDp4PQRUrfZVPVBlPwnBnlA2nL2QGl1bILrnVBItB
0nd+LGGnoRVLuvhGXN/Ytq8IMZiLtcOP9FIj4jIVBEFHjZnlr9uCe26FPCxbz8Rc
fXXyfIN8lT7B+ZwW5oFUmVNQ7w96vVewXgw7c/q4ikDDteorfT59ZWCIZYKCZz0N
oihZdAy4YrL1l7blJ+ycr6BFIz4y6fgWQFX+PuFIs7oJmFRPE1nj7wqOzSklJZAl
4Wr5cNVrN9FLK28JO0agdH/l0swVY+ytJKGAhKoscMQZff9JGQ3Y581ZBwARAQAB
tCVBbm5hIEZpZ3VlaXJlZG8gR29tZXMgPG5hdmlAdmxobC5kZXY+iQJXBBMBCgBB
AhsBBQsJCAcDBRUKCQgLBRYDAgEAAh4BAheAAhkBFiEEro7Xbs7MsnUO4Id2aJkC
kqepjF4FAmVekV8FCQPKSpEACgkQaJkCkqepjF6lYQ//XlYhsVyDgewwRIzGQ5M/
+j5mVeV/ZFKOuJB5jNZIbdXd8Ed+nRJ4NcHFUADCEUTzPFwjsqz/KcFdcxp4ykME
YTyvQQA4jQMsCzslwsf8mr/MtM1yaCyL+F8nFalEFqaBxBlH/xo/6oOxSJ0yVtZA
4FIFd4OVwBZAwo63yy2GCT3N0CT+aZZA+P5DIuS7UfIAyg2zzYvms1IalEw7qrRU
lmihZMJUMBLs/25bPJtiKN3csq9/kMYl0Hog0z4j175PIyEgViwMcAtcEwyPodR4
yJ4XhpyCHnd/CB6OAMV9u0hCV/cPi+M8pdnnuw1mgfHb6WMfnCYZwKsqqk2ocb+H
gfCqkMyYQcqsAB5GR2bZVKBcNuyDbiPQVjMcjIu+DF58hjib5xn3dOdWaWPtnYDw
6IX/5aLeDsqskpUMioMhXzbOXp9zKCKJjzriYls0Z2Iuve6CZRV/WZMZy1wvXHh+
RMkUZF3POlBWFASew2rSJOoEdOe2wF/SbeP8lZf+pwV+0HIh39q1GAg6m3FjFNKO
uYynPO8eLn3x32QYkkzElcQsLGfuHlnekyYDUXsYdP49McmXw6UVXthmH4bYynAt
wK8QEkK5+qdh/LIUb9CK1uGsL/hP+h+yGL/4nOBZ73KBv9bvFJzsLBkjOmEv7Cvk
oi+53/HVAEaBjT21MSDfTeC0LEFubmEgKG5hdmkpIEZpZ3VlaXJlZG8gR29tZXMg
PG5hdmlAdmxobC5kZXY+iQJUBBMBCgA+AhsBBQsJCAcDBRUKCQgLBRYDAgEAAh4B
AheAFiEEro7Xbs7MsnUO4Id2aJkCkqepjF4FAmVekV8FCQPKSpEACgkQaJkCkqep
jF4VAhAAmLG996Q6BvnVEeUQJv9MCfFLpw9ToaLgf3WbtX3phwmrVVmPwyOsJ80z
DwJRvbD6Z84fHJsJADJWGmn2Mw4BRYDjZxOaabenFxglPIj736Pt7chyMt9fXw8O
aA23mILs6A/z0MCebhY59tz9o7wxm7W6ruL1ta3w4nHD6q6aag3hM4ki3aI8+u2U
ECvwW+rv07TGvskX0FqJ8mkI/yO2J33rfpXaL48/QKx4+YfGkhLNlgSc+ISyEOXf
V8BdvKvhMtVru8r4XlBd3U2cLpyi72VypZfpLJHbsG8P3ECeFOZyRjgoBS38FQWj
EeVRzCHCovlE9fPK5ddKHrOew+29Iew4VgmRsPJ8HJtA6+/SL0jaCl9Vd2aOc1IV
qMflUg+V/sXrWC/mQqOAbnPGkcmp4PpZ9GciyloPXerEgLNZIIgVNFJPgdZzxBXc
Dx3asgE3hx7caZQl3MoBfUJtYzMOQu0hADw5xq7i/SBJA9Tao8Zy8HW6YzPmw6Oo
MCbQlIdlGKxGjVTkaf8jjWzrm7IpmVX04hR5HQ4RxmDgEIM/PDnMQPpaM4eluga8
FGDU4U4dXHpSVH0WK3zCzCpXG724UEGh/Z+XWecbvNbwZk/Low7bLY1ZBc6QhCEz
AMTwH7rPUSifaX6m4i956obj9fueiVPzp2M2ua0dAbgfRWbr7oG5AQ0EY3uDRwEI
ALQyOoro/DVGO5tErWuBrwpfcIO1O3ONbrUvQF3AVPnnTWZ64emiRpnESei8QKA4
tzgKJvDL7tQS72TkFYRhXtv9VbnUBMUXrBHRC2FpukL0XhOoAtWuM4gYm/VMzv+g
5zyPAm4v85jFJj8WolBC8p7Bm2OxVugoMXyHVqjQIa9yHyt5VI372Pi5G7OQ+p+s
NQlt7qfMkxIR1sa0aUG+J/PHkASasUr+72HJeUNb9Y86axRluaJsWnH1xGfCfUVP
Y5Wx9wTdt0vP5MnxSGD8Usq2PqzKMdXkzj7WAf0SOsUY/t6SkDrygXKr1ZOfRLo3
i5PEji9jpQlciig5lRtFWAcAEQEAAYkD0gQYAQoAJgIbAhYhBK6O127OzLJ1DuCH
dmiZApKnqYxeBQJlXpGzBQkDxEHsAaDA1CAEGQEKAH0WIQQ5BklG+RGbGoTdw5/B
1OnVrOhSmgUCY3uDR18UgAAAAAAuAChpc3N1ZXItZnByQG5vdGF0aW9ucy5vcGVu
cGdwLmZpZnRoaG9yc2VtYW4ubmV0MzkwNjQ5NDZGOTExOUIxQTg0RERDMzlGQzFE
NEU5RDVBQ0U4NTI5QQAKCRDB1OnVrOhSmmDvB/0fkBKSSffbZDlM4N+hS0Isa7X5
WUxLd5aRHwpCcxhwG+ltmsSGX/0tLkpbgaNJ2kBRCJLld3NUPc+0XKamVxQEWS+2
YgSruIayhuLuwB3KSPVFgSxNCTUXQs0n9ASijzvROY/M+uFHv4n9uiYRCfmczwvp
GvhuinaPn9caYaNPXyIqxU4A51e7Vo3GVF1yyIVGRN+tRBRBdP3xBCFF2a6U9tl7
siBCArGYMP52cBS7d+/WxihEAAQN55MNmGsWrVt/htGF8HjmkgPxOaGjBXMcmotu
UmF0RTRmbl+dIqgNcf6N3hv6yQqCRhjzPC00l41IBnY0jKPANsdZY8ev1L95CRBo
mQKSp6mMXh2fEACe5Wo5zJyLzvczNA5l6reFect8BmBDvEczrDZaRfMSVO0+MLXZ
ZpZy/DRtkvHojNPWW28oFJRCIR/six7qV5h3bFNilI16hwqUy3Qv1pjahxdWEa3J
bD92wD0U6GbkoTPWyyp6O90voxZlN2rFWd4/oiLo2HPewXDyqC+zmn1qe8OWXu1R
1WG09b4kyEHMya2558jcX2x9QvQ+YsqjrKXSz3LieFBdZSH8Tnckbrs7yB/Ol+JY
4bQTMdPzzB+QsJjpYG4XaKEg7MD15ox/+u7nkOEHNrnXx8g5DFWEK1NaQtQPdOWr
dpgtQy45AEr0rTTQn5SGr5TUxV0GId0hv9reXWp78gtE06Bp0+pyf4xTSmfuYiM5
eJYlxhNzv+ipjWLy2tCtlh85UIHMvTABv0aE4/3V1SeEjbsbaOB8+bX53jEdLQ7v
BJECeJw+eH11ZG4hRT5ZrEVmAXrgkbacASJEpU0R1JI+9Jmn0EkfrtXyVqTEcfm4
v8jGxNE6xBdrVO/BjYsOs75ndPf2wEgFdbdRyqpEieIl2Xl6+W8ydjM5aOxtMSBp
/yh+fj6m1E7kcxpKEBWrDBzpgG/fhEDUFfs6JM2ioz2Yf5AoByCVsHI9WESY1dqO
E/ILTkjlc9wvzlrQ789V3hH8TtsQbnjBuLAFlhlcPvC4qiJEwOw+LEeo1rkBDQRj
e4NzAQgA0PmJIGpxgb9jcUyq4VPvX1unCyFuM0qPE3+ZMCR3pUYbSDbRTNC9eeaz
UDsZpoH8U6HAQHl6lcepvpNzidoXlIqwG8mMTGqXruYAnCkDZpewuovqYGHNUZo2
GL+tWs4eWSVDcqkchOSdpdHVsh7Udnoj/hvZccFlmLptd92bWGsm95Azty17W67H
YfVfaBUWNf3YUnWMIVm1JUGJggUXtgGKv0kTdsVrYhC0fO9UHFNV4S25/J9hecdW
op++3+87rNyxXsyzPuct5MrRAo39Kv/3Y5xgwMk/oQNBHDr+ba+SorDGXuRjVsbF
asQ1qZh92RtMogJ4tbfwl6ZuxZWdRQARAQABiQI8BBgBCgAmAhsMFiEEro7Xbs7M
snUO4Id2aJkCkqepjF4FAmVekbQFCQPEQcAACgkQaJkCkqepjF5CEg//YVezsvhd
alD4AuDKSpz2qP1XtS1MVWVKFefbo3jxJxxlG+J/uVsTPSA807m7Rj1WfpXqtUUD
iKwe0ttTDLiDTgaeO8Pcl1k3hw+06+niyE+SevyOWoQKGkbR+DKZZrsbhJ6nQE3n
stDC9abwMvhqQB6Pap2ugRv3Xbrnlbz3YEl+jAvycL09QTM3I6ARQ5HgMqi/SUg6
BUlc/9UsH6Mp9pnvn547i0R9LZJ+i1dSySyUjfjEFmOyV5CGSEOOT1w8Iw832GCa
qWjmCP9WxhnxzrVuC/7Svb5zPhAN8Idn+vvbtaQHCqBfD0Rc+nUygdCMAWfcVQja
iNXO6h7pC3Uq2apRPEYngimdvtiRUvVQFw7yWmM1CO3NX2XtbXe0WwmOZcBdm5pQ
zOA9TMHE3iAxGf8A/S5DNogmntRQJbWzxoGNpOtTCY9mkxKCJ0Yr+vj/93rh8zNX
CVSregJLbUbSbRw3tN+W6Fvadzh2fyPbfHSoKS1bt2hT1vL3QmLsQNmYBrLr4cti
QmrZ59BVVbfcRC24n3ciJiK2dsplIWIEMbk6r3j9cnEJWwXpQ75UnyikL7w1+ogJ
du3ao3XS+HjhCGIYXlIgdOqNH7gRf65/rwxi0fV4Za/XfoGHr7IdEdaF2ZRKqO50
cAZ3vJE/w4nWzbfsAsCYHZpbyoCl4uGZDfO4MwRje4UtFgkrBgEEAdpHDwEBB0A8
WFk7WDImkdDnNO0UZWj5KOKH+1BQ4wqHZgqE15++cIkCPAQYAQoAJgIbIBYhBK6O
127OzLJ1DuCHdmiZApKnqYxeBQJlXpG0BQkDxEAGAAoJEGiZApKnqYxe4cYP/iLI
AshTbx6joJRpFrAvQCEaPwbTw4Vypv5VYVkGJMk/eMYUkqwpQ+sgAvF8R3Yy1g+t
a7hWSZ1r3AIbmfMjMECODoKynu6+p7L+9pJH+LQDKg7CWjCwyGO0jIus+iOZJIyU
QdqaTAjhwbb74JXMccOq7TlrZNBU78adAmbqVnnHnYV4CjeviIMy5PSYHhXRQBZJ
RQ5bI5zlKbyEbkSSXgu5TNI2c/guAPZzUga6DOMXwOmGKBOsSlwsaKXLvryVHgD2
pKCexXFedXBERoKwnCMhsjlH0CSs3SRItMIR9Ta03ZQmDcLXVUAVU8qT1nRLiDLs
7Y+2XqBKVk/5PLK9JvtG49aeFONQcEZXJLupfM4najl8CP6jqIHG/L0fc57aX7Va
x9L2dGP+FyNjLT5b6Vmege2NcJTok7KzEXOdzwQL4pE51ra75kn/Pv5bfYneqORA
KEtQwde3HnkAncJKYigA5h53gw0ZtJ85P04k3vh/dovAaUfPcZ1SFC7VcjkB5hM1
7qh0P+pjFyDjJX+vtzxJxm+Nps99nJqm3LDGcXT7ZELPtsPJ0ru1mFKJqvthNO9R
mpI2zQFhQKp1jE+1DXuwoCCXMGoENuCDpujTy3tOuRD4bhCFAVr3NGz9cnb8R/iY
AkD2/QF/XGfCohF35zCk/kResgbif4cZ+4f1wUB0
=wvX3
-----END PGP PUBLIC KEY BLOCK-----

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 638 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[orbea]

On Sat, 12 Oct 2024 10:12:56 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Hello,
> 
> Historically, all versions of CPython were slotted in a single
> package, i.e.:
> 
>   dev-lang/python:3.N
> 
> This approach has been causing a major annoyance for users -- due to
> Portage "greedy" upgrade behavior, any time a new Python version was
> keyworded, Portage insisted on installing it, even though user's
> selected targets did not request the specific version.  The
> potentially worst consequence of that would be random user scripts
> stopping to work, as they suddenly start using new Python, while all
> their dependencies are still installed per PYTHON_TARGETS.
> 
> Upstream has recently added freethreading support to CPython.  Since
> this support is not ABI compatible with the regular build, we need to
> introduce a separate target for it, and to package it separately.
> In the planned patchset, I've already put it as a separate package
> (dev- lang/python-freethreading), because otherwise Portage would
> insist on upgrading to it!
> 
> However, I think the cleanest way forward would be to stop slotting
> CPython like this, and instead have a separate package for each
> version, just like the vast majority of distributions do, i.e.:
> 
>   dev-lang/python3_N
> 
> This naturally means that only the specific version requested (e.g.
> via targets) would be installed, and no cross-slot autoupgrades would
> happen.  Ideally, I'd like to start doing that with Python 3.14 whose
> first alpha is expected next week.  Depending on how they handle
> freethreading, we'd end up having the first or both of:
> 
>   dev-lang/python3_14
>   dev-lang/python3_14t
> 
> (Alternatives: python-3_14, python-freethreading-3_14? Though I think
> following PYTHON_TARGETS is cleaner here.)
> 
> As a side notice, the existing versions would probably remain as-is
> until removal, since there's really no gain in splitting them, given
> we'd have to retain compatibility with existing depstrings.
> 
> Comments?
> 

Would you be willing to carry the LibreSSL patches in tree if you do
this? Its already a serious chore keeping the python ebuilds in sync
with ::gentoo and if you start to double the versions that will only be
more ridiculous.

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Andreas Sturmlechner]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

On Samstag, 12. Oktober 2024 15:52:45 MESZ orbea wrote:
> Its already a serious chore keeping the python ebuilds in sync
> with ::gentoo and if you start to double the versions that will only be
> more ridiculous.
Life choices.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 789 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Nowa Ammerlaan]

On 12/10/2024 10:12, Michał Górny wrote:
> This approach has been causing a major annoyance for users -- due to
> Portage "greedy" upgrade behavior, any time a new Python version was
> keyworded, Portage insisted on installing it, even though user's
> selected targets did not request the specific version.  The potentially
> worst consequence of that would be random user scripts stopping to work,
> as they suddenly start using new Python, while all their dependencies
> are still installed per PYTHON_TARGETS.

I don't understand the problem. Say we have some custom user script with 
#!/usr/bin/python, then this still goes through the python-exec 
mechanism. That config file is protected, so some new python version 
cannot suddenly replace the /usr/bin/python without explicit action from 
the system administrator. So in what scenario exactly does the current 
behavior cause breakage?

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1243 bytes --]

On Sat, 2024-10-12 at 10:12 +0200, Michał Górny wrote:
> This naturally means that only the specific version requested (e.g. via
> targets) would be installed, and no cross-slot autoupgrades would
> happen.  Ideally, I'd like to start doing that with Python 3.14 whose
> first alpha is expected next week.  Depending on how they handle
> freethreading, we'd end up having the first or both of:
> 
>   dev-lang/python3_14
>   dev-lang/python3_14t
> 
> (Alternatives: python-3_14, python-freethreading-3_14? Though I think
> following PYTHON_TARGETS is cleaner here.)
> 
> As a side notice, the existing versions would probably remain as-is
> until removal, since there's really no gain in splitting them, given
> we'd have to retain compatibility with existing depstrings.
> 
> Comments?

Given all the opposition, I retract this.  While this doesn't really
change anything per se, I get that you can't stand the idea that someone
wouldn't use slots for something where slot use doesn't really improve
anything, and in fact makes users' lives worse.  When someone complains
that Portage suddenly installs Python 3.13 freethreading or Python 3.14
on their systems, it's on you.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Mitchell Dorrell]

[-- Attachment #1: Type: text/plain, Size: 2994 bytes --]

On Sat, Oct 12, 2024, 14:03 Michał Górny <mgorny@gentoo.org> wrote:

> On Sat, 2024-10-12 at 10:12 +0200, Michał Górny wrote:
> > This naturally means that only the specific version requested (e.g. via
> > targets) would be installed, and no cross-slot autoupgrades would
> > happen.  Ideally, I'd like to start doing that with Python 3.14 whose
> > first alpha is expected next week.  Depending on how they handle
> > freethreading, we'd end up having the first or both of:
> >
> >   dev-lang/python3_14
> >   dev-lang/python3_14t
> >
> > (Alternatives: python-3_14, python-freethreading-3_14? Though I think
> > following PYTHON_TARGETS is cleaner here.)
> >
> > As a side notice, the existing versions would probably remain as-is
> > until removal, since there's really no gain in splitting them, given
> > we'd have to retain compatibility with existing depstrings.
> >
> > Comments?
>
> Given all the opposition, I retract this.


I hope my questions weren't interpreted as opposition. I'm mostly neutral,
perhaps leaning a bit in favor.

When someone complains that Portage suddenly installs Python 3.13
> freethreading or Python 3.14 on their systems, it's on you.
>
> --
> Best regards,
> Michał Górny


I like slots for saying "these are different versions of the same
fundamental package, which you might want to install simultaneously,
because dependent packages may or may not need a specific version.", so
yes, on paper I agree that slots feel correct for different Python
versions. However, if there's another aspect of compatibility
(freethreading vs. non-freethreading), then I understand that slots don't
handle that well, and neither do USE flags.

I would _love_ to have something in between slots and USE flags -- perhaps
call them VARIANT flags, where differing combinations of VARIANT flags can
be installed simultaneously, but without carrying any implication of one
VARIANT combination being "newer" than another. This would be fantastic for
multilib systems that want different USE flags for x86 and amd64 builds (my
64-bit software stack GREATLY inflates my otherwise-minimal 32-bit stack).
As another example, sci-chemistry/gromacs offers MPI support for clustering
capabilities, but this actually impairs performance when running on a
single machine, so there's good reason to desire installing both
simultaneously (the ebuild already handles simultaneous single-precision
and double-precision builds).

This would be a major change to Portage, however, and I imagine we still
need a short-term solution even if the VARIANT idea is embraced and
implemented rapidly.

Perhaps a pair of USE flags could control the compiling/installation of
freethreading and non-freethreading builds, with a REQUIRED_USE mandating
at least one of them? If not, I think separate packages might be the best
(interim) solution unless/until a new Portage feature is implemented to
handle it all more elegantly.

-MD

[-- Attachment #2: Type: text/html, Size: 3916 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Sam James]

Michał Górny <mgorny@gentoo.org> writes:

> Hello,
>
> Historically, all versions of CPython were slotted in a single package,
> i.e.:
>
>   dev-lang/python:3.N
>

I feel like this whole thing happened so fast I didn't have a chance to
comment properly. I understand you've retracted it but I'd like to add
some context and background and so on for future reference anyway.

> This approach has been causing a major annoyance for users -- due to
> Portage "greedy" upgrade behavior, any time a new Python version was
> keyworded, Portage insisted on installing it, even though user's
> selected targets did not request the specific version.  The potentially
> worst consequence of that would be random user scripts stopping to work,
> as they suddenly start using new Python, while all their dependencies
> are still installed per PYTHON_TARGETS.
>

This is bug #702806 which happens with python-any-r1 which has an
any-of dependency on dev-lang/python. That's why we don't see it with
e.g. Qt. It's a bit annoying but not terrible.

> Upstream has recently added freethreading support to CPython.  Since
> this support is not ABI compatible with the regular build, we need to
> introduce a separate target for it, and to package it separately.
> In the planned patchset, I've already put it as a separate package (dev-
> lang/python-freethreading), because otherwise Portage would insist
> on upgrading to it!
>

It wouldn't! See above.

It would, however, if we made it eligible for python-any-r1, but to be
honest, I think we should exclude the freethreaded build from that. It's
all risk (and/or downsides) with no real gain, as I don't expect a
whole-freethreaded system is going to be possible any time soon anyway.

> However, I think the cleanest way forward would be to stop slotting
> CPython like this, and instead have a separate package for each version,
> just like the vast majority of distributions do, i.e.:
>
>   dev-lang/python3_N
>

As others have noted, such a proposal needs specific arguments as to why
SLOTs aren't a good fit. I agree with you that they're not always a good
fit -- SQLite and libxml2 are good examples you gave downthread, but
the onus is on the one making the proposal.

Now, for Python, there's a few disadvantages:
* losing the ordering on PV for e.g. has_version (we could add a helper
in python-utils-r1 for this);

* losing the ability to consistently set package.use/package.env for all
Pythons, like always enabling PGO or tests;

* disruption to scripts which have reasonably assumed we'd always have a
dev-lang/python (we'd need to do something like we have planned for
pkgmoves, I think -- make Portage know about it and suggest alternatives
intelligently/rewrite it transparently when given as an argument).


> This naturally means that only the specific version requested (e.g. via
> targets) would be installed, and no cross-slot autoupgrades would
> happen.  Ideally, I'd like to start doing that with Python 3.14 whose
> first alpha is expected next week.  Depending on how they handle
> freethreading, we'd end up having the first or both of:
>
>   dev-lang/python3_14
>   dev-lang/python3_14t
>

It's worth noting that we *do* this for pypy, but we retain
dev-python/pypy3. I'm not a huge fan of it there but I know why we have
it -- so that one can test new versions of pypy in parallel even when
they supply the same implementation/version of the Python language.

> (Alternatives: python-3_14, python-freethreading-3_14? Though I think
> following PYTHON_TARGETS is cleaner here.)
>
> As a side notice, the existing versions would probably remain as-is
> until removal, since there's really no gain in splitting them, given
> we'd have to retain compatibility with existing depstrings.
>
> Comments?

thanks,
sam

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2772 bytes --]

On Mon, 2024-10-14 at 01:43 +0100, Sam James wrote:
> 
> > However, I think the cleanest way forward would be to stop slotting
> > CPython like this, and instead have a separate package for each version,
> > just like the vast majority of distributions do, i.e.:
> > 
> >   dev-lang/python3_N
> > 
> 
> As others have noted, such a proposal needs specific arguments as to why
> SLOTs aren't a good fit. I agree with you that they're not always a good
> fit -- SQLite and libxml2 are good examples you gave downthread, but
> the onus is on the one making the proposal.
> 
> Now, for Python, there's a few disadvantages:
> * losing the ordering on PV for e.g. has_version (we could add a helper
> in python-utils-r1 for this);

I don't get this.  You can't use has_version directly without specifying
the slot, because it's going to match different versions.  And there's
no real difference between specifying a slot and a different package
name.

Well, unless you mean doing a meaningless has_version match for the sake
of it.  Then, yes, unslotting fixes that -- i.e. removes that ability.

> * losing the ability to consistently set package.use/package.env for all
> Pythons, like always enabling PGO or tests;

We aren't losing it, you just need to repeat it.  Just like right now
you can apply these per-slot or restrict version ranges, so there's no
guarantee of consistency either.

> * disruption to scripts which have reasonably assumed we'd always have a
> dev-lang/python (we'd need to do something like we have planned for
> pkgmoves, I think -- make Portage know about it and suggest alternatives
> intelligently/rewrite it transparently when given as an argument).

Yes, this is a fair point, and the logic in pkgcheck is pretty horibble,
so I guess going for slotting just to avoid having to fix that
and deploy the fix makes sense.

> > This naturally means that only the specific version requested (e.g. via
> > targets) would be installed, and no cross-slot autoupgrades would
> > happen.  Ideally, I'd like to start doing that with Python 3.14 whose
> > first alpha is expected next week.  Depending on how they handle
> > freethreading, we'd end up having the first or both of:
> > 
> >   dev-lang/python3_14
> >   dev-lang/python3_14t
> > 
> 
> It's worth noting that we *do* this for pypy, but we retain
> dev-python/pypy3. I'm not a huge fan of it there but I know why we have
> it -- so that one can test new versions of pypy in parallel even when
> they supply the same implementation/version of the Python language.

Technically, we could merge PyPy into a single package, as long as we
use verisons such as 2.7.7.3.17:2.7, 3.10.7.3.17:3.10, etc.


-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] [RFC] Splitting dev-lang/python into per-slot packages, starting with 3.14

[Sam James]

Michał Górny <mgorny@gentoo.org> writes:

> On Mon, 2024-10-14 at 01:43 +0100, Sam James wrote:
>> 
>> > However, I think the cleanest way forward would be to stop slotting
>> > CPython like this, and instead have a separate package for each version,
>> > just like the vast majority of distributions do, i.e.:
>> > 
>> >   dev-lang/python3_N
>> > 
>> 
>> As others have noted, such a proposal needs specific arguments as to why
>> SLOTs aren't a good fit. I agree with you that they're not always a good
>> fit -- SQLite and libxml2 are good examples you gave downthread, but
>> the onus is on the one making the proposal.
>> 
>> Now, for Python, there's a few disadvantages:
>> * losing the ordering on PV for e.g. has_version (we could add a helper
>> in python-utils-r1 for this);
>
> I don't get this.  You can't use has_version directly without specifying
> the slot, because it's going to match different versions.  And there's
> no real difference between specifying a slot and a different package
> name.
>
> Well, unless you mean doing a meaningless has_version match for the sake
> of it.  Then, yes, unslotting fixes that -- i.e. removes that ability.

Not for the sake of it, I was thinking of:

    if has_version >=dev-lang/python-3.10 ; then
        # Skip bogus tests relying on legacy behaviour
    fi

but of course, that doesn't work for pypy, which I forgot about ;)

>
>> * losing the ability to consistently set package.use/package.env for all
>> Pythons, like always enabling PGO or tests;
>
> We aren't losing it, you just need to repeat it.  Just like right now
> you can apply these per-slot or restrict version ranges, so there's no
> guarantee of consistency either.

You can't wildcard on it, though, so you have to explicitly list it
for all Pythons. I'm not sure what you mean by the consistency point.

You can restrict it right now if you want to via slot or version ranges,
but we have no way of doing a wildcard on package name?

(I actually think we could do with wildcard matching on version at
least in /etc/portage but finding some syntax which is free for it isn't
easy. It would be useful for e.g. masking .0 kernels.)

>
>> * disruption to scripts which have reasonably assumed we'd always have a
>> dev-lang/python (we'd need to do something like we have planned for
>> pkgmoves, I think -- make Portage know about it and suggest alternatives
>> intelligently/rewrite it transparently when given as an argument).
>
> Yes, this is a fair point, and the logic in pkgcheck is pretty horibble,
> so I guess going for slotting just to avoid having to fix that
> and deploy the fix makes sense.
>
>> > This naturally means that only the specific version requested (e.g. via
>> > targets) would be installed, and no cross-slot autoupgrades would
>> > happen.  Ideally, I'd like to start doing that with Python 3.14 whose
>> > first alpha is expected next week.  Depending on how they handle
>> > freethreading, we'd end up having the first or both of:
>> > 
>> >   dev-lang/python3_14
>> >   dev-lang/python3_14t
>> > 
>> 
>> It's worth noting that we *do* this for pypy, but we retain
>> dev-python/pypy3. I'm not a huge fan of it there but I know why we have
>> it -- so that one can test new versions of pypy in parallel even when
>> they supply the same implementation/version of the Python language.
>
> Technically, we could merge PyPy into a single package, as long as we
> use verisons such as 2.7.7.3.17:2.7, 3.10.7.3.17:3.10, etc.

Ah, good point.