Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Michał Górny]

On Fri, 2 Nov 2012 20:35:38 -0700
Brian Harring <ferringb@×××××.com> wrote:

> On Fri, Nov 02, 2012 at 10:03:05AM +0100, Micha?? G??rny wrote:
> > On Thu, 1 Nov 2012 15:42:42 -0700
> > Brian Harring <ferringb@×××××.com> wrote:
> >
> > > That's a shebang based version of what I proposed a while back.  It
> > > works now and has tests.  It's written to be basically a drop in for
> > > existing python.eclass generation of shebangs, w/ the
> > > hardlinking/de-duplication/farking-fast/fix python3.2
> > > /usr/bin/sphinx-build scenario.
> >
> > Could you please provide an explanation of what it does? I mean, a few
> > points of what happens when you run it in various scenarios. That's 600
> > lines of C code, I think Python folks would appreciate not having to
> > read that line by line.
>
> Grumble.  Don't think you read my commit messages at all, just
> responded looking for points to claim "it sucks".

Unless I'm missing something, the commit message says how to use it
and what are its advantages. But thanks for the list, that's what I was
asking for.

> What I have written here is a python-shebang based solution.  That
> shebang target includes the EPYTHON's it supports.
>
> For the usual sphinx-build example, sphinx-build-2.7 goes from
> #!/usr/bin/python2.7
>
> to
>
> #!/usr/bin/python-shebang python2.7,python3.2
>
> This means that sphinx-build-2.7 and sphinx-build-3.2 are now the same
> content.  They can be hardlinked together (space savings).
>
> This additionally means that sphinx-build can be hardlinked to the
> same underlying inode.  Why?  Because python-shebang is smart enough
> to recognize the context it is invoked in.
>
> This additionally means that the follow commands:
>
> eselect python set 3.2
> python2.7 /usr/bin/sphinx-build
>
> properly work- invoking sphinx-bulid in a 2.7 context (for anyone
> trying to write distutils/setup bits that aren't gentoo/EPYTHON
> specific, this is fricking useful to say the least).

I think you've missed my question: what code does the 'sphinx-build'
contain, except for the shebang? Is that a Python version
of the wrapper? Some kind of os.exec()?

> > > Frankly, you should be looking at this imo, rather than trying
> > > standalone files.  Yes, files is simpler- but you'll wind up sooner or
> > > later rebuilding it into what I already built out here.
> >
> > I think you are missing the point I'm raising here. Mostly because your
> > python-wrapper doesn't handle per-version preferences like python-exec
> > does, unless I'm missing something.
>
> You're misunderstanding what was written; hell, I even referenced
> that it supports EPYTHON preference in #5 (java-config cycle
> breaking).

EPYTHON preference is rarely used by users. Users are using
eselect-python. And eselect-python uses two levels of preferences:

1. 'Main' python interpreter preference,

2. Per-Python2/Python3 interpreter preference.

With the former being stored in a file, the latter in symlinks.
The goal of my patches were to store both in files so that the symlink
reading code could be removed to make the code simpler.

> > > """
> > > Add a python-shebang utility, slave python-wrapper to it.
> > >
> > > Going forward, the intent is to push the PYTHON_ABIs of a given
> > > script down into it's shebang, pointing the shebang at python-shebang.
> > >
> > > In this way, all known/supported abi's are available at the time of
> > > execution; further, if the target is told to respect-EPYTHON (meaning
> > > no searching for something to execute, either active python version or
> > > EPYTHON var), we can push this down into that list and have that code
> > > handle it.
> > >
> > > [...]
> > >
> > > to the simpler form of:
> > > for x in sphinx{,-{2.7,3.2}}; do echo > $x <<EOF
> > > #!/usr/bin/python-shebang python2.7,python3.2
> > > <code>
> > > EOF
> > > done
> >
> > Hmm, are you assuming that /usr/bin/sphinx carries the actual code?
> > Or some special wrapper code?
> >
> > Not to mention you still bind the whole thing to installed package.
> > If you need to fix those files, you need to reinstall all packages
> > installing them.
>
> This critique makes no sense... and I'm pretty sure that any point you
> may manage to come up with, will apply directly to python-exec just as
> equally.
>
> Either clarify your claim, or drop the noise there.

python-exec can't be executed like that currently, correct. By I can
make it so just by replacing the python-exec executable as the symlinks
will make it effectively to every single package.

In your case, any change won't apply globally.

> > > The gains of this are thus:
> >
> > Could you please compare it to the modern solution (python-exec) rather
> > than the deprecated one? You are no longer on the uncharted waters.
>
> Respectfully, your 'modern solution' (vs deprecated) is purely in your
> head; distutils/python eclasses of your making aren't the tree.
>
> They're your playground right now- one that you're pushing, but I've
> yet to see real traction on.  That's just reality; you may think it's
> a done deal (or will be), but the on the ground reality right now-
> including the pushback you've been getting from people in certain
> cases- makes me unwilling to pin my hopes on your work.

It is just a difference in methods. You want to push your solution to
the whole tree at once and see what happens next. I rather prefer
testing it on new ebuilds first, then getting it stable and thinking
about replacing the old one when the new one is ready.

> Either way, you asked, so here's the list of python-exec con's in
> comparison to python-shebang:
>
> 1) Your solution actually doesn't work as a fricking wrapper for a
> core usage case; `python /usr/bin/sphinx-build`.  That perfectly
> innocuous/valid invocation would result in the interpretter trying
> to execute an elf binary.  That pretty much renders your solution as a
> no go right from the get go.  Your only way to fix that is to have
> sphinx-build be a python implementation that does wrapping, and use a
> shebang of python-exec (so you can maintain speed for direct
> /usr/bin/sphinx-build invocations); meaning once you spotted that
> flaw, you'd evolve your attempt into something nearing what I wrote.

No, not really. I still am afraid that you're missing some point of how
Python interpreter works but considering the fight you put protecting
'python /usr/bin/c-file', I won't continue that topic until you finally
answer my questions wrt that file.

By the way, I doubt it is a 'core usage case'. Maybe just because it
never worked as intended, maybe because people simply don't do that
usually because people usually don't care whether stuff in /usr/bin
is Python, C or shell.

> 2) Everytime there is a new python interpretter, python-exec has to
> be re-built updating it's hardcoded list of interpretters.  Goes without
> saying, this sucks a fair bit; your solution there likely would be
> walk env.d/python/* in some fashion extracting the list of targets (or
> walk /usr/bin/python* and run into problems w/ the names used there
> now matching EPYTHON usage).  End result, you'd evolve it into
> something nearing what I did w/ shebangs.

How often does this happen? Is this really problematic?

And how an user can disable a particular (say, problematic) Python
implementation globally in your solution? Without reinstalling all
the packages which provide scripts for it?

> 3) It's a hardcoded lookup list that is forced for all scripts; if
> that list is python3.2,python3.1,python2.7, and the active python
> version doesn't intersect that (say jython-2.5), your approach forces
> 3.2.  My solution actually allows the script, in the absense of an
> enforcement via active python or EPYTHON, to use it's own preferred
> ordering.  Using pkgcore as an example, we prefer py2k execution over
> py3k- py2k is faster since the extensions are enabled.  This
> capability doesn't exist in your solution, nor the existing wrapper;
> it's a useful byproduct of how my solution is structured.  Etc etc,
> you know the response; where you'll go with this sooner or later is
> where python-shebang is already.

And, hmm, are you the first person requesting such a feature over
the whole lifetime of python-wrapper?

I believe that hardcoded list is *simpler*. If user enabled jython-2.5
(suicidal type?), then he's aware of the consequences. And in my
solution, preferred Python 2 will be used. Then preferred Python 3.
Then Python 2.7, 2.6, 2.5... then 3.2. It is predictable.

In your solution, it is completely unpredictable. Developer can
(supposedly) do whatever he wants. Most of the devs probably won't use
that, except for a few cases where users will simply hit (if ever)
an unexpected behavior.

> 5) Via going with a standalone package from eselect-python, you're
> introducing the potential for the two to be out of sync/incompatible.
> Addressable via folding it inline into eselect-python (as I did), but
> it's a con against your solution in it's current claimed form.  Same
> thing, you'll do what I did sooner or later since it's the better
> approach.

It's a better development practice to split things. You should've known
that by now. In any case, python-exec can work *without*
eselect-python. You can try it if you don't believe.

> 7) My solution doesn't hardcode /usr/bin/* pathways.  User sticks a
> python2.7 into /usr/local/ (for whatever reason they choose to do so),
> it will honor that.

Errr? Where does your suggestion come from?

As far as I'm aware, user can put into per-implementation copy whatever
he wants. Including a shell script, a C executable or whatever. It will
work. With your, it won't.

As far as I'm aware, you solution can't handle per-implementation
Python options (command-line arguments) either. In mine, it's as simple
as dropping them onto the shebang.

> 8) Your solution doesn't work in the context of being used for
> /usr/bin/python wrapper; it's close, but as you yourself said, it
> requires symlinks (python-python2.7).

Or adding a conditional which your solution simply does. So, where are
you going with this point? To the fact that I'm asking others for
opinion while you're ignoring their existence?

> 9) No tests.  Every. Fucking. Pathway. is tested with what I wrote.
> Python-exec lacks that, else issue #1 would've been spotted up front.

Great. Awesome. You are so awesome I can't even spell it.

Now feel free to grep the bugzie. That 'issue #1' is known. So far,
I haven't get anyone to say he believes that's really important
or useful. 

> The only flaws with it at this point are thus:
>
> 1) There is a 127 char limit on shebang length.  This is addressable
> via having python-shebang reach grab the list from the target itself
> if that limit is encountered.
>
> 2) fleshing out the tools for doing de-duplication; it's easy enough,
> I've just been busy with other things, and this code was done as a
> side project for entertainment.  I'll finish it soon enough (else if
> someone wants to sort the remaining issues, I can detail exactly how).
>
>
> The nice part about my solution here is that it's not something that
> is hinged mgorny's python/distutils eclasses becoming the norm; it's
> usable now for existing python eclass (simple enough to plug it in),
> and any followup solution would be insane not to use it.

Did you ever have a bit of modesty? I think this is what gets me so mad
at you. Every thing you do is so perfect and bug-free everyone else
should be insane not to start using it immediately. Except when it
doesn't work at all but then others are just 'getting you pissy'.

-- 
Best regards,
Michał Górny