Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Brian Harring]

On Fri, Nov 02, 2012 at 10:03:05AM +0100, Micha?? G??rny wrote:
> On Thu, 1 Nov 2012 15:42:42 -0700
> Brian Harring <ferringb@×××××.com> wrote:
>
> > On Thu, Nov 01, 2012 at 10:27:19PM +0100, Micha?? G??rny wrote:
> > > On Thu, 1 Nov 2012 14:48:55 -0400
> > > Mike Gilbert <floppym@g.o> wrote:
> > >
> > > > On Thu, Nov 1, 2012 at 11:44 AM, Mike Gilbert <floppymaster@×××××.com> wrote:
> > > > > On Thu, Nov 1, 2012 at 7:54 AM, Micha?? G??rny <mgorny@g.o> wrote:
> > > > >> The idea is very simple: /etc/env.d/python/python[23] with a one-line
> > > > >> value similar to the main interpreter /config file.
> > > > >>
> > > > >> That should be simpler & more reliable than reading a symlink. And at
> > > > >> some point we can replace the symlink with an $EPYTHON-aware wrapper
> > > > >> as well.
> > > > >
> > > > > I don't understand the point of this. Do we have some need to enable
> > > > > EPYTHON usages for scripts that have a python2 or python3 shebang?
> > > > >
> > > > > I also don't understand how a text file is more reliable than a
> > > > > symlink; they are basically the same thing, but the symlink has a
> > > > > different file mode.
> > >
> > > Ah, and I'd forget. I have the following dream:
> > >
> > >   /etc/env.d/python/python2
> > >   /etc/env.d/python/python3
> > >   /etc/env.d/python/config -> python2
> > >
> > > So, python{2,3} keep the per-version interpreters, and config is just
> > > a symlink to one of them. Two bird with one stone -- readlink to get
> > > which group is enabled, cat to get the exact interpreter. I'm proud
> > > of myself!
> >
> > I too have a dream; git://pkgcore.org/eslect-python .
> >
> > That's a shebang based version of what I proposed a while back.  It
> > works now and has tests.  It's written to be basically a drop in for
> > existing python.eclass generation of shebangs, w/ the
> > hardlinking/de-duplication/farking-fast/fix python3.2
> > /usr/bin/sphinx-build scenario.
>
> Could you please provide an explanation of what it does? I mean, a few
> points of what happens when you run it in various scenarios. That's 600
> lines of C code, I think Python folks would appreciate not having to
> read that line by line.

Grumble.  Don't think you read my commit messages at all, just 
responded looking for points to claim "it sucks".

What I have written here is a python-shebang based solution.  That 
shebang target includes the EPYTHON's it supports.

For the usual sphinx-build example, sphinx-build-2.7 goes from
#!/usr/bin/python2.7

to

#!/usr/bin/python-shebang python2.7,python3.2

This means that sphinx-build-2.7 and sphinx-build-3.2 are now the same 
content.  They can be hardlinked together (space savings).

This additionally means that sphinx-build can be hardlinked to the 
same underlying inode.  Why?  Because python-shebang is smart enough 
to recognize the context it is invoked in.

This additionally means that the follow commands:

eselect python set 3.2
python2.7 /usr/bin/sphinx-build

properly work- invoking sphinx-bulid in a 2.7 context (for anyone 
trying to write distutils/setup bits that aren't gentoo/EPYTHON 
specific, this is fricking useful to say the least).

python-shebang does thus:

1) parse argv[1]; split it first on ' ' (everything following is split 
by ' ', and used as args for the python interp invoked).  The leftmost
is split on ','; this is the list of interpretter targets this script 
was installed for.

2) identify the context it was invoked via.  If invoked via an interp 
specific pathway (aka, /usr/bin/sphinx-build-2.7; a forced python2.7 
context) then it honors that, and re-execs to python2.7 $0 w/ the 
additional interp args.  That re-exec fail, it does the usual failure 
return.

3) Making it here means it was invoked in a non forced interp context; 
meaning /usr/bin/sphinx-build.  If the installation required EPYTHON 
to be honored always (this actually is used if you analyzed the tree), 
w/in the interp targets there is a EPYTHON value- literally, 
"EPYTHON".  If EPYTHON env var is set, and EPYTHON is set, force that 
interpretter or fail.

3) If EPYTHON is set but the script wasn't marked as "honor EPYTHON 
always", check the list of interp targets, if we can the EPYTHON 
target, re-exec to it.

4) Making it here means that env EPYTHON wasn't forced, nor can it be 
honored.  Grab the active python version, re-exec to that if it's a 
support interpretter for this script.

5) If it ain't, loop over the list of supported targets, l->r, trying 
to re-exec in that ordering.


> > The remaining work for that is thus:
> >
> > 1) If the EPYTHON targets aren't given via shebang arg, then it needs
> >  to fallback to grabbing the targets from the file (easy enough).
> >
> > 2) Add a few helpers/wrappers to make it easier to do the
> > deduplication/shebang rewriting.
>
> How would you perform deduplication while you're at it? Could you
> measure or at least consider the benefits of it?

I described this in the original dev ml thread you responded to 
bitching about symlinks; rewrite the shebang, md5 comparison, what's 
the same is hardlinked; if all are the same, hardlink the wrapper 
(meaning if sphinx-build-{2.7,3.2} are the same md5, hardlink 
sphinx-build to the same underlying inode).

It's pretty simple.


> > Frankly, you should be looking at this imo, rather than trying
> > standalone files.  Yes, files is simpler- but you'll wind up sooner or
> > later rebuilding it into what I already built out here.
>
> I think you are missing the point I'm raising here. Mostly because your
> python-wrapper doesn't handle per-version preferences like python-exec
> does, unless I'm missing something.

You're misunderstanding what was written; hell, I even referenced 
that it supports EPYTHON preference in #5 (java-config cycle 
breaking).


> > """
> > Add a python-shebang utility, slave python-wrapper to it.
> >
> > Going forward, the intent is to push the PYTHON_ABIs of a given
> > script down into it's shebang, pointing the shebang at python-shebang.
> >
> > In this way, all known/supported abi's are available at the time of
> > execution; further, if the target is told to respect-EPYTHON (meaning
> > no searching for something to execute, either active python version or
> > EPYTHON var), we can push this down into that list and have that code
> > handle it.
> >
> > [...]
> >
> > to the simpler form of:
> > for x in sphinx{,-{2.7,3.2}}; do echo > $x <<EOF
> > #!/usr/bin/python-shebang python2.7,python3.2
> > <code>
> > EOF
> > done
>
> Hmm, are you assuming that /usr/bin/sphinx carries the actual code?
> Or some special wrapper code?
>
> Not to mention you still bind the whole thing to installed package.
> If you need to fix those files, you need to reinstall all packages
> installing them.

This critique makes no sense... and I'm pretty sure that any point you 
may manage to come up with, will apply directly to python-exec just as 
equally.

Either clarify your claim, or drop the noise there.


> > The gains of this are thus:
>
> Could you please compare it to the modern solution (python-exec) rather
> than the deprecated one? You are no longer on the uncharted waters.

Respectfully, your 'modern solution' (vs deprecated) is purely in your 
head; distutils/python eclasses of your making aren't the tree.  

They're your playground right now- one that you're pushing, but I've 
yet to see real traction on.  That's just reality; you may think it's 
a done deal (or will be), but the on the ground reality right now- 
including the pushback you've been getting from people in certain 
cases- makes me unwilling to pin my hopes on your work.


Either way, you asked, so here's the list of python-exec con's in 
comparison to python-shebang:

1) Your solution actually doesn't work as a fricking wrapper for a 
core usage case; `python /usr/bin/sphinx-build`.  That perfectly 
innocuous/valid invocation would result in the interpretter trying 
to execute an elf binary.  That pretty much renders your solution as a 
no go right from the get go.  Your only way to fix that is to have 
sphinx-build be a python implementation that does wrapping, and use a 
shebang of python-exec (so you can maintain speed for direct 
/usr/bin/sphinx-build invocations); meaning once you spotted that 
flaw, you'd evolve your attempt into something nearing what I wrote.

2) Everytime there is a new python interpretter, python-exec has to 
be re-built updating it's hardcoded list of interpretters.  Goes without 
saying, this sucks a fair bit; your solution there likely would be 
walk env.d/python/* in some fashion extracting the list of targets (or 
walk /usr/bin/python* and run into problems w/ the names used there 
now matching EPYTHON usage).  End result, you'd evolve it into 
something nearing what I did w/ shebangs.

3) It's a hardcoded lookup list that is forced for all scripts; if 
that list is python3.2,python3.1,python2.7, and the active python 
version doesn't intersect that (say jython-2.5), your approach forces 
3.2.  My solution actually allows the script, in the absense of an 
enforcement via active python or EPYTHON, to use it's own preferred 
ordering.  Using pkgcore as an example, we prefer py2k execution over 
py3k- py2k is faster since the extensions are enabled.  This 
capability doesn't exist in your solution, nor the existing wrapper; 
it's a useful byproduct of how my solution is structured.  Etc etc, 
you know the response; where you'll go with this sooner or later is 
where python-shebang is already.

5) Via going with a standalone package from eselect-python, you're 
introducing the potential for the two to be out of sync/incompatible.  
Addressable via folding it inline into eselect-python (as I did), but 
it's a con against your solution in it's current claimed form.  Same 
thing, you'll do what I did sooner or later since it's the better 
approach.

6) With N=# of python implementations, you're forcing N copies of that 
script.  My solution will de-dupe it down to where the content is the 
same; most scripts, that means all script-* and script collapses down 
to one single inode.  If py3k content differs from py2k, and it's 
installed for py2k/py3k, this means in that case 3 inodes.  Your 
solution has no way to avoid the inode/space overhead, not w/out 
switching the shebang.

7) My solution doesn't hardcode /usr/bin/* pathways.  User sticks a 
python2.7 into /usr/local/ (for whatever reason they choose to do so), 
it will honor that.

8) Your solution doesn't work in the context of being used for 
/usr/bin/python wrapper; it's close, but as you yourself said, it 
requires symlinks (python-python2.7).

9) No tests.  Every. Fucking. Pathway. is tested with what I wrote.  
Python-exec lacks that, else issue #1 would've been spotted up front.


What you've got only works for direct invocation of a target via 
/usr/bin/sphinx-build; python /usr/bin/sphinx-build won't fly; nor can 
it be used as the the python wrapper w/out changing how we install the 
interpretter.


Python shebang however works all existing scenarios, and fixes the 
core scenario that got me started here; `python2.7 
/usr/bin/sphinx-build`.

The only flaws with it at this point are thus:

1) There is a 127 char limit on shebang length.  This is addressable 
via having python-shebang reach grab the list from the target itself 
if that limit is encountered.

2) fleshing out the tools for doing de-duplication; it's easy enough, 
I've just been busy with other things, and this code was done as a 
side project for entertainment.  I'll finish it soon enough (else if 
someone wants to sort the remaining issues, I can detail exactly how).


The nice part about my solution here is that it's not something that 
is hinged mgorny's python/distutils eclasses becoming the norm; it's 
usable now for existing python eclass (simple enough to plug it in), 
and any followup solution would be insane not to use it.

~harring