Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Brian Harring]

On Sat, Nov 03, 2012 at 08:33:46AM +0100, Micha?? G??rny wrote:
> On Fri, 2 Nov 2012 20:35:38 -0700
> Brian Harring <ferringb@×××××.com> wrote:
>
> > On Fri, Nov 02, 2012 at 10:03:05AM +0100, Micha?? G??rny wrote:
> > > On Thu, 1 Nov 2012 15:42:42 -0700
> > > Brian Harring <ferringb@×××××.com> wrote:
> > What I have written here is a python-shebang based solution.  That
> > shebang target includes the EPYTHON's it supports.
> >
> > For the usual sphinx-build example, sphinx-build-2.7 goes from
> > #!/usr/bin/python2.7
> >
> > to
> >
> > #!/usr/bin/python-shebang python2.7,python3.2
> >
> > This means that sphinx-build-2.7 and sphinx-build-3.2 are now the same
> > content.  They can be hardlinked together (space savings).
> >
> > This additionally means that sphinx-build can be hardlinked to the
> > same underlying inode.  Why?  Because python-shebang is smart enough
> > to recognize the context it is invoked in.
> >
> > This additionally means that the follow commands:
> >
> > eselect python set 3.2
> > python2.7 /usr/bin/sphinx-build
> >
> > properly work- invoking sphinx-bulid in a 2.7 context (for anyone
> > trying to write distutils/setup bits that aren't gentoo/EPYTHON
> > specific, this is fricking useful to say the least).
>
> I think you've missed my question: what code does the 'sphinx-build'
> contain, except for the shebang? Is that a Python version
> of the wrapper? Some kind of os.exec()?

Covered this in the email.

As I said, if dedup'ing <script>-* results in the same 
content... then the wrapper as I said, is a hardlink to it.  It's 
literally the script itself, no point in needing indirection since a 
$INTERP /usr/bin/sphinx-build would do the correct thing.

If the content varies across the python interp specific scripts, then 
a simple wrapper transfers control back to shebang, passing on the 
forced interpretter.  That wrapper is generated via thus:


abi_list=python2.7,python3.2,etc,you,get,the,jist

cat > wherever-to-write-the-wrapper << EOF
#!$(type -P python-shebang) ${abi_list}${interpreter_args:+ ${interpreter_args}}
import os, sys
os.environ["EPYTHON"]=os.path.basename(sys.executable)
targets="${abi_list}"
if hasattr(os, "execvp"):
        os.execvp("python-shebang", [sys.argv[0], targets] + sys.argv)
import subprocess
sys.exit(subprocess.Popen(["python-shebang", targets] + sys.argv).wait())
EOF


> > > > Frankly, you should be looking at this imo, rather than trying
> > > > standalone files.  Yes, files is simpler- but you'll wind up sooner or
> > > > later rebuilding it into what I already built out here.
> > >
> > > I think you are missing the point I'm raising here. Mostly because your
> > > python-wrapper doesn't handle per-version preferences like python-exec
> > > does, unless I'm missing something.
> >
> > You're misunderstanding what was written; hell, I even referenced
> > that it supports EPYTHON preference in #5 (java-config cycle
> > breaking).
>
> EPYTHON preference is rarely used by users. Users are using
> eselect-python. And eselect-python uses two levels of preferences:
>
> 1. 'Main' python interpreter preference,
>
> 2. Per-Python2/Python3 interpreter preference.
>
> With the former being stored in a file, the latter in symlinks.
> The goal of my patches were to store both in files so that the symlink
> reading code could be removed to make the code simpler.

I've yet to see a real world justification for #2 in terms of 
it actually being desirable.

I've not supported #2 in python-shebang; that said if the python team 
(as a whole, not just mgorny yelling I didn't add it) wants it, it's 
quick to add the support and requisite tests.  See my comments further 
down however, I strongly think such control is incorrect.


> > > > """
> > > > Add a python-shebang utility, slave python-wrapper to it.
> > > >
> > > > Going forward, the intent is to push the PYTHON_ABIs of a given
> > > > script down into it's shebang, pointing the shebang at python-shebang.
> > > >
> > > > In this way, all known/supported abi's are available at the time of
> > > > execution; further, if the target is told to respect-EPYTHON (meaning
> > > > no searching for something to execute, either active python version or
> > > > EPYTHON var), we can push this down into that list and have that code
> > > > handle it.
> > > >
> > > > [...]
> > > >
> > > > to the simpler form of:
> > > > for x in sphinx{,-{2.7,3.2}}; do echo > $x <<EOF
> > > > #!/usr/bin/python-shebang python2.7,python3.2
> > > > <code>
> > > > EOF
> > > > done
> > >
> > > Hmm, are you assuming that /usr/bin/sphinx carries the actual code?
> > > Or some special wrapper code?
> > >
> > > Not to mention you still bind the whole thing to installed package.
> > > If you need to fix those files, you need to reinstall all packages
> > > installing them.
> >
> > This critique makes no sense... and I'm pretty sure that any point you
> > may manage to come up with, will apply directly to python-exec just as
> > equally.
> >
> > Either clarify your claim, or drop the noise there.
>
> python-exec can't be executed like that currently, correct. By I can
> make it so just by replacing the python-exec executable as the symlinks
> will make it effectively to every single package.
>
> In your case, any change won't apply globally.

This isn't much of a claimed con; yes, there is an API contract here.  

That's specifically that python-shebang is invoked w/ minimally two 
arguments; the first being the EPYTHON's for the intended target; the 
second being the intended target.

That's it.  Nothing more.  As for "having to repair it"- were it 
required, it's easy enough to spot 'em (loop over vdb contents 
identifying files with /usr/bin/python-shebang as the shebang); that 
said, it's never going to be needed.


> > > > The gains of this are thus:
> > >
> > > Could you please compare it to the modern solution (python-exec) rather
> > > than the deprecated one? You are no longer on the uncharted waters.
> >
> > Respectfully, your 'modern solution' (vs deprecated) is purely in your
> > head; distutils/python eclasses of your making aren't the tree.
> >
> > They're your playground right now- one that you're pushing, but I've
> > yet to see real traction on.  That's just reality; you may think it's
> > a done deal (or will be), but the on the ground reality right now-
> > including the pushback you've been getting from people in certain
> > cases- makes me unwilling to pin my hopes on your work.
>
> It is just a difference in methods. You want to push your solution to
> the whole tree at once and see what happens next. I rather prefer
> testing it on new ebuilds first, then getting it stable and thinking
> about replacing the old one when the new one is ready.

Mgorny, quit it with the insinuations that I'm not paying attention to 
QA.

My solution doesn't break existing deployment, has tests, and 
actually works; yours breaks real world usage already and requires 
arbitrary renaming of scripts... which can induce it's own breakage 
(any script that know to find a python2.7 version of a script at 
<script-name>-2.7 would break).

Minimally, any solution pushed is going to require QA; this includes 
validating the implementation (already got that covered), and full 
scale testing (which I've already been doing locally as initial 
steps).

Find it funny you claim I'm just trying to jam it into the tree; 
deployment of this wasn't mentioned since I've not finished it in full 
yet.

Beyond local testing and tinderbox runs, the rough deployment plan 
in my head is thus; it's fairly cautious additionally:

1) add a use flag (defaulted off) that turns on the new implementation
2) Nudge adventurous folk to flip it on and get more testing
3) Minimally get prefix involved since if there are issues, it'll come 
from them; get them converted over.
4) Bug rate hits zero, drop the flag and make it the new norm for 
python.eclass and your 8 ebuilds.


> > Either way, you asked, so here's the list of python-exec con's in
> > comparison to python-shebang:
> >
> > 1) Your solution actually doesn't work as a fricking wrapper for a
> > core usage case; `python /usr/bin/sphinx-build`.  That perfectly
> > innocuous/valid invocation would result in the interpretter trying
> > to execute an elf binary.  That pretty much renders your solution as a
> > no go right from the get go.  Your only way to fix that is to have
> > sphinx-build be a python implementation that does wrapping, and use a
> > shebang of python-exec (so you can maintain speed for direct
> > /usr/bin/sphinx-build invocations); meaning once you spotted that
> > flaw, you'd evolve your attempt into something nearing what I wrote.
>
> No, not really. I still am afraid that you're missing some point of how
> Python interpreter works but considering the fight you put protecting
> 'python /usr/bin/c-file', I won't continue that topic until you finally
> answer my questions wrt that file.

Either your english sucks, or you're not understanding your proffered 
solution doesn't work... or you're not going to acknowledge it 
doesn't actually work.  Honestly not sure which considering the 
insanity of your claims here.

You propose, using trial for this example, to have trial-{2.7,3.1,3.2} 
be interpretter specific versions of the script, and  and to ln -s 
/path/to/python-exec /usr/bin/trial

That means /usr/bin/trial is an elf binary.

This means, regardless of what crack-adled fricking claim you make 
about my cpython knowledge... python /usr/bin/trial, which resolves 
to, `python /usr/bin/python-exec`... ain't going to work.  Ever.  
Never ever.

Ironically, floppym pointed out this sort of flaw in my initial brain 
dump of a symlink approach (which lead to the python-shebang 
approach, emails you responded to but aparently didn't fix in your 
own proposal).  You were active in that thread, trying pretty heavily 
to call the proposal hack/horrible/crappy/a meany, you should've 
caught this flaw and dealt w/ it in your own implementation.


> By the way, I doubt it is a 'core usage case'. Maybe just because it
> never worked as intended, maybe because people simply don't do that
> usually because people usually don't care whether stuff in /usr/bin
> is Python, C or shell.

re-execing and invoking scripts w/in a source via using sys.executable 
is standard.  If in doubt, go sample the python universe.  Hell, think 
through standard re-exec; what you've got implemented here doesn't 
preserve argv[0] as the raw script name (which is a regression from 
existing solutions); were it to do so, you'd be triggering this fault 
via re-exec.

Even if you wanted to make such a claim, you're ignoring that python 
/usr/bin/trial works *now*, and you'd be breaking that.  Meaning 
minimally a regression of what frankly, is insane to break (perl 
/usr/bin/spamassassin shouldn't fail anymore than python 
/usr/bin/emerge should fail).


> > 2) Everytime there is a new python interpretter, python-exec has to
> > be re-built updating it's hardcoded list of interpretters.  Goes without
> > saying, this sucks a fair bit; your solution there likely would be
> > walk env.d/python/* in some fashion extracting the list of targets (or
> > walk /usr/bin/python* and run into problems w/ the names used there
> > now matching EPYTHON usage).  End result, you'd evolve it into
> > something nearing what I did w/ shebangs.
>
> How often does this happen? Is this really problematic?

It's a sign that your solution is hacky, frankly.

We shouldn't have to rebuild binaries for configuration data 
changing- you may not like it, but that's usually a sign of bad code.


> And how an user can disable a particular (say, problematic) Python
> implementation globally in your solution? Without reinstalling all
> the packages which provide scripts for it?

Depends on your definition of 'disable'.  With your solution, the only 
way to disable the interpretter in question is thus

1) Unmerge the interpretter
2) mangle the python abi list for python-exec via rebuilding it.

However.  #2 carries with it one helluva of an unstated gotcha; that 
every single time the available interpretters change, it has to be 
recompiled, or the full list of targets has to be passed in every 
time.  Both of those are arbitrary... lets call it 'hobbiest' 
limitations that aren't production worthy.

Either way, in my solution, if we wanted users to be able to have 
pypy-1.8 installed, have things compiled for it, but blacklist it 
after the fact (while leaving the interpretter installed) I'd just add 
a file into env.d listing the blacklisted interpretters, and update 
python-shebang accordingly.

Keep in mind since the patchset targets python-eselect, deploying that 
is an easy shot.

I'm not particularly convinced you're presenting a real world 
scenario however.  Regardless, it's easily addressed via my solution.


> > 3) It's a hardcoded lookup list that is forced for all scripts; if
> > that list is python3.2,python3.1,python2.7, and the active python
> > version doesn't intersect that (say jython-2.5), your approach forces
> > 3.2.  My solution actually allows the script, in the absense of an
> > enforcement via active python or EPYTHON, to use it's own preferred
> > ordering.  Using pkgcore as an example, we prefer py2k execution over
> > py3k- py2k is faster since the extensions are enabled.  This
> > capability doesn't exist in your solution, nor the existing wrapper;
> > it's a useful byproduct of how my solution is structured.  Etc etc,
> > you know the response; where you'll go with this sooner or later is
> > where python-shebang is already.
>
> And, hmm, are you the first person requesting such a feature over
> the whole lifetime of python-wrapper?

I'm the first person who got pissed enough to address it.  Most of the 
time it works fine- but when you invoke something like trial or 
sphinx-build, things that analyze *other* python namespaces, it 
breaks.

It's inarguably broken; arfrever's proposed solution to it when I 
pointed it out was to hardcode gentooisms into the solution- I don't 
agree with that, fix it properly as I've done.


> I believe that hardcoded list is *simpler*. If user enabled jython-2.5
> (suicidal type?), then he's aware of the consequences. And in my
> solution, preferred Python 2 will be used. Then preferred Python 3.
> Then Python 2.7, 2.6, 2.5... then 3.2. It is predictable.
>
> In your solution, it is completely unpredictable. Developer can
> (supposedly) do whatever he wants. Most of the devs probably won't use
> that, except for a few cases where users will simply hit (if ever)
> an unexpected behavior.

Fact is, the option is opened up via the shebang approach to solving 
this.

You're advocating that the fallback choice should be user controlled.  
Frankly... that's the wrong approach.

If a script is invoked- offlineimap, iotop, cvs2*, btrfs-bcp, 
revdep-rebuild, etc; the user actually rarely cares which 
implementation is used.  Up front, they decided which implementations 
they wanted to allow- but w/in that, sans the active interpretter, 
the user does not *care*.

That's just fact.  A user won't give a shit if it's offlineimap 2.7 or 
offlineimap 3.2.   The implementation however *may* have a preference 
there- pkgcore/snakeoil aren't the only pkgs that have py2k extensions 
but not py3k extension.  If you looked at that level of python, you'd 
recognize the API changes weren't exactly minor.  Doable, but a pita.

The kicker here is if we wanted users to be able to override the 
defaults... it's doable with my solution; just extend python-shebang.

That said, your solution provides no way to enumerate the pkgs own 
preferences- not without shoving that information into a new file 
(yet another file required in comparison to my solution).


> > 5) Via going with a standalone package from eselect-python, you're
> > introducing the potential for the two to be out of sync/incompatible.
> > Addressable via folding it inline into eselect-python (as I did), but
> > it's a con against your solution in it's current claimed form.  Same
> > thing, you'll do what I did sooner or later since it's the better
> > approach.
>
> It's a better development practice to split things.
> You should've known that by now.
> In any case, python-exec can work *without*
> eselect-python. You can try it if you don't believe.

I didn't claim it couldn't; I'm pointing out such a separation has no 
gain, but has costs.

Python-wrapper already exists in python-eselect; python-wrapper is 
dependent on the internal details of how eselect python stores it's 
configuration.  This is fine- python-wrapper can be kept lockstep up 
to date with eselect pythons internals.

Your solution however cannot be, not without leveling deps to force it 
to be lock step (and the transition there will require compatibility 
code left in eselect python).  If you don't see this, then I suggest 
you try modifying eselect python to move the config file, or change 
the semantics of how/what it stores.

Regardless, there's zero reason to separate them.  Separation is done 
when it makes sense, not for random shits and giggles.

python-shebang is effectively a generalized/sane python-wrapper; 
python-wrapper itself is just a re-exec to python-shebang after all.  
The proper place, and the place it will wind up at (regardless of your 
bitching) is eselect python.


> > 7) My solution doesn't hardcode /usr/bin/* pathways.  User sticks a
> > python2.7 into /usr/local/ (for whatever reason they choose to do so),
> > it will honor that.
>
> Errr? Where does your suggestion come from?
>
> As far as I'm aware, user can put into per-implementation copy whatever
> he wants. Including a shell script, a C executable or whatever. It will
> work.

Your solution is reliant on the hardcoded shebang w/in each script; 
meaning /usr/bin/python* or whatever it may be.

Literally, sphinx-build-2.7, it has the hardcoded pathway 
#!/usr/bin/python2.7

Now, if you wanted that to not be hardcoded, it would be
#!/usr/bin/env python2.7

But in doing so, you no longer can pass arguments to the interpretter.

> With your, it won't.

If you're going to make such claims, at least bother to read the 
source please.  My solution searches path; meaning if it's trying to 
find python2.7, it'll search the path for it (same basic logic 
python-wrapper uses).

Kicker is, it's a minor point in favor of python-shebang.  Can either 
ignore the point, or make it into a bigger issue... your choice.


> As far as I'm aware, you solution can't handle per-implementation
> Python options (command-line arguments) either. In mine, it's as simple
> as dropping them onto the shebang.

If you'd read the email you're responding to, you'd see that per 
implementation python options are supported- and work fine now.

Look at the arg parsing bit I posted; or go read the !#@*ing source 
already rather than immediately clicking reply.


> > 8) Your solution doesn't work in the context of being used for
> > /usr/bin/python wrapper; it's close, but as you yourself said, it
> > requires symlinks (python-python2.7).
>
> Or adding a conditional which your solution simply does. So, where are
> you going with this point? To the fact that I'm asking others for
> opinion while you're ignoring their existence?

What conditional am I adding?  

Your solution isn't usable *now* as a drop in for python-wrapper; it 
would have to be python-python2.7.

In the same vane, you're implementation actually requires scripts to 
be named trial-python2.7 trial-python3.2 etc.

That's not "asking peoples views"; that's the reality of your 
implementation.


> > 9) No tests.  Every. Fucking. Pathway. is tested with what I wrote.
> > Python-exec lacks that, else issue #1 would've been spotted up front.
>
> Great. Awesome. You are so awesome I can't even spell it.
>
> Now feel free to grep the bugzie. That 'issue #1' is known. So far,
> I haven't get anyone to say he believes that's really important
> or useful.

Your implementation mandates the following:

*) scripts be changed from sphinx-build-2.7 to sphinx-build-python2.7.  

*) mandates symlinks of python-EPYTHON were it to be attempted to be 
used for python-wrapper.

*) removes the ability to do $PYTHON_INTERP /usr/bin/script

These are the sort of things that would've been identified with tests.  

Meaning you're "lets start using python-exec" emails would've been a 
helluva lot more accurate in those cases.


Pretty much, thus:
*) python-exec regresses from existing standard wrapper- you no longer 
can do python /usr/bin/sphinx-build and it requires a recompile 
everytime a new python interpretter is installed.
*) python-shebang has no regressions from the standard, fixes the 
python2.7 /usr/bin/sphinx-bulid scenario, and decreases the number of 
files on disk.

pretty much, that's the end of conversation with you mgorny.

Python herd, assuming you've not ignored the thread due to the noise, 
your views would be useful.
~harring