[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.

[Mike Gilbert]

On Thu, Oct 18, 2012 at 5:15 AM, Brian Harring <ferringb@×××××.com> wrote:
> If folks haven't looked at python_generate_wrapper_scripts in
> python.eclass, I'd suggest doing so.  For examples of it's usage, grep
> for 'python_generate_wrapper_scripts' in /usr/bin/; any place you see
> it, look for <that-script-name>-${PYTHON_TARGETS} (for example,
> /usr/bin/sphinx-build{,-2.7,-3.2}.
>
> Each usage there is a separate custom script for that specific binary;
> if there is a bug in the script, well, we're screwed- requires
> re-merging the package.
>
> This setup, at least on my hardware, is .04s added to every
> invocation; this is ignoring the inode cost for each, and the issue if
> a bug ever appears in the script generation code (in which case we're
> screwed- would require re-merging the package).
>
> In parallel, we've got python-wrapper (ls /usr/bin/python -l); this is
> provided by eselect-python and basically discern what the active
> python version is, and use that in the absense of any directives.
> This is implemented in C, and is reasonably sane; the overhead for
> that is basically non-existant.
>
> Roughly, I'm proposing we do away with python eclass's
> generate_python_wrapper_scripts generation of a script, instead having
> that just symlink to a binary provided by eselect-python that handles
> this.  This centralizes the implementation (fix in one spot), and
> would allow a c version to be used- basically eliminating the
> overhead.
>
>
> There's a trick to this; currently, those generated scripts hardcode
> the allowed/known python versions for that package.  We obviously have
> to preserve that; I propose we shove it into the symlink path.
>
> Basically, we add a /usr/libexec/python directory; within it, we have
> a wrapper binary (explained below), and a set of symlinks pointing at
> the root of that directory.  To cover our current python versions, the
> following would suffice:
>
> for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8}
> \2.7-pypy-1.9; do
>   ln -s ./ /usr/libexec/python/$x
> done
>
> While that seems insane, there is a reason; via that, we can encode
> the allowed versions into the symlink.  Using pkgcore's pquery for
> example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3)
> instead of a wrapper script at /usr/bin/pquery, we'd have thus:
>
> targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
> targets=$(IFS=/;echo -n "${targets[*]}")
> # This results in
> # targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
> ln -s "/usr/libexec/python/${targets}/wrapper" \
>   /usr/bin/pquery
>
> /usr/libexec/python/wrapper upon invocation, takes a look at argv[0];
> sees how it was invoked basically.  This will be the /usr/bin/whatever
> pathway.  It reads the symlink, in the process getting the allowed
> versions and preferred order of the versions.
>
> Few notes; vast majority of filesystems will store the symlink target
> into the core inode if at all possible- in doing so, this avoids
> wasting an inode and is only limited by the length of the target.
> That length is capped by PATH_MAX- which can range from 256 to 4k (or
> higher).
>
> For the pquery example above, that comes out to ~73 bytes for the
> symlink pathway; well under PATH_MAX.
>
> For the scenarios where PATH_MAX caps the symlink pathway, or for
> whatever reason we don't want to use that trick, a tree of files
> contained within /usr/libexec/python/ holding the allowed versions for
> the matching pathway would suffice.
>
> Either proposal here would be far faster than what we've got now; also
> will use less space (ancillary benefit).
>
> One subtle aspect here is that if we did this, it makes it possible to
> avoid losing the invocation information- currently if you did
> `/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things
> are implemented now (specifically the two layers of wrappers)- you'll
> get python2.7 running that sphinx-build invocation.
>
> This is wrong (it's directly discarding what the invocation
> requested), although you're only going to see it for scripts that
> do python introspection.
>
> Via doing the restructuring I'm mentioning above, that issue can be
> fixed, while making things faster/saner.
>
> On a related note; we currently install multiple versions of the same
> script- the only difference being the shebang.  If one ignores the
> shebang, in some cases this is necessary- where the script is 2to3
> translated, and the code for py2k vs py3k differs.  For most, the only
> difference is in the shebang however.
>
> While it's minor in space savings, it's possible to eliminate that
> redundancy via a shebang target that looks at the pathway it was
> invoked via.  Fairly easy actually, and basically zero overhead if
> done.
>
> Either way, thoughts?
>
> What I'm proposing isn't perfect, but I'm of the view it's a step up
> from what's in place now- and via centralizing this crap, makes it
> easier to change/maintain this going forward as necessary.
> ~harring

If we are somehow going to eliminate the installation of a separate
script for each python version, then the symlink idea sounds like a
good solution for expressing the supported python versions. However,
I'm really not sure how you would eliminate the separate scripts, in
light of the 2to3 issue.

Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example:
invoking python on a binary (or a symlink to a binary) isn't going to
work at all. So I don't see how you've solved that issue.

Back to the discussion djc pointed out: mgorny has implemented a
binary solution already, called "python-exec", in conjunction with
python-r1.eclass.

https://bitbucket.org/mgorny/python-exec

python-exec assumes there will be a separate script for each version,
and just calls exec several times. In shell code, it would look like
this:

scriptname=$0 # argv[0]
exec scriptname-${EPYTHON}
exec scriptname-$(< /etc/env.d/python/config)
for x in python2.7 python2.6 python3.2 python3.1 ... ; do
  exec scriptname-${x}
done

The list of implementations for the loop at the end is hard-coded when
python-exec is built. This is the weak point in the solution. It could
be improved with your symlink idea.

What are your thoughts on mgorny's python-exec solution?

We could adapt it for python.eclass if you really want to support
that. Probably by modifying python.eclass to install scripts like
sphinx-python2.7 instead of sphinx-2.7.