Re: [gentoo-dev] reworking python-wrapper and wrapper scripts.

[Michał Górny]

On Thu, 18 Oct 2012 02:15:43 -0700
Brian Harring <ferringb@×××××.com> wrote:

> There's a trick to this; currently, those generated scripts hardcode
> the allowed/known python versions for that package.  We obviously have
> to preserve that; I propose we shove it into the symlink path.
>
> Basically, we add a /usr/libexec/python directory; within it, we have
> a wrapper binary (explained below), and a set of symlinks pointing at
> the root of that directory.  To cover our current python versions, the
> following would suffice:
>
> for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8}
> \2.7-pypy-1.9; do
>   ln -s ./ /usr/libexec/python/$x
> done
>
> While that seems insane, there is a reason; via that, we can encode
> the allowed versions into the symlink.  Using pkgcore's pquery for
> example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3)
> instead of a wrapper script at /usr/bin/pquery, we'd have thus:
>
> targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
> targets=$(IFS=/;echo -n "${targets[*]}")
> # This results in
> # targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
> ln -s "/usr/libexec/python/${targets}/wrapper" \
>   /usr/bin/pquery
>
> /usr/libexec/python/wrapper upon invocation, takes a look at argv[0];
> sees how it was invoked basically.  This will be the /usr/bin/whatever
> pathway.  It reads the symlink, in the process getting the allowed
> versions and preferred order of the versions.
>
> Few notes; vast majority of filesystems will store the symlink target
> into the core inode if at all possible- in doing so, this avoids
> wasting an inode and is only limited by the length of the target.
> That length is capped by PATH_MAX- which can range from 256 to 4k (or
> higher).
>
> For the pquery example above, that comes out to ~73 bytes for the
> symlink pathway; well under PATH_MAX.
>
> For the scenarios where PATH_MAX caps the symlink pathway, or for
> whatever reason we don't want to use that trick, a tree of files
> contained within /usr/libexec/python/ holding the allowed versions for
> the matching pathway would suffice.

While I agree that it's a clever trick, I doubt it's worth the effort.
Did you got any numbers proving it being superior over, say, trying to
exec() scripts like I do in python-exec?

While I can imagine, that in an worst case that bunch of exec()s is
going to be definitely slower than storing the list anyway, I doubt
such a bad case is often.

Considering that the most common Python version used now is Python 2,
how often doesn't the script support that Python version? That's a very
rare case, and often just executing "foo-${EPYTHON}" works. In your
case, that common case involves readlink() + parsing + exec().

Even in case of Python 3 being selected, I doubt the overhead
of multiple exec()s on the scripts not supporting Python 3 is really
relevant. Please measure it if you believe so.

To be honest, I don't see any real advantage in this solution. It is
complex; understanding it requires explanation or some thinking.
The code will be fragile, and I'm not even sure if I'm not missing
something important here.

Thus, unless you've got a good arguments how this solution is superior
to the straightforward one done in python-exec, or numbers proving that
it is more efficient in a way noticeable to our users, -1. Smart hack,
yes, but not really beneficial.

> Either proposal here would be far faster than what we've got now; also
> will use less space (ancillary benefit).
>
> One subtle aspect here is that if we did this, it makes it possible to
> avoid losing the invocation information- currently if you did
> `/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things
> are implemented now (specifically the two layers of wrappers)- you'll
> get python2.7 running that sphinx-build invocation.
>
> This is wrong (it's directly discarding what the invocation
> requested), although you're only going to see it for scripts that
> do python introspection.
>
> Via doing the restructuring I'm mentioning above, that issue can be
> fixed, while making things faster/saner.

I don't see how this is relevant to the wrapper. As Mike pointed out,
python3.2 is the actual Python executable, and the wrapper you're
suggesting is a C executable -- something that does not really work
together like that.

So please elaborate on how you are actually going to solve this. Hope
it's not through patching Python...

-- 
Best regards,
Michał Górny