1 |
On Thu, Oct 18, 2012 at 5:15 AM, Brian Harring <ferringb@×××××.com> wrote: |
2 |
> If folks haven't looked at python_generate_wrapper_scripts in |
3 |
> python.eclass, I'd suggest doing so. For examples of it's usage, grep |
4 |
> for 'python_generate_wrapper_scripts' in /usr/bin/; any place you see |
5 |
> it, look for <that-script-name>-${PYTHON_TARGETS} (for example, |
6 |
> /usr/bin/sphinx-build{,-2.7,-3.2}. |
7 |
> |
8 |
> Each usage there is a separate custom script for that specific binary; |
9 |
> if there is a bug in the script, well, we're screwed- requires |
10 |
> re-merging the package. |
11 |
> |
12 |
> This setup, at least on my hardware, is .04s added to every |
13 |
> invocation; this is ignoring the inode cost for each, and the issue if |
14 |
> a bug ever appears in the script generation code (in which case we're |
15 |
> screwed- would require re-merging the package). |
16 |
> |
17 |
> In parallel, we've got python-wrapper (ls /usr/bin/python -l); this is |
18 |
> provided by eselect-python and basically discern what the active |
19 |
> python version is, and use that in the absense of any directives. |
20 |
> This is implemented in C, and is reasonably sane; the overhead for |
21 |
> that is basically non-existant. |
22 |
> |
23 |
> Roughly, I'm proposing we do away with python eclass's |
24 |
> generate_python_wrapper_scripts generation of a script, instead having |
25 |
> that just symlink to a binary provided by eselect-python that handles |
26 |
> this. This centralizes the implementation (fix in one spot), and |
27 |
> would allow a c version to be used- basically eliminating the |
28 |
> overhead. |
29 |
> |
30 |
> |
31 |
> There's a trick to this; currently, those generated scripts hardcode |
32 |
> the allowed/known python versions for that package. We obviously have |
33 |
> to preserve that; I propose we shove it into the symlink path. |
34 |
> |
35 |
> Basically, we add a /usr/libexec/python directory; within it, we have |
36 |
> a wrapper binary (explained below), and a set of symlinks pointing at |
37 |
> the root of that directory. To cover our current python versions, the |
38 |
> following would suffice: |
39 |
> |
40 |
> for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8} |
41 |
> \2.7-pypy-1.9; do |
42 |
> ln -s ./ /usr/libexec/python/$x |
43 |
> done |
44 |
> |
45 |
> While that seems insane, there is a reason; via that, we can encode |
46 |
> the allowed versions into the symlink. Using pkgcore's pquery for |
47 |
> example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3) |
48 |
> instead of a wrapper script at /usr/bin/pquery, we'd have thus: |
49 |
> |
50 |
> targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy ) |
51 |
> targets=$(IFS=/;echo -n "${targets[*]}") |
52 |
> # This results in |
53 |
> # targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy |
54 |
> ln -s "/usr/libexec/python/${targets}/wrapper" \ |
55 |
> /usr/bin/pquery |
56 |
> |
57 |
> /usr/libexec/python/wrapper upon invocation, takes a look at argv[0]; |
58 |
> sees how it was invoked basically. This will be the /usr/bin/whatever |
59 |
> pathway. It reads the symlink, in the process getting the allowed |
60 |
> versions and preferred order of the versions. |
61 |
> |
62 |
> Few notes; vast majority of filesystems will store the symlink target |
63 |
> into the core inode if at all possible- in doing so, this avoids |
64 |
> wasting an inode and is only limited by the length of the target. |
65 |
> That length is capped by PATH_MAX- which can range from 256 to 4k (or |
66 |
> higher). |
67 |
> |
68 |
> For the pquery example above, that comes out to ~73 bytes for the |
69 |
> symlink pathway; well under PATH_MAX. |
70 |
> |
71 |
> For the scenarios where PATH_MAX caps the symlink pathway, or for |
72 |
> whatever reason we don't want to use that trick, a tree of files |
73 |
> contained within /usr/libexec/python/ holding the allowed versions for |
74 |
> the matching pathway would suffice. |
75 |
> |
76 |
> Either proposal here would be far faster than what we've got now; also |
77 |
> will use less space (ancillary benefit). |
78 |
> |
79 |
> One subtle aspect here is that if we did this, it makes it possible to |
80 |
> avoid losing the invocation information- currently if you did |
81 |
> `/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things |
82 |
> are implemented now (specifically the two layers of wrappers)- you'll |
83 |
> get python2.7 running that sphinx-build invocation. |
84 |
> |
85 |
> This is wrong (it's directly discarding what the invocation |
86 |
> requested), although you're only going to see it for scripts that |
87 |
> do python introspection. |
88 |
> |
89 |
> Via doing the restructuring I'm mentioning above, that issue can be |
90 |
> fixed, while making things faster/saner. |
91 |
> |
92 |
> On a related note; we currently install multiple versions of the same |
93 |
> script- the only difference being the shebang. If one ignores the |
94 |
> shebang, in some cases this is necessary- where the script is 2to3 |
95 |
> translated, and the code for py2k vs py3k differs. For most, the only |
96 |
> difference is in the shebang however. |
97 |
> |
98 |
> While it's minor in space savings, it's possible to eliminate that |
99 |
> redundancy via a shebang target that looks at the pathway it was |
100 |
> invoked via. Fairly easy actually, and basically zero overhead if |
101 |
> done. |
102 |
> |
103 |
> Either way, thoughts? |
104 |
> |
105 |
> What I'm proposing isn't perfect, but I'm of the view it's a step up |
106 |
> from what's in place now- and via centralizing this crap, makes it |
107 |
> easier to change/maintain this going forward as necessary. |
108 |
> ~harring |
109 |
|
110 |
If we are somehow going to eliminate the installation of a separate |
111 |
script for each python version, then the symlink idea sounds like a |
112 |
good solution for expressing the supported python versions. However, |
113 |
I'm really not sure how you would eliminate the separate scripts, in |
114 |
light of the 2to3 issue. |
115 |
|
116 |
Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example: |
117 |
invoking python on a binary (or a symlink to a binary) isn't going to |
118 |
work at all. So I don't see how you've solved that issue. |
119 |
|
120 |
Back to the discussion djc pointed out: mgorny has implemented a |
121 |
binary solution already, called "python-exec", in conjunction with |
122 |
python-r1.eclass. |
123 |
|
124 |
https://bitbucket.org/mgorny/python-exec |
125 |
|
126 |
python-exec assumes there will be a separate script for each version, |
127 |
and just calls exec several times. In shell code, it would look like |
128 |
this: |
129 |
|
130 |
scriptname=$0 # argv[0] |
131 |
exec scriptname-${EPYTHON} |
132 |
exec scriptname-$(< /etc/env.d/python/config) |
133 |
for x in python2.7 python2.6 python3.2 python3.1 ... ; do |
134 |
exec scriptname-${x} |
135 |
done |
136 |
|
137 |
The list of implementations for the loop at the end is hard-coded when |
138 |
python-exec is built. This is the weak point in the solution. It could |
139 |
be improved with your symlink idea. |
140 |
|
141 |
What are your thoughts on mgorny's python-exec solution? |
142 |
|
143 |
We could adapt it for python.eclass if you really want to support |
144 |
that. Probably by modifying python.eclass to install scripts like |
145 |
sphinx-python2.7 instead of sphinx-2.7. |