1 |
Python has little concept of cross-compiling but it turns out that |
2 |
making it work isn't so hard after all. |
3 |
|
4 |
Platform-specific details are held within _sysconfigdata.py, |
5 |
sysconfig.py, and various distutils files. If we simply symlink these |
6 |
from SYSROOT into an empty directory and add that directory to |
7 |
PYTHONPATH then we can utilise the build host's Python with the target |
8 |
host's settings. |
9 |
|
10 |
The pkg-config files were already being symlinked in a similar manner |
11 |
but now we source them from within SYSROOT. |
12 |
|
13 |
In order for PYTHONPATH to be respected, Python must be executed via |
14 |
the wrapper, which was not the case before. We previously relied |
15 |
solely on the PATH but now PYTHON must point to the wrapper rather |
16 |
than the usual location under /usr/bin. However, we only do this when |
17 |
SYSROOT or EPREFIX are effectively set to avoid unnecessary |
18 |
complexity. This has required some rejigging in the way that PYTHON is |
19 |
set but it should remain compatible with existing packages. |
20 |
|
21 |
The python_wrapper_setup function that handles all this has had its |
22 |
arguments reordered because no one ever uses the path/workdir |
23 |
argument, which makes specifying other arguments tedious. |
24 |
|
25 |
Some packages rely on python-config but luckily this is just a shell |
26 |
script so it can be executed from within SYSROOT. This is bending the |
27 |
rules of PMS slightly but Python leaves us with little choice. I also |
28 |
wrote those rules so I'm allowed to bend them. ;) |
29 |
|
30 |
PYTHON_INCLUDEDIR, PYTHON_LIBPATH, and their associated functions are |
31 |
generally used during src_configure or src_compile and, as such, they |
32 |
now need to have SYSROOT prepended. |
33 |
|
34 |
python_doheader uses PYTHON_INCLUDEDIR to install new headers and |
35 |
therefore needs the value without SYSROOT. It was already stripping |
36 |
EPREFIX before so now it simply strips SYSROOT as well. Similar |
37 |
instances of this can do likewise or call the functions with SYSROOT |
38 |
unset to achieve the same effect. |
39 |
|
40 |
To make all this work, we are assuming that CPython is located at |
41 |
/usr/$(get_libdir)/${EPYTHON}, which is admittedly a little circular |
42 |
when we are querying Python for the right paths. I feel the reason |
43 |
that python_export was rewritten to query these dynamically was less |
44 |
because someone might install Python to some weird location and more |
45 |
to avoid special handling for each of the different |
46 |
implementations. And what of those other implementations? |
47 |
|
48 |
Being Java-based, Jython is installed under the platform-neutral |
49 |
/usr/share and presumably has few other platform-specific |
50 |
aspects. Enabling native extensions appears non-trivial and none of |
51 |
our module packages have enabled support for it anyway. |
52 |
|
53 |
I think PyPy could potentially support cross-compiling but it |
54 |
hardcodes the native extension filename suffix within its own binaries |
55 |
with no way to override it. Perhaps we could patch this in somehow but |
56 |
I'm afraid I don't care enough. |
57 |
|
58 |
Together with the following changes to distutils-r1.eclass, I have now |
59 |
been able to cross-compile a large number of packages with native |
60 |
Python extensions, most with no changes at all, and the rest with only |
61 |
minor fixes. |
62 |
|
63 |
Closes: https://bugs.gentoo.org/503874 |
64 |
Bug: https://bugs.gentoo.org/648652 |
65 |
Signed-off-by: James Le Cuirot <chewi@g.o> |
66 |
--- |
67 |
eclass/python-utils-r1.eclass | 120 ++++++++++++++++++++++++++-------- |
68 |
1 file changed, 92 insertions(+), 28 deletions(-) |
69 |
|
70 |
diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass |
71 |
index 1a549f49f3f2..607af1b52f75 100644 |
72 |
--- a/eclass/python-utils-r1.eclass |
73 |
+++ b/eclass/python-utils-r1.eclass |
74 |
@@ -211,9 +211,15 @@ _python_impl_matches() { |
75 |
# |
76 |
# distutils-r1: Set within any of the python sub-phase functions. |
77 |
# |
78 |
-# Example value: |
79 |
+# If SYSROOT or EPREFIX are effectively set then this will point to an |
80 |
+# automatically generated wrapper rather than the usual path under |
81 |
+# /usr/bin in order to accommodate cross-compiling. We could do this all |
82 |
+# the time but it would add unnecessary complexity. |
83 |
+# |
84 |
+# Example values: |
85 |
# @CODE |
86 |
# /usr/bin/python2.7 |
87 |
+# /var/tmp/portage/dev-python/pyblake2-1.2.3/temp/python2.7/bin/python2.7 |
88 |
# @CODE |
89 |
|
90 |
# @ECLASS-VARIABLE: EPYTHON |
91 |
@@ -256,6 +262,10 @@ _python_impl_matches() { |
92 |
# Set and exported on request using python_export(). |
93 |
# Requires a proper build-time dependency on the Python implementation. |
94 |
# |
95 |
+# This is prepended with SYSROOT in order to accommodate |
96 |
+# cross-compiling. You may need to strip SYSROOT and EPREFIX if using it |
97 |
+# to install new headers. |
98 |
+# |
99 |
# Example value: |
100 |
# @CODE |
101 |
# /usr/include/python2.7 |
102 |
@@ -270,6 +280,9 @@ _python_impl_matches() { |
103 |
# Valid only for CPython. Requires a proper build-time dependency |
104 |
# on the Python implementation. |
105 |
# |
106 |
+# This is prepended with SYSROOT in order to accommodate |
107 |
+# cross-compiling. |
108 |
+# |
109 |
# Example value: |
110 |
# @CODE |
111 |
# /usr/lib64/libpython2.7.so |
112 |
@@ -314,6 +327,10 @@ _python_impl_matches() { |
113 |
# Valid only for CPython. Requires a proper build-time dependency |
114 |
# on the Python implementation and on pkg-config. |
115 |
# |
116 |
+# This is prepended with SYSROOT in order to accommodate |
117 |
+# cross-compiling. You generally should not execute files within SYSROOT |
118 |
+# but python-config is always a shell script. |
119 |
+# |
120 |
# Example value: |
121 |
# @CODE |
122 |
# /usr/bin/python2.7-config |
123 |
@@ -380,6 +397,10 @@ python_export() { |
124 |
esac |
125 |
debug-print "${FUNCNAME}: implementation: ${impl}" |
126 |
|
127 |
+ # Many variables below need a PYTHON variable but we should not |
128 |
+ # export it unless explicitly requested so use _PYTHON instead. |
129 |
+ local _PYTHON |
130 |
+ |
131 |
for var; do |
132 |
case "${var}" in |
133 |
EPYTHON) |
134 |
@@ -387,21 +408,21 @@ python_export() { |
135 |
debug-print "${FUNCNAME}: EPYTHON = ${EPYTHON}" |
136 |
;; |
137 |
PYTHON) |
138 |
- export PYTHON=${EPREFIX}/usr/bin/${impl} |
139 |
+ python_wrapper_setup ${impl} PYTHON |
140 |
debug-print "${FUNCNAME}: PYTHON = ${PYTHON}" |
141 |
;; |
142 |
PYTHON_SITEDIR) |
143 |
- [[ -n ${PYTHON} ]] || die "PYTHON needs to be set for ${var} to be exported, or requested before it" |
144 |
+ python_wrapper_setup ${impl} _PYTHON |
145 |
# sysconfig can't be used because: |
146 |
# 1) pypy doesn't give site-packages but stdlib |
147 |
# 2) jython gives paths with wrong case |
148 |
- PYTHON_SITEDIR=$("${PYTHON}" -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib())') || die |
149 |
+ PYTHON_SITEDIR=$("${_PYTHON}" -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib())') || die |
150 |
export PYTHON_SITEDIR |
151 |
debug-print "${FUNCNAME}: PYTHON_SITEDIR = ${PYTHON_SITEDIR}" |
152 |
;; |
153 |
PYTHON_INCLUDEDIR) |
154 |
- [[ -n ${PYTHON} ]] || die "PYTHON needs to be set for ${var} to be exported, or requested before it" |
155 |
- PYTHON_INCLUDEDIR=$("${PYTHON}" -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_inc())') || die |
156 |
+ python_wrapper_setup ${impl} _PYTHON |
157 |
+ PYTHON_INCLUDEDIR=${SYSROOT}$("${_PYTHON}" -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_inc())') || die |
158 |
export PYTHON_INCLUDEDIR |
159 |
debug-print "${FUNCNAME}: PYTHON_INCLUDEDIR = ${PYTHON_INCLUDEDIR}" |
160 |
|
161 |
@@ -411,8 +432,8 @@ python_export() { |
162 |
fi |
163 |
;; |
164 |
PYTHON_LIBPATH) |
165 |
- [[ -n ${PYTHON} ]] || die "PYTHON needs to be set for ${var} to be exported, or requested before it" |
166 |
- PYTHON_LIBPATH=$("${PYTHON}" -c 'import os.path, sysconfig; print(os.path.join(sysconfig.get_config_var("LIBDIR"), sysconfig.get_config_var("LDLIBRARY")) if sysconfig.get_config_var("LDLIBRARY") else "")') || die |
167 |
+ python_wrapper_setup ${impl} _PYTHON |
168 |
+ PYTHON_LIBPATH=${SYSROOT}$("${_PYTHON}" -c 'import os.path, sysconfig; print(os.path.join(sysconfig.get_config_var("LIBDIR"), sysconfig.get_config_var("LDLIBRARY")) if sysconfig.get_config_var("LDLIBRARY") else "")') || die |
169 |
export PYTHON_LIBPATH |
170 |
debug-print "${FUNCNAME}: PYTHON_LIBPATH = ${PYTHON_LIBPATH}" |
171 |
|
172 |
@@ -457,9 +478,9 @@ python_export() { |
173 |
|
174 |
case "${impl}" in |
175 |
python*) |
176 |
- [[ -n ${PYTHON} ]] || die "PYTHON needs to be set for ${var} to be exported, or requested before it" |
177 |
- flags=$("${PYTHON}" -c 'import sysconfig; print(sysconfig.get_config_var("ABIFLAGS") or "")') || die |
178 |
- val=${PYTHON}${flags}-config |
179 |
+ python_wrapper_setup ${impl} _PYTHON |
180 |
+ flags=$("${_PYTHON}" -c 'import sysconfig; print(sysconfig.get_config_var("ABIFLAGS") or "")') || die |
181 |
+ val=${ESYSROOT-${SYSROOT}${EPREFIX}}/usr/bin/${PYTHON##*/}${flags}-config |
182 |
;; |
183 |
*) |
184 |
die "${impl}: obtaining ${var} not supported" |
185 |
@@ -954,7 +975,7 @@ python_doheader() { |
186 |
local d PYTHON_INCLUDEDIR=${PYTHON_INCLUDEDIR} |
187 |
[[ ${PYTHON_INCLUDEDIR} ]] || python_export PYTHON_INCLUDEDIR |
188 |
|
189 |
- d=${PYTHON_INCLUDEDIR#${EPREFIX}} |
190 |
+ d=${PYTHON_INCLUDEDIR#${ESYSROOT-${SYSROOT}${EPREFIX}}} |
191 |
|
192 |
( |
193 |
insopts -m 0644 |
194 |
@@ -964,7 +985,7 @@ python_doheader() { |
195 |
} |
196 |
|
197 |
# @FUNCTION: python_wrapper_setup |
198 |
-# @USAGE: [<path> [<impl>]] |
199 |
+# @USAGE: [<impl> [<pyvar> [<path>]]] |
200 |
# @DESCRIPTION: |
201 |
# Create proper 'python' executable and pkg-config wrappers |
202 |
# (if available) in the directory named by <path>. Set up PATH |
203 |
@@ -973,6 +994,9 @@ python_doheader() { |
204 |
# The wrappers will be created for implementation named by <impl>, |
205 |
# or for one named by ${EPYTHON} if no <impl> passed. |
206 |
# |
207 |
+# The path to the 'python' executable wrapper is exported to the |
208 |
+# variable named <pyvar> if given. |
209 |
+# |
210 |
# If the named directory contains a python symlink already, it will |
211 |
# be assumed to contain proper wrappers already and only environment |
212 |
# setup will be done. If wrapper update is requested, the directory |
213 |
@@ -980,25 +1004,41 @@ python_doheader() { |
214 |
python_wrapper_setup() { |
215 |
debug-print-function ${FUNCNAME} "${@}" |
216 |
|
217 |
- local impl=${2:-${EPYTHON}} |
218 |
- local workdir=${1:-${T}/${impl}} |
219 |
+ local impl=${1:-${EPYTHON}} |
220 |
+ local pyvar=${2} |
221 |
+ local lpyvar=_${pyvar:-PYTHON} |
222 |
+ |
223 |
+ # Use separate directories for SYSROOT in case we need to execute |
224 |
+ # Python in the context of the build host by unsetting SYSROOT. |
225 |
+ local workdir=${3:-${T}/${impl}${SYSROOT:+-sysroot}} |
226 |
|
227 |
[[ ${workdir} ]] || die "${FUNCNAME}: no workdir specified." |
228 |
[[ ${impl} ]] || die "${FUNCNAME}: no impl nor EPYTHON specified." |
229 |
|
230 |
+ local EPYTHON |
231 |
+ python_export "${impl}" EPYTHON |
232 |
+ |
233 |
+ # We could use BROOT here but using the PATH accommodates |
234 |
+ # cross-prefix where the PATH is sometimes manipulated to prefer |
235 |
+ # build tools from the target prefix (i.e. EPREFIX). |
236 |
+ # |
237 |
+ # Also make sure we don't pick up an existing wrapper by replacing |
238 |
+ # instances of ${T} with a bogus location. The workdir can be |
239 |
+ # overridden but hopefully it will be somewhere under ${T}. |
240 |
+ local ${lpyvar}=$(PATH=${PATH//${T}//dev/null} type -P "${EPYTHON}" || die "${FUNCNAME}: can't find ${EPYTHON} in PATH") |
241 |
+ |
242 |
+ local pysysroot=${ESYSROOT-${SYSROOT%/}${EPREFIX}} |
243 |
+ |
244 |
if [[ ! -x ${workdir}/bin/python ]]; then |
245 |
_python_check_dead_variables |
246 |
|
247 |
- mkdir -p "${workdir}"/{bin,pkgconfig} || die |
248 |
+ mkdir -p "${workdir}"/{bin,lib,pkgconfig} || die |
249 |
|
250 |
# Clean up, in case we were supposed to do a cheap update. |
251 |
rm -f "${workdir}"/bin/python{,2,3}{,-config} || die |
252 |
rm -f "${workdir}"/bin/2to3 || die |
253 |
rm -f "${workdir}"/pkgconfig/python{,2,3}.pc || die |
254 |
|
255 |
- local EPYTHON PYTHON |
256 |
- python_export "${impl}" EPYTHON PYTHON |
257 |
- |
258 |
local pyver pyother |
259 |
if python_is_python3; then |
260 |
pyver=3 |
261 |
@@ -1012,37 +1052,53 @@ python_wrapper_setup() { |
262 |
# note: we don't use symlinks because python likes to do some |
263 |
# symlink reading magic that breaks stuff |
264 |
# https://bugs.gentoo.org/show_bug.cgi?id=555752 |
265 |
- cat > "${workdir}/bin/python" <<-_EOF_ || die |
266 |
- #!/bin/sh |
267 |
- exec "${PYTHON}" "\${@}" |
268 |
- _EOF_ |
269 |
- cp "${workdir}/bin/python" "${workdir}/bin/python${pyver}" || die |
270 |
- chmod +x "${workdir}/bin/python" "${workdir}/bin/python${pyver}" || die |
271 |
+ echo '#!/bin/sh' > "${workdir}/bin/python" || die |
272 |
|
273 |
local nonsupp=( "python${pyother}" "python${pyother}-config" ) |
274 |
|
275 |
# CPython-specific |
276 |
if [[ ${EPYTHON} == python* ]]; then |
277 |
+ local pysysrootlib=${pysysroot}/usr/$(get_libdir) |
278 |
+ |
279 |
cat > "${workdir}/bin/python-config" <<-_EOF_ || die |
280 |
#!/bin/sh |
281 |
- exec "${PYTHON}-config" "\${@}" |
282 |
+ exec "${pysysroot}/usr/bin/${EPYTHON}-config" "\${@}" |
283 |
_EOF_ |
284 |
cp "${workdir}/bin/python-config" \ |
285 |
"${workdir}/bin/python${pyver}-config" || die |
286 |
chmod +x "${workdir}/bin/python-config" \ |
287 |
"${workdir}/bin/python${pyver}-config" || die |
288 |
|
289 |
+ if [[ -n ${pysysroot} ]]; then |
290 |
+ # Use host-specific data from SYSROOT. Python 2 looks |
291 |
+ # for _sysconfigdata while Python 3 uses |
292 |
+ # _PYTHON_SYSCONFIGDATA_NAME. |
293 |
+ ln -s "${pysysrootlib}/${EPYTHON}"/_sysconfigdata*.py "${workdir}"/lib/_sysconfigdata.py || die |
294 |
+ |
295 |
+ # Use distutils/sysconfig from SYSROOT as parts of it |
296 |
+ # have GENTOO_LIBDIR baked in at Python build time. |
297 |
+ ln -s "${pysysrootlib}/${EPYTHON}"/{distutils,sysconfig.py} "${workdir}"/lib/ || die |
298 |
+ |
299 |
+ # Add env vars to python wrapper accordingly. |
300 |
+ echo "PYTHONPATH=\"${workdir}/lib\${PYTHONPATH:+:}\${PYTHONPATH}\" _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata \\" \ |
301 |
+ >> "${workdir}/bin/python" || die |
302 |
+ fi |
303 |
+ |
304 |
# Python 2.6+. |
305 |
- ln -s "${PYTHON/python/2to3-}" "${workdir}"/bin/2to3 || die |
306 |
+ ln -s "${EPYTHON/python/2to3-}" "${workdir}"/bin/2to3 || die |
307 |
|
308 |
# Python 2.7+. |
309 |
- ln -s "${EPREFIX}"/usr/$(get_libdir)/pkgconfig/${EPYTHON/n/n-}.pc \ |
310 |
+ ln -s "${pysysrootlib}"/pkgconfig/${EPYTHON/n/n-}.pc \ |
311 |
"${workdir}"/pkgconfig/python.pc || die |
312 |
ln -s python.pc "${workdir}"/pkgconfig/python${pyver}.pc || die |
313 |
else |
314 |
nonsupp+=( 2to3 python-config "python${pyver}-config" ) |
315 |
fi |
316 |
|
317 |
+ echo "exec \"${!lpyvar}\" \"\${@}\"" >> "${workdir}"/bin/python || die |
318 |
+ tee "${workdir}"/bin/{python${pyver},"${EPYTHON}"} < "${workdir}"/bin/python >/dev/null || die |
319 |
+ chmod +x "${workdir}"/bin/{python,python${pyver},"${EPYTHON}"} || die |
320 |
+ |
321 |
local x |
322 |
for x in "${nonsupp[@]}"; do |
323 |
cat >"${workdir}"/bin/${x} <<-_EOF_ || die |
324 |
@@ -1064,6 +1120,14 @@ python_wrapper_setup() { |
325 |
PKG_CONFIG_PATH=${workdir}/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}} |
326 |
fi |
327 |
export PATH PKG_CONFIG_PATH |
328 |
+ |
329 |
+ if [[ -n ${pyvar} ]]; then |
330 |
+ if [[ -n ${pysysroot} ]]; then |
331 |
+ export -- ${pyvar}=${workdir}/bin/${EPYTHON} |
332 |
+ else |
333 |
+ export -- ${pyvar}=${!lpyvar} |
334 |
+ fi |
335 |
+ fi |
336 |
} |
337 |
|
338 |
# @FUNCTION: python_is_python3 |
339 |
-- |
340 |
2.19.2 |