Gentoo Archives: gentoo-dev

From: James Le Cuirot <chewi@g.o>
To: gentoo-dev <gentoo-dev@l.g.o>
Cc: James Le Cuirot <chewi@g.o>
Subject: [gentoo-dev] [PATCH 04/12] python-utils-r1.eclass: Use wrapper scripts to fix cross-compiling
Date: Thu, 03 Jan 2019 21:41:39
Message-Id: 20190103213924.22835-5-chewi@gentoo.org
In Reply to: [gentoo-dev] [PATCH] Eclass changes for cross-compiling Python modules by James Le Cuirot
1 Python has little concept of cross-compiling but it turns out that
2 making it work isn't so hard after all.
3
4 Platform-specific details are held within _sysconfigdata.py,
5 sysconfig.py, and various distutils files. If we simply symlink these
6 from SYSROOT into an empty directory and add that directory to
7 PYTHONPATH then we can utilise the build host's Python with the target
8 host's settings.
9
10 The pkg-config files were already being symlinked in a similar manner
11 but now we source them from within SYSROOT.
12
13 In order for PYTHONPATH to be respected, Python must be executed via
14 the wrapper, which was not the case before. We previously relied
15 solely on the PATH but now PYTHON must point to the wrapper rather
16 than the usual location under /usr/bin. However, we only do this when
17 SYSROOT or EPREFIX are effectively set to avoid unnecessary
18 complexity. This has required some rejigging in the way that PYTHON is
19 set but it should remain compatible with existing packages.
20
21 The python_wrapper_setup function that handles all this has had its
22 arguments reordered because no one ever uses the path/workdir
23 argument, which makes specifying other arguments tedious.
24
25 Some packages rely on python-config but luckily this is just a shell
26 script so it can be executed from within SYSROOT. This is bending the
27 rules of PMS slightly but Python leaves us with little choice. I also
28 wrote those rules so I'm allowed to bend them. ;)
29
30 PYTHON_INCLUDEDIR, PYTHON_LIBPATH, and their associated functions are
31 generally used during src_configure or src_compile and, as such, they
32 now need to have SYSROOT prepended.
33
34 python_doheader uses PYTHON_INCLUDEDIR to install new headers and
35 therefore needs the value without SYSROOT. It was already stripping
36 EPREFIX before so now it simply strips SYSROOT as well. Similar
37 instances of this can do likewise or call the functions with SYSROOT
38 unset to achieve the same effect.
39
40 To make all this work, we are assuming that CPython is located at
41 /usr/$(get_libdir)/${EPYTHON}, which is admittedly a little circular
42 when we are querying Python for the right paths. I feel the reason
43 that python_export was rewritten to query these dynamically was less
44 because someone might install Python to some weird location and more
45 to avoid special handling for each of the different
46 implementations. And what of those other implementations?
47
48 Being Java-based, Jython is installed under the platform-neutral
49 /usr/share and presumably has few other platform-specific
50 aspects. Enabling native extensions appears non-trivial and none of
51 our module packages have enabled support for it anyway.
52
53 I think PyPy could potentially support cross-compiling but it
54 hardcodes the native extension filename suffix within its own binaries
55 with no way to override it. Perhaps we could patch this in somehow but
56 I'm afraid I don't care enough.
57
58 Together with the following changes to distutils-r1.eclass, I have now
59 been able to cross-compile a large number of packages with native
60 Python extensions, most with no changes at all, and the rest with only
61 minor fixes.
62
63 Closes: https://bugs.gentoo.org/503874
64 Bug: https://bugs.gentoo.org/648652
65 Signed-off-by: James Le Cuirot <chewi@g.o>
66 ---
67 eclass/python-utils-r1.eclass | 120 ++++++++++++++++++++++++++--------
68 1 file changed, 92 insertions(+), 28 deletions(-)
69
70 diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
71 index 1a549f49f3f2..607af1b52f75 100644
72 --- a/eclass/python-utils-r1.eclass
73 +++ b/eclass/python-utils-r1.eclass
74 @@ -211,9 +211,15 @@ _python_impl_matches() {
75 #
76 # distutils-r1: Set within any of the python sub-phase functions.
77 #
78 -# Example value:
79 +# If SYSROOT or EPREFIX are effectively set then this will point to an
80 +# automatically generated wrapper rather than the usual path under
81 +# /usr/bin in order to accommodate cross-compiling. We could do this all
82 +# the time but it would add unnecessary complexity.
83 +#
84 +# Example values:
85 # @CODE
86 # /usr/bin/python2.7
87 +# /var/tmp/portage/dev-python/pyblake2-1.2.3/temp/python2.7/bin/python2.7
88 # @CODE
89
90 # @ECLASS-VARIABLE: EPYTHON
91 @@ -256,6 +262,10 @@ _python_impl_matches() {
92 # Set and exported on request using python_export().
93 # Requires a proper build-time dependency on the Python implementation.
94 #
95 +# This is prepended with SYSROOT in order to accommodate
96 +# cross-compiling. You may need to strip SYSROOT and EPREFIX if using it
97 +# to install new headers.
98 +#
99 # Example value:
100 # @CODE
101 # /usr/include/python2.7
102 @@ -270,6 +280,9 @@ _python_impl_matches() {
103 # Valid only for CPython. Requires a proper build-time dependency
104 # on the Python implementation.
105 #
106 +# This is prepended with SYSROOT in order to accommodate
107 +# cross-compiling.
108 +#
109 # Example value:
110 # @CODE
111 # /usr/lib64/libpython2.7.so
112 @@ -314,6 +327,10 @@ _python_impl_matches() {
113 # Valid only for CPython. Requires a proper build-time dependency
114 # on the Python implementation and on pkg-config.
115 #
116 +# This is prepended with SYSROOT in order to accommodate
117 +# cross-compiling. You generally should not execute files within SYSROOT
118 +# but python-config is always a shell script.
119 +#
120 # Example value:
121 # @CODE
122 # /usr/bin/python2.7-config
123 @@ -380,6 +397,10 @@ python_export() {
124 esac
125 debug-print "${FUNCNAME}: implementation: ${impl}"
126
127 + # Many variables below need a PYTHON variable but we should not
128 + # export it unless explicitly requested so use _PYTHON instead.
129 + local _PYTHON
130 +
131 for var; do
132 case "${var}" in
133 EPYTHON)
134 @@ -387,21 +408,21 @@ python_export() {
135 debug-print "${FUNCNAME}: EPYTHON = ${EPYTHON}"
136 ;;
137 PYTHON)
138 - export PYTHON=${EPREFIX}/usr/bin/${impl}
139 + python_wrapper_setup ${impl} PYTHON
140 debug-print "${FUNCNAME}: PYTHON = ${PYTHON}"
141 ;;
142 PYTHON_SITEDIR)
143 - [[ -n ${PYTHON} ]] || die "PYTHON needs to be set for ${var} to be exported, or requested before it"
144 + python_wrapper_setup ${impl} _PYTHON
145 # sysconfig can't be used because:
146 # 1) pypy doesn't give site-packages but stdlib
147 # 2) jython gives paths with wrong case
148 - PYTHON_SITEDIR=$("${PYTHON}" -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib())') || die
149 + PYTHON_SITEDIR=$("${_PYTHON}" -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib())') || die
150 export PYTHON_SITEDIR
151 debug-print "${FUNCNAME}: PYTHON_SITEDIR = ${PYTHON_SITEDIR}"
152 ;;
153 PYTHON_INCLUDEDIR)
154 - [[ -n ${PYTHON} ]] || die "PYTHON needs to be set for ${var} to be exported, or requested before it"
155 - PYTHON_INCLUDEDIR=$("${PYTHON}" -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_inc())') || die
156 + python_wrapper_setup ${impl} _PYTHON
157 + PYTHON_INCLUDEDIR=${SYSROOT}$("${_PYTHON}" -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_inc())') || die
158 export PYTHON_INCLUDEDIR
159 debug-print "${FUNCNAME}: PYTHON_INCLUDEDIR = ${PYTHON_INCLUDEDIR}"
160
161 @@ -411,8 +432,8 @@ python_export() {
162 fi
163 ;;
164 PYTHON_LIBPATH)
165 - [[ -n ${PYTHON} ]] || die "PYTHON needs to be set for ${var} to be exported, or requested before it"
166 - PYTHON_LIBPATH=$("${PYTHON}" -c 'import os.path, sysconfig; print(os.path.join(sysconfig.get_config_var("LIBDIR"), sysconfig.get_config_var("LDLIBRARY")) if sysconfig.get_config_var("LDLIBRARY") else "")') || die
167 + python_wrapper_setup ${impl} _PYTHON
168 + PYTHON_LIBPATH=${SYSROOT}$("${_PYTHON}" -c 'import os.path, sysconfig; print(os.path.join(sysconfig.get_config_var("LIBDIR"), sysconfig.get_config_var("LDLIBRARY")) if sysconfig.get_config_var("LDLIBRARY") else "")') || die
169 export PYTHON_LIBPATH
170 debug-print "${FUNCNAME}: PYTHON_LIBPATH = ${PYTHON_LIBPATH}"
171
172 @@ -457,9 +478,9 @@ python_export() {
173
174 case "${impl}" in
175 python*)
176 - [[ -n ${PYTHON} ]] || die "PYTHON needs to be set for ${var} to be exported, or requested before it"
177 - flags=$("${PYTHON}" -c 'import sysconfig; print(sysconfig.get_config_var("ABIFLAGS") or "")') || die
178 - val=${PYTHON}${flags}-config
179 + python_wrapper_setup ${impl} _PYTHON
180 + flags=$("${_PYTHON}" -c 'import sysconfig; print(sysconfig.get_config_var("ABIFLAGS") or "")') || die
181 + val=${ESYSROOT-${SYSROOT}${EPREFIX}}/usr/bin/${PYTHON##*/}${flags}-config
182 ;;
183 *)
184 die "${impl}: obtaining ${var} not supported"
185 @@ -954,7 +975,7 @@ python_doheader() {
186 local d PYTHON_INCLUDEDIR=${PYTHON_INCLUDEDIR}
187 [[ ${PYTHON_INCLUDEDIR} ]] || python_export PYTHON_INCLUDEDIR
188
189 - d=${PYTHON_INCLUDEDIR#${EPREFIX}}
190 + d=${PYTHON_INCLUDEDIR#${ESYSROOT-${SYSROOT}${EPREFIX}}}
191
192 (
193 insopts -m 0644
194 @@ -964,7 +985,7 @@ python_doheader() {
195 }
196
197 # @FUNCTION: python_wrapper_setup
198 -# @USAGE: [<path> [<impl>]]
199 +# @USAGE: [<impl> [<pyvar> [<path>]]]
200 # @DESCRIPTION:
201 # Create proper 'python' executable and pkg-config wrappers
202 # (if available) in the directory named by <path>. Set up PATH
203 @@ -973,6 +994,9 @@ python_doheader() {
204 # The wrappers will be created for implementation named by <impl>,
205 # or for one named by ${EPYTHON} if no <impl> passed.
206 #
207 +# The path to the 'python' executable wrapper is exported to the
208 +# variable named <pyvar> if given.
209 +#
210 # If the named directory contains a python symlink already, it will
211 # be assumed to contain proper wrappers already and only environment
212 # setup will be done. If wrapper update is requested, the directory
213 @@ -980,25 +1004,41 @@ python_doheader() {
214 python_wrapper_setup() {
215 debug-print-function ${FUNCNAME} "${@}"
216
217 - local impl=${2:-${EPYTHON}}
218 - local workdir=${1:-${T}/${impl}}
219 + local impl=${1:-${EPYTHON}}
220 + local pyvar=${2}
221 + local lpyvar=_${pyvar:-PYTHON}
222 +
223 + # Use separate directories for SYSROOT in case we need to execute
224 + # Python in the context of the build host by unsetting SYSROOT.
225 + local workdir=${3:-${T}/${impl}${SYSROOT:+-sysroot}}
226
227 [[ ${workdir} ]] || die "${FUNCNAME}: no workdir specified."
228 [[ ${impl} ]] || die "${FUNCNAME}: no impl nor EPYTHON specified."
229
230 + local EPYTHON
231 + python_export "${impl}" EPYTHON
232 +
233 + # We could use BROOT here but using the PATH accommodates
234 + # cross-prefix where the PATH is sometimes manipulated to prefer
235 + # build tools from the target prefix (i.e. EPREFIX).
236 + #
237 + # Also make sure we don't pick up an existing wrapper by replacing
238 + # instances of ${T} with a bogus location. The workdir can be
239 + # overridden but hopefully it will be somewhere under ${T}.
240 + local ${lpyvar}=$(PATH=${PATH//${T}//dev/null} type -P "${EPYTHON}" || die "${FUNCNAME}: can't find ${EPYTHON} in PATH")
241 +
242 + local pysysroot=${ESYSROOT-${SYSROOT%/}${EPREFIX}}
243 +
244 if [[ ! -x ${workdir}/bin/python ]]; then
245 _python_check_dead_variables
246
247 - mkdir -p "${workdir}"/{bin,pkgconfig} || die
248 + mkdir -p "${workdir}"/{bin,lib,pkgconfig} || die
249
250 # Clean up, in case we were supposed to do a cheap update.
251 rm -f "${workdir}"/bin/python{,2,3}{,-config} || die
252 rm -f "${workdir}"/bin/2to3 || die
253 rm -f "${workdir}"/pkgconfig/python{,2,3}.pc || die
254
255 - local EPYTHON PYTHON
256 - python_export "${impl}" EPYTHON PYTHON
257 -
258 local pyver pyother
259 if python_is_python3; then
260 pyver=3
261 @@ -1012,37 +1052,53 @@ python_wrapper_setup() {
262 # note: we don't use symlinks because python likes to do some
263 # symlink reading magic that breaks stuff
264 # https://bugs.gentoo.org/show_bug.cgi?id=555752
265 - cat > "${workdir}/bin/python" <<-_EOF_ || die
266 - #!/bin/sh
267 - exec "${PYTHON}" "\${@}"
268 - _EOF_
269 - cp "${workdir}/bin/python" "${workdir}/bin/python${pyver}" || die
270 - chmod +x "${workdir}/bin/python" "${workdir}/bin/python${pyver}" || die
271 + echo '#!/bin/sh' > "${workdir}/bin/python" || die
272
273 local nonsupp=( "python${pyother}" "python${pyother}-config" )
274
275 # CPython-specific
276 if [[ ${EPYTHON} == python* ]]; then
277 + local pysysrootlib=${pysysroot}/usr/$(get_libdir)
278 +
279 cat > "${workdir}/bin/python-config" <<-_EOF_ || die
280 #!/bin/sh
281 - exec "${PYTHON}-config" "\${@}"
282 + exec "${pysysroot}/usr/bin/${EPYTHON}-config" "\${@}"
283 _EOF_
284 cp "${workdir}/bin/python-config" \
285 "${workdir}/bin/python${pyver}-config" || die
286 chmod +x "${workdir}/bin/python-config" \
287 "${workdir}/bin/python${pyver}-config" || die
288
289 + if [[ -n ${pysysroot} ]]; then
290 + # Use host-specific data from SYSROOT. Python 2 looks
291 + # for _sysconfigdata while Python 3 uses
292 + # _PYTHON_SYSCONFIGDATA_NAME.
293 + ln -s "${pysysrootlib}/${EPYTHON}"/_sysconfigdata*.py "${workdir}"/lib/_sysconfigdata.py || die
294 +
295 + # Use distutils/sysconfig from SYSROOT as parts of it
296 + # have GENTOO_LIBDIR baked in at Python build time.
297 + ln -s "${pysysrootlib}/${EPYTHON}"/{distutils,sysconfig.py} "${workdir}"/lib/ || die
298 +
299 + # Add env vars to python wrapper accordingly.
300 + echo "PYTHONPATH=\"${workdir}/lib\${PYTHONPATH:+:}\${PYTHONPATH}\" _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata \\" \
301 + >> "${workdir}/bin/python" || die
302 + fi
303 +
304 # Python 2.6+.
305 - ln -s "${PYTHON/python/2to3-}" "${workdir}"/bin/2to3 || die
306 + ln -s "${EPYTHON/python/2to3-}" "${workdir}"/bin/2to3 || die
307
308 # Python 2.7+.
309 - ln -s "${EPREFIX}"/usr/$(get_libdir)/pkgconfig/${EPYTHON/n/n-}.pc \
310 + ln -s "${pysysrootlib}"/pkgconfig/${EPYTHON/n/n-}.pc \
311 "${workdir}"/pkgconfig/python.pc || die
312 ln -s python.pc "${workdir}"/pkgconfig/python${pyver}.pc || die
313 else
314 nonsupp+=( 2to3 python-config "python${pyver}-config" )
315 fi
316
317 + echo "exec \"${!lpyvar}\" \"\${@}\"" >> "${workdir}"/bin/python || die
318 + tee "${workdir}"/bin/{python${pyver},"${EPYTHON}"} < "${workdir}"/bin/python >/dev/null || die
319 + chmod +x "${workdir}"/bin/{python,python${pyver},"${EPYTHON}"} || die
320 +
321 local x
322 for x in "${nonsupp[@]}"; do
323 cat >"${workdir}"/bin/${x} <<-_EOF_ || die
324 @@ -1064,6 +1120,14 @@ python_wrapper_setup() {
325 PKG_CONFIG_PATH=${workdir}/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}
326 fi
327 export PATH PKG_CONFIG_PATH
328 +
329 + if [[ -n ${pyvar} ]]; then
330 + if [[ -n ${pysysroot} ]]; then
331 + export -- ${pyvar}=${workdir}/bin/${EPYTHON}
332 + else
333 + export -- ${pyvar}=${!lpyvar}
334 + fi
335 + fi
336 }
337
338 # @FUNCTION: python_is_python3
339 --
340 2.19.2

Replies