1 |
Example output for maven-bin-3.6.2 (bug 704618): |
2 |
|
3 |
* QA Notice: Unresolved soname dependencies: |
4 |
* |
5 |
* /usr/share/maven-bin-3.6/lib/jansi-native/freebsd32/libjansi.so: libc.so.7 libutil.so.9 |
6 |
* /usr/share/maven-bin-3.6/lib/jansi-native/freebsd64/libjansi.so: libc.so.7 libutil.so.9 |
7 |
* |
8 |
|
9 |
This warning comes up when a library or executable has one or |
10 |
more soname dependencies (found in its NEEDED.ELF.2 metadata) |
11 |
that could not be resolved by usual means. If you run ldd |
12 |
on files like these then it will report a "not found" error |
13 |
for each unresolved soname dependency. In order to correct |
14 |
problems with soname dependency resolution, use one or more |
15 |
of the approaches described in the following sections. |
16 |
|
17 |
Content of the NEEDED.ELF.2 metadata file may be useful |
18 |
for debugging purposes. Find the NEEDED.ELF.2 file in the |
19 |
${D}/../build-info/ directory after the ebuild src_install |
20 |
phase completes, or in the /var/db/pkg/*/*/ directory for an |
21 |
installed package. Each line of the NEEDED.ELF.2 file contains |
22 |
semicolon separated values for a single ELF file. The soname |
23 |
dependencies are found in the DT_NEEDED column: |
24 |
|
25 |
E_MACHINE;path;DT_SONAME;DT_RUNPATH;DT_NEEDED;multilib category |
26 |
|
27 |
External dependencies |
28 |
|
29 |
For packages that install pre-built binaries, it may be possible |
30 |
to resolve soname dependencies simply by adding dependencies |
31 |
for one or more other packages that are known to provide the |
32 |
needed sonames. |
33 |
|
34 |
Removal of unecessary files |
35 |
|
36 |
For packages that install pre-built binaries, it may be possible |
37 |
to resolve soname dependencies simply by removing unnecessary |
38 |
files which have unresolved soname dependencies. For example, |
39 |
some pre-built binary packages include binaries intended for |
40 |
irrelevant architectures or operating systems, and these files |
41 |
can simply be removed because they are unnecessary. |
42 |
|
43 |
Addition of DT_RUNPATH entries |
44 |
|
45 |
If the relevant dependencies are installed in a location that |
46 |
is not included in the dynamic linker search path, then it's |
47 |
necessary for files to include a DT_RUNPATH entry which refers |
48 |
to the appropriate directory. The special $ORIGIN value can |
49 |
be used to create a relative path reference in DT_RUNPATH, |
50 |
where $ORIGIN is a placeholder for the directory where the |
51 |
file having the DT_RUNPATH entry is located. |
52 |
|
53 |
For pre-built binaries, it may be necessary to fix up |
54 |
DT_RUNPATH using patchelf --set-rpath. For example, use |
55 |
patchelf --set-rpath '$ORIGIN' if a given binary should link |
56 |
to libraries found in the same directory as the binary itself, |
57 |
or use patchelf --set-rpath '$ORIGIN/libs' if a given binary |
58 |
should link to libraries found in a subdirectory named libs |
59 |
found in the same directory as the binary itself. |
60 |
|
61 |
For binaries built from source, a flag like |
62 |
-Wl,-rpath,/path/of/directory/containing/libs will create |
63 |
binaries with the desired DT_RUNPATH entry. |
64 |
|
65 |
Bug: https://bugs.gentoo.org/704320 |
66 |
Signed-off-by: Zac Medico <zmedico@g.o> |
67 |
--- |
68 |
doc/qa.docbook | 74 +++++++++++++++ |
69 |
lib/_emerge/EbuildPhase.py | 63 ++++++++++--- |
70 |
lib/portage/dep/soname/SonameAtom.py | 12 ++- |
71 |
lib/portage/util/_dyn_libs/soname_deps_qa.py | 98 ++++++++++++++++++++ |
72 |
4 files changed, 232 insertions(+), 15 deletions(-) |
73 |
create mode 100644 lib/portage/util/_dyn_libs/soname_deps_qa.py |
74 |
|
75 |
diff --git a/doc/qa.docbook b/doc/qa.docbook |
76 |
index 28ff6cf8e..cf8a3bc8d 100644 |
77 |
--- a/doc/qa.docbook |
78 |
+++ b/doc/qa.docbook |
79 |
@@ -127,6 +127,80 @@ |
80 |
</para> |
81 |
</sect1> |
82 |
|
83 |
+ <sect1 id='qa-unresolved-soname-dependencies'> |
84 |
+ <title>Unresolved soname dependencies</title> |
85 |
+ <para> |
86 |
+ <programlisting> |
87 |
+ QA Notice: Unresolved soname dependencies |
88 |
+ </programlisting> |
89 |
+ </para> |
90 |
+ <para> |
91 |
+ This warning comes up when a library or executable has one or more |
92 |
+ soname dependencies (found in its NEEDED.ELF.2 metadata) that could |
93 |
+ not be resolved by usual means. If you run <command>ldd</command> on |
94 |
+ files like these then it will report a "not found" error for each |
95 |
+ unresolved soname dependency. In order to correct problems with |
96 |
+ soname dependency resolution, use one or more of the approaches |
97 |
+ described in the following sections. |
98 |
+ </para> |
99 |
+ <para> |
100 |
+ Content of the NEEDED.ELF.2 metadata file may be useful for |
101 |
+ debugging purposes. Find the NEEDED.ELF.2 file in the |
102 |
+ ${D}/../build-info/ directory after the ebuild src_install phase |
103 |
+ completes, or in the /var/db/pkg/*/*/ directory for an installed |
104 |
+ package. Each line of the NEEDED.ELF.2 file contains semicolon |
105 |
+ separated values for a single ELF file. The soname dependencies are |
106 |
+ found in the DT_NEEDED column: |
107 |
+ <programlisting> |
108 |
+ E_MACHINE;path;DT_SONAME;DT_RUNPATH;DT_NEEDED;multilib category |
109 |
+ </programlisting> |
110 |
+ </para> |
111 |
+ <sect2 id='qa-unresolved-soname-dependencies-resolved-bu-external-dependencies'> |
112 |
+ <title>External dependencies</title> |
113 |
+ <para> |
114 |
+ For packages that install pre-built binaries, it may be possible to |
115 |
+ resolve soname dependencies simply by adding dependencies for one |
116 |
+ or more other packages that are known to provide the needed sonames. |
117 |
+ </para> |
118 |
+ </sect2> |
119 |
+ <sect2 id='qa-unresolved-soname-dependencies-resolved-by-removal-of-unecessary-files'> |
120 |
+ <title>Removal of unecessary files</title> |
121 |
+ <para> |
122 |
+ For packages that install pre-built binaries, it may be possible to |
123 |
+ resolve soname dependencies simply by removing unnecessary files |
124 |
+ which have unresolved soname dependencies. For example, some pre-built |
125 |
+ binary packages include binaries intended for irrelevant architectures |
126 |
+ or operating systems, and these files can simply be removed because |
127 |
+ they are unnecessary. |
128 |
+ </para> |
129 |
+ </sect2> |
130 |
+ <sect2 id='qa-unresolved-soname-dependencies-resolved-by-addition-of-dt-runpath-entries'> |
131 |
+ <title>Addition of DT_RUNPATH entries</title> |
132 |
+ <para> |
133 |
+ If the relevant dependencies are installed in a location that is not |
134 |
+ included in the dynamic linker search path, then it's necessary for |
135 |
+ files to include a DT_RUNPATH entry which refers to the appropriate |
136 |
+ directory. The special $ORIGIN value can be used to create a relative |
137 |
+ path reference in DT_RUNPATH, where $ORIGIN is a placeholder for the |
138 |
+ directory where the file having the DT_RUNPATH entry is located. |
139 |
+ </para> |
140 |
+ <para> |
141 |
+ For pre-built binaries, it may be necessary to fix up DT_RUNPATH using |
142 |
+ <command>patchelf --set-rpath</command>. For example, use |
143 |
+ <command>patchelf --set-rpath '$ORIGIN'</command> if a given binary |
144 |
+ should link to libraries found in the same directory as the binary |
145 |
+ itself, or use <command>patchelf --set-rpath '$ORIGIN/libs'</command> |
146 |
+ if a given binary should link to libraries found in a subdirectory |
147 |
+ named libs found in the same directory as the binary itself. |
148 |
+ </para> |
149 |
+ <para> |
150 |
+ For binaries built from source, a flag like |
151 |
+ <option>-Wl,-rpath,/path/of/directory/containing/libs</option> will |
152 |
+ create binaries with the desired DT_RUNPATH entry. |
153 |
+ </para> |
154 |
+ </sect2> |
155 |
+ </sect1> |
156 |
+ |
157 |
<sect1 id='qa-abs-lib-link'> |
158 |
<title>Absolute Symlink In Library Directory</title> |
159 |
<para> |
160 |
diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py |
161 |
index 50e3dd1f4..828e6e126 100644 |
162 |
--- a/lib/_emerge/EbuildPhase.py |
163 |
+++ b/lib/_emerge/EbuildPhase.py |
164 |
@@ -1,6 +1,8 @@ |
165 |
-# Copyright 1999-2018 Gentoo Foundation |
166 |
+# Copyright 1999-2020 Gentoo Authors |
167 |
# Distributed under the terms of the GNU General Public License v2 |
168 |
|
169 |
+from __future__ import unicode_literals |
170 |
+ |
171 |
import functools |
172 |
import gzip |
173 |
import io |
174 |
@@ -14,10 +16,17 @@ from _emerge.EbuildProcess import EbuildProcess |
175 |
from _emerge.CompositeTask import CompositeTask |
176 |
from _emerge.PackagePhase import PackagePhase |
177 |
from _emerge.TaskSequence import TaskSequence |
178 |
+from portage.package.ebuild._ipc.QueryCommand import QueryCommand |
179 |
+from portage.util._dyn_libs.soname_deps_qa import ( |
180 |
+ _get_all_provides, |
181 |
+ _get_unresolved_soname_deps, |
182 |
+) |
183 |
from portage.package.ebuild.prepare_build_dirs import (_prepare_workdir, |
184 |
_prepare_fake_distdir, _prepare_fake_filesdir) |
185 |
+from portage.util.futures.compat_coroutine import coroutine |
186 |
from portage.util import writemsg |
187 |
from portage.util._async.AsyncTaskFuture import AsyncTaskFuture |
188 |
+from portage.util.futures.executor.fork import ForkExecutor |
189 |
|
190 |
try: |
191 |
from portage.xml.metadata import MetaDataXML |
192 |
@@ -281,7 +290,7 @@ class EbuildPhase(CompositeTask): |
193 |
fd, logfile = tempfile.mkstemp() |
194 |
os.close(fd) |
195 |
post_phase = _PostPhaseCommands(background=self.background, |
196 |
- commands=post_phase_cmds, fd_pipes=self.fd_pipes, |
197 |
+ commands=post_phase_cmds, elog=self._elog, fd_pipes=self.fd_pipes, |
198 |
logfile=logfile, phase=self.phase, scheduler=self.scheduler, |
199 |
settings=settings) |
200 |
self._start_task(post_phase, self._post_phase_exit) |
201 |
@@ -315,13 +324,6 @@ class EbuildPhase(CompositeTask): |
202 |
self._die_hooks() |
203 |
return |
204 |
|
205 |
- if self.phase == "install": |
206 |
- out = io.StringIO() |
207 |
- _post_src_install_soname_symlinks(self.settings, out) |
208 |
- msg = out.getvalue() |
209 |
- if msg: |
210 |
- self.scheduler.output(msg, log_path=log_path) |
211 |
- |
212 |
self._current_task = None |
213 |
self.wait() |
214 |
return |
215 |
@@ -414,7 +416,7 @@ class EbuildPhase(CompositeTask): |
216 |
|
217 |
class _PostPhaseCommands(CompositeTask): |
218 |
|
219 |
- __slots__ = ("commands", "fd_pipes", "logfile", "phase", "settings") |
220 |
+ __slots__ = ("commands", "elog", "fd_pipes", "logfile", "phase", "settings") |
221 |
|
222 |
def _start(self): |
223 |
if isinstance(self.commands, list): |
224 |
@@ -436,4 +438,43 @@ class _PostPhaseCommands(CompositeTask): |
225 |
logfile=self.logfile, phase=self.phase, |
226 |
scheduler=self.scheduler, settings=self.settings, **kwargs)) |
227 |
|
228 |
- self._start_task(tasks, self._default_final_exit) |
229 |
+ self._start_task(tasks, self._commands_exit) |
230 |
+ |
231 |
+ def _commands_exit(self, task): |
232 |
+ |
233 |
+ if self._default_exit(task) != os.EX_OK: |
234 |
+ self._async_wait() |
235 |
+ return |
236 |
+ |
237 |
+ if self.phase == 'install': |
238 |
+ out = io.StringIO() |
239 |
+ _post_src_install_soname_symlinks(self.settings, out) |
240 |
+ msg = out.getvalue() |
241 |
+ if msg: |
242 |
+ self.scheduler.output(msg, log_path=self.settings.get("PORTAGE_LOG_FILE")) |
243 |
+ |
244 |
+ # This operates on REQUIRES metadata generated by the above function call. |
245 |
+ future = self._soname_deps_qa() |
246 |
+ # If an unexpected exception occurs, then this will raise it. |
247 |
+ future.add_done_callback(lambda future: future.result()) |
248 |
+ self._start_task(AsyncTaskFuture(future=future), self._default_final_exit) |
249 |
+ else: |
250 |
+ self._default_final_exit(task) |
251 |
+ |
252 |
+ @coroutine |
253 |
+ def _soname_deps_qa(self): |
254 |
+ |
255 |
+ vardb = QueryCommand.get_db()[self.settings['EROOT']]['vartree'].dbapi |
256 |
+ |
257 |
+ all_provides = (yield self.scheduler.run_in_executor(ForkExecutor(loop=self.scheduler), _get_all_provides, vardb)) |
258 |
+ |
259 |
+ unresolved = _get_unresolved_soname_deps(os.path.join(self.settings['PORTAGE_BUILDDIR'], 'build-info'), all_provides) |
260 |
+ |
261 |
+ if unresolved: |
262 |
+ unresolved.sort() |
263 |
+ qa_msg = ["QA Notice: Unresolved soname dependencies:"] |
264 |
+ qa_msg.append("") |
265 |
+ qa_msg.extend("\t%s: %s" % (filename, " ".join(sorted(soname_deps))) |
266 |
+ for filename, soname_deps in unresolved) |
267 |
+ qa_msg.append("") |
268 |
+ self.elog("eqawarn", qa_msg) |
269 |
diff --git a/lib/portage/dep/soname/SonameAtom.py b/lib/portage/dep/soname/SonameAtom.py |
270 |
index a7dad973d..f32082568 100644 |
271 |
--- a/lib/portage/dep/soname/SonameAtom.py |
272 |
+++ b/lib/portage/dep/soname/SonameAtom.py |
273 |
@@ -1,4 +1,4 @@ |
274 |
-# Copyright 2015 Gentoo Foundation |
275 |
+# Copyright 2015-2020 Gentoo Authors |
276 |
# Distributed under the terms of the GNU General Public License v2 |
277 |
|
278 |
from __future__ import unicode_literals |
279 |
@@ -10,7 +10,7 @@ from portage import _encodings, _unicode_encode |
280 |
class SonameAtom(object): |
281 |
|
282 |
__slots__ = ("multilib_category", "soname", "_hash_key", |
283 |
- "_hash_value") |
284 |
+ "_hash_value", "_immutable") |
285 |
|
286 |
# Distiguishes package atoms from other atom types |
287 |
package = False |
288 |
@@ -21,10 +21,14 @@ class SonameAtom(object): |
289 |
object.__setattr__(self, "_hash_key", |
290 |
(multilib_category, soname)) |
291 |
object.__setattr__(self, "_hash_value", hash(self._hash_key)) |
292 |
+ object.__setattr__(self, "_immutable", True) |
293 |
|
294 |
def __setattr__(self, name, value): |
295 |
- raise AttributeError("SonameAtom instances are immutable", |
296 |
- self.__class__, name, value) |
297 |
+ if getattr(self, '_immutable', False): |
298 |
+ raise AttributeError("SonameAtom instances are immutable", |
299 |
+ self.__class__, name, value) |
300 |
+ # This is needed for unpickling. |
301 |
+ object.__setattr__(self, name, value) |
302 |
|
303 |
def __hash__(self): |
304 |
return self._hash_value |
305 |
diff --git a/lib/portage/util/_dyn_libs/soname_deps_qa.py b/lib/portage/util/_dyn_libs/soname_deps_qa.py |
306 |
new file mode 100644 |
307 |
index 000000000..6840bb602 |
308 |
--- /dev/null |
309 |
+++ b/lib/portage/util/_dyn_libs/soname_deps_qa.py |
310 |
@@ -0,0 +1,98 @@ |
311 |
+# Copyright 2020 Gentoo Authors |
312 |
+# Distributed under the terms of the GNU General Public License v2 |
313 |
+ |
314 |
+import io |
315 |
+ |
316 |
+from portage import ( |
317 |
+ _encodings, |
318 |
+ _unicode_encode, |
319 |
+ os, |
320 |
+) |
321 |
+from portage.dep.soname.parse import parse_soname_deps |
322 |
+from portage.util._dyn_libs.NeededEntry import NeededEntry |
323 |
+ |
324 |
+ |
325 |
+def _get_all_provides(vardb): |
326 |
+ """ |
327 |
+ Get all of the sonames provided by all of the installed packages. |
328 |
+ This does not bother to acquire a lock, since its pretty safe to |
329 |
+ assume that any packages merged or unmerged while this function |
330 |
+ is running must be irrelevant. |
331 |
+ |
332 |
+ @param vardb: an installed package database |
333 |
+ @type vardb: vardbapi |
334 |
+ @rtype: frozenset |
335 |
+ @return: a frozenset od SonameAtom instances provided by all |
336 |
+ installed packages |
337 |
+ """ |
338 |
+ |
339 |
+ all_provides = [] |
340 |
+ |
341 |
+ for cpv in vardb.cpv_all(): |
342 |
+ try: |
343 |
+ provides, = vardb.aux_get(cpv, ['PROVIDES']) |
344 |
+ except KeyError: |
345 |
+ # Since we don't hold a lock, assume this is due to a |
346 |
+ # concurrent unmerge, and PROVIDES from the unmerged package |
347 |
+ # are most likely negligible due to topologically sorted |
348 |
+ # merge order. Also, note that it's possible for aux_get |
349 |
+ # to succeed and return empty PROVIDES metadata if the file |
350 |
+ # disappears (due to unmerge) before it can be read. |
351 |
+ pass |
352 |
+ else: |
353 |
+ if provides: |
354 |
+ all_provides.extend(parse_soname_deps(provides)) |
355 |
+ |
356 |
+ return frozenset(all_provides) |
357 |
+ |
358 |
+ |
359 |
+def _get_unresolved_soname_deps(metadata_dir, all_provides): |
360 |
+ """ |
361 |
+ Get files with unresolved soname dependencies. |
362 |
+ |
363 |
+ @param metadata_dir: directory containing package metadata files |
364 |
+ named REQUIRES and NEEDED.ELF.2 |
365 |
+ @type metadata_dir: str |
366 |
+ @param all_provides: a frozenset on SonameAtom instances provided by |
367 |
+ all installed packages |
368 |
+ @type all_provides: frozenset |
369 |
+ @rtype: list |
370 |
+ @return: list of tuple(filename, tuple(unresolved sonames)) |
371 |
+ """ |
372 |
+ try: |
373 |
+ with io.open(_unicode_encode(os.path.join(metadata_dir, 'REQUIRES'), |
374 |
+ encoding=_encodings['fs'], errors='strict'), |
375 |
+ mode='rt', encoding=_encodings['repo.content'], errors='strict') as f: |
376 |
+ requires = frozenset(parse_soname_deps(f.read())) |
377 |
+ except EnvironmentError: |
378 |
+ return [] |
379 |
+ |
380 |
+ unresolved_by_category = {} |
381 |
+ for atom in requires: |
382 |
+ if atom not in all_provides: |
383 |
+ unresolved_by_category.setdefault(atom.multilib_category, set()).add(atom.soname) |
384 |
+ |
385 |
+ needed_filename = os.path.join(metadata_dir, "NEEDED.ELF.2") |
386 |
+ with io.open(_unicode_encode(needed_filename, encoding=_encodings['fs'], errors='strict'), |
387 |
+ mode='rt', encoding=_encodings['repo.content'], errors='strict') as f: |
388 |
+ needed = f.readlines() |
389 |
+ |
390 |
+ unresolved_by_file = [] |
391 |
+ for l in needed: |
392 |
+ l = l.rstrip("\n") |
393 |
+ if not l: |
394 |
+ continue |
395 |
+ entry = NeededEntry.parse(needed_filename, l) |
396 |
+ missing = unresolved_by_category.get(entry.multilib_category) |
397 |
+ if not missing: |
398 |
+ continue |
399 |
+ # NOTE: This can contain some false positives in the case of |
400 |
+ # missing DT_RPATH settings, since it's possible that a subset |
401 |
+ # package files have the desired DT_RPATH settings. However, |
402 |
+ # since reported sonames are unresolved for at least some file(s), |
403 |
+ # false positives or this sort should not be not too annoying. |
404 |
+ missing = [soname for soname in entry.needed if soname in missing] |
405 |
+ if missing: |
406 |
+ unresolved_by_file.append((entry.filename, tuple(missing))) |
407 |
+ |
408 |
+ return unresolved_by_file |
409 |
-- |
410 |
2.21.0 |