Gentoo Archives: gentoo-commits

From: Fabian Groffen <grobian@g.o>
To: gentoo-commits@l.g.o
Subject: [gentoo-commits] proj/portage:prefix commit in: cnf/, lib/portage/util/futures/_asyncio/, man/, lib/portage/, ...
Date: Thu, 06 Dec 2018 12:57:46
Message-Id: 1544101047.ad67685714c348aad22a3819b4d0326822a49425.grobian@gentoo
1 commit: ad67685714c348aad22a3819b4d0326822a49425
2 Author: Fabian Groffen <grobian <AT> gentoo <DOT> org>
3 AuthorDate: Thu Dec 6 12:55:51 2018 +0000
4 Commit: Fabian Groffen <grobian <AT> gentoo <DOT> org>
5 CommitDate: Thu Dec 6 12:57:27 2018 +0000
6 URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=ad676857
7
8 Squashed commit of the following due to GCO sign-off sever check:
9
10 commit c9cd57da216f21de49f8a57a4a692f8116d27ead
11 Author: Zac Medico <zmedico <AT> gentoo.org>
12 Date: Tue Aug 7 21:04:19 2018 -0700
13
14 Updates for portage-2.3.45 release
15
16 commit 807ac3d9d6eecead73f59d399b30559e5c731587
17 Author: Michael Orlitzky <mjo <AT> gentoo.org>
18 Date: Tue Aug 7 12:46:04 2018 -0400
19
20 bin/install-qa-check.d: add new 90bad-bin-group-write QA check.
21
22 System executables that are writable by a non-root user pose a
23 security risk. Anyone who can write to an executable can change its
24 behavior. If that executable is later run with elevated privileges
25 (say, by root, when the machine starts), then the non-root user can
26 escalate his own privileges to those of the person running the
27 modified executable.
28
29 The 90bad-bin-owner check already addresses one cause for a non-root
30 user to be able to modify an executable: because he owns it. This
31 commit adds another check, to ensure that no non-root *groups* have
32 write access to any system executables. On a "normal" system, all
33 system executables should be writable only by the super-user's group,
34 if any. To avoid false-positives, non-"normal" systems (like prefix)
35 are skipped.
36
37 Closes: https://bugs.gentoo.org/629398
38
39 commit 04e71a831bc42f2a0de1694dd2013eac0414e007
40 Author: Michael Orlitzky <mjo <AT> gentoo.org>
41 Date: Tue Aug 7 12:46:03 2018 -0400
42
43 bin/install-qa-check.d: add new 90bad-bin-owner QA check.
44
45 System executables that are not owned by root pose a security
46 risk. The owner of the executable is free to modify it at any time;
47 so, for example, he can change a daemon's behavior to make it
48 malicious before the next time the service is started (usually by
49 root).
50
51 On a "normal" system, the superuser should own every system executable
52 (even setuid ones, for security reasons). This commit adds a new
53 install-time check that reports any such binaries with a QA
54 warning. To avoid false positives, non-"normal" systems (like prefix)
55 are skipped at the moment.
56
57 Bug: https://bugs.gentoo.org/629398
58
59 commit eb81c8081ae53e862f9f0f82c6119f936ca896e7
60 Author: Zac Medico <zmedico <AT> gentoo.org>
61 Date: Tue Aug 7 11:29:51 2018 -0700
62
63 action_metadata: refresh eclass cache (bug 663022)
64
65 After sync, the eclass cache can contain stale data, therefore
66 use the update_eclasses method to refresh it.
67
68 Bug: https://bugs.gentoo.org/663022
69
70 commit 3f462e983b2e08107e0f7e4a2753d8deca4ac62a
71 Author: Zac Medico <zmedico <AT> gentoo.org>
72 Date: Mon Aug 6 16:38:55 2018 -0700
73
74 Refer to "ebuild repository" rather than "portage tree"
75
76 commit 78c52aa79707756d3b816fdbfa9004f7b9dcac41
77 Author: Zac Medico <zmedico <AT> gentoo.org>
78 Date: Mon Aug 6 11:42:29 2018 -0700
79
80 repoman: remove obsolete herds checks
81
82 commit 6b4252d3a0f12808a5bcce888b7f68e1f84b5301
83 Author: Zac Medico <zmedico <AT> gentoo.org>
84 Date: Sat Jul 28 14:22:42 2018 -0700
85
86 Add asyncio.create_subprocess_exec support for python2 (bug 662388)
87
88 The asyncio.create_subprocess_exec function is essential for
89 using subprocesses in coroutines, so add support to do this
90 for python2. This paves the way for extensive use of coroutines
91 in portage, since coroutines are well-suited for many portage
92 tasks that involve subprocesses.
93
94 Bug: https://bugs.gentoo.org/662388
95
96 commit f391b2cc5384fc38e99a0598cb3de2346e297c25
97 Author: Zac Medico <zmedico <AT> gentoo.org>
98 Date: Sat Aug 4 13:18:47 2018 -0700
99
100 compression_probe: decompress zstd --long=31 (bug 634980)
101
102 In order to decompress files compressed with zstd --long=31, add
103 --long=31 to the zstd decompress options. Even though zstd compression
104 does not support --long=31 on 32-bit platforms, decompression with
105 --long=31 still works as long as the file was compressed with a
106 smaller windowLog.
107
108 Reported-by: Martin Väth <martin <AT> mvath.de>
109 Bug: https://bugs.gentoo.org/634980
110
111 commit 12aa832c91a6f2c2faee5a2e839b7f1ed5a0ee6e
112 Author: Francesco Turco <fturco <AT> fastmail.fm>
113 Date: Sat Aug 4 20:47:40 2018 +0200
114
115 install-qa-check.d: fix hardened <AT> g.o e-mail address
116
117 Closes: https://github.com/gentoo/portage/pull/350
118
119 Signed-off-by: Fabian Groffen <grobian <AT> gentoo.org>
120
121 README | 4 +-
122 RELEASE-NOTES | 11 ++
123 bin/ebuild | 2 +-
124 bin/ebuild.sh | 2 +-
125 bin/install-qa-check.d/10executable-issues | 2 +-
126 bin/install-qa-check.d/90bad-bin-group-write | 55 ++++++
127 bin/install-qa-check.d/90bad-bin-owner | 48 ++++++
128 cnf/make.conf.example | 8 +-
129 lib/_emerge/actions.py | 2 +-
130 lib/_emerge/create_world_atom.py | 2 +-
131 lib/_emerge/depgraph.py | 2 +-
132 lib/_emerge/search.py | 2 +-
133 lib/portage/__init__.py | 2 +-
134 lib/portage/_global_updates.py | 2 +-
135 lib/portage/dbapi/porttree.py | 2 +-
136 lib/portage/dbapi/vartree.py | 2 +-
137 lib/portage/emaint/modules/merges/merges.py | 2 +-
138 lib/portage/metadata.py | 6 +-
139 lib/portage/news.py | 2 +-
140 lib/portage/package/ebuild/config.py | 2 +-
141 lib/portage/sync/modules/rsync/rsync.py | 2 +-
142 .../util/futures/asyncio/test_subprocess_exec.py | 184 ++++++---------------
143 lib/portage/util/compression_probe.py | 2 +-
144 lib/portage/util/futures/_asyncio/__init__.py | 53 ++++++
145 lib/portage/util/futures/_asyncio/process.py | 107 ++++++++++++
146 lib/portage/util/futures/_asyncio/streams.py | 96 +++++++++++
147 lib/portage/util/futures/compat_coroutine.py | 6 +-
148 man/emerge.1 | 10 +-
149 man/make.conf.5 | 2 +-
150 man/portage.5 | 6 +-
151 repoman/README | 2 +-
152 repoman/lib/repoman/__init__.py | 2 +-
153 repoman/lib/repoman/checks/herds/__init__.py | 0
154 repoman/lib/repoman/checks/herds/herdbase.py | 135 ---------------
155 repoman/lib/repoman/checks/herds/metadata.py | 26 ---
156 repoman/lib/repoman/utilities.py | 2 +-
157 repoman/man/repoman.1 | 2 +-
158 setup.py | 2 +-
159 38 files changed, 469 insertions(+), 330 deletions(-)
160
161 diff --git a/README b/README
162 index db660f3b8..4ab10c6b5 100644
163 --- a/README
164 +++ b/README
165 @@ -12,8 +12,8 @@ About Portage
166
167 Portage is a package management system based on ports collections. The
168 Package Manager Specification Project (PMS) standardises and documents
169 -the behaviour of Portage so that the Portage tree can be used by other
170 -package managers.
171 +the behaviour of Portage so that ebuild repositories can be used by
172 +other package managers.
173
174
175 Dependencies
176
177 diff --git a/RELEASE-NOTES b/RELEASE-NOTES
178 index b03d0ae3f..bda514fdb 100644
179 --- a/RELEASE-NOTES
180 +++ b/RELEASE-NOTES
181 @@ -1,6 +1,17 @@
182 Release Notes; upgrade information mainly.
183 Features/major bugfixes are listed in NEWS
184
185 +portage-2.3.45
186 +==================================
187 +* Bug Fixes:
188 + - Bug 373209 FEATURES=test/USE=test mapping via USE_ORDER
189 + - Bug 629398 QA Notice for executables writable by non-root user
190 + - Bug 634980 zstd --long=31 binary package decompression support
191 + - Bug 662388 asyncio.create_subprocess_exec support for python2
192 + - Bug 662668 emerge --keep-going AttributeError
193 + - Bug 663022 FileNotFoundError with FEATURES=metadata-transfer
194 +
195 +
196 portage-2.3.44
197 ==================================
198 * Bug Fixes:
199
200 diff --git a/bin/ebuild b/bin/ebuild
201 index b87f2a1d7..f54bf3ce5 100755
202 --- a/bin/ebuild
203 +++ b/bin/ebuild
204 @@ -143,7 +143,7 @@ if not os.path.isabs(ebuild):
205 mycwd = portage.normalize_path(pwd)
206 ebuild = os.path.join(mycwd, ebuild)
207 ebuild = portage.normalize_path(ebuild)
208 -# portdbapi uses the canonical path for the base of the portage tree, but
209 +# portdbapi uses the canonical path for the base of the ebuild repository, but
210 # subdirectories of the base can be built from symlinks (like crossdev does).
211 ebuild_portdir = os.path.realpath(
212 os.path.dirname(os.path.dirname(os.path.dirname(ebuild))))
213
214 diff --git a/bin/ebuild.sh b/bin/ebuild.sh
215 index f76a48d8e..e3c84150e 100755
216 --- a/bin/ebuild.sh
217 +++ b/bin/ebuild.sh
218 @@ -279,7 +279,7 @@ inherit() {
219 then
220 # This is disabled in the *rm phases because they frequently give
221 # false alarms due to INHERITED in /var/db/pkg being outdated
222 - # in comparison the the eclasses from the portage tree. It's
223 + # in comparison to the eclasses from the ebuild repository. It's
224 # disabled for nofetch, since that can be called by repoman and
225 # that triggers bug #407449 due to repoman not exporting
226 # non-essential variables such as INHERITED.
227
228 diff --git a/bin/install-qa-check.d/10executable-issues b/bin/install-qa-check.d/10executable-issues
229 index 8a2c8e875..6b33d281d 100644
230 --- a/bin/install-qa-check.d/10executable-issues
231 +++ b/bin/install-qa-check.d/10executable-issues
232 @@ -128,7 +128,7 @@ elf_check() {
233 eqawarn
234 eqawarn " Please include the following list of files in your report:"
235 eqawarn " Note: Bugs should be filed for the respective maintainers"
236 - eqawarn " of the package in question and not hardened@g.o."
237 + eqawarn " of the package in question and not hardened@g.o."
238 eqawarn "${f}"
239 __vecho -ne '\n'
240 die_msg="${die_msg} execstacks"
241
242 diff --git a/bin/install-qa-check.d/90bad-bin-group-write b/bin/install-qa-check.d/90bad-bin-group-write
243 new file mode 100644
244 index 000000000..786dde712
245 --- /dev/null
246 +++ b/bin/install-qa-check.d/90bad-bin-group-write
247 @@ -0,0 +1,55 @@
248 +# Copyright 1999-2018 Gentoo Foundation
249 +# Distributed under the terms of the GNU General Public License v2
250 +
251 +bad_bin_group_write_check() {
252 + # Warn about globally-installed executables (in /bin, /usr/bin, /sbin,
253 + # /usr/sbin, or /opt/bin) that are group-writable by a nonzero GID.
254 +
255 + # This check doesn't work on non-root prefix installations at
256 + # the moment, because every executable therein is owned by a
257 + # nonzero GID.
258 + [[ "${EUID}" -ne "0" || "${PORTAGE_INST_UID}" -ne "0" ]] && return
259 +
260 + local d f found=()
261 +
262 + for d in "${ED%/}/opt/bin" "${ED%/}/bin" "${ED%/}/usr/bin" \
263 + "${ED%/}/sbin" "${ED%/}/usr/sbin"; do
264 + [[ -d "${d}" ]] || continue
265 +
266 + # Read the results of the "find" command into the "found" array.
267 + #
268 + # Use -L to catch symlinks whose targets are vulnerable,
269 + # even though it won't catch ABSOLUTE symlinks until the package
270 + # is RE-installed (the first time around, the target won't exist).
271 + #
272 + # We match the GID and not the name "root" here because (for
273 + # example) on FreeBSD, the superuser group is "wheel".
274 + #
275 + # We don't make an exception for setguid executables here, because
276 + # a group-writable setguid executable is likely a mistake. By
277 + # altering the contents of the executable, a member of the group
278 + # can allow everyone (i.e. the people running it) to obtain the
279 + # full privileges available to that group. While only existing
280 + # group members can make that choice, it's a decision usually
281 + # limited to the system administrator.
282 + while read -r -d '' f; do
283 + found+=( "${f}" )
284 + done < <(find -L "${d}" \
285 + -maxdepth 1 \
286 + -type f \
287 + -perm /g+w \
288 + ! -gid 0 \
289 + -print0)
290 + done
291 +
292 + if [[ ${found[@]} ]]; then
293 + eqawarn "system executables group-writable by nonzero gid:"
294 + for f in "${found[@]}"; do
295 + # Strip off the leading destdir before outputting the path.
296 + eqawarn " ${f#${D%/}}"
297 + done
298 + fi
299 +}
300 +
301 +bad_bin_group_write_check
302 +:
303
304 diff --git a/bin/install-qa-check.d/90bad-bin-owner b/bin/install-qa-check.d/90bad-bin-owner
305 new file mode 100644
306 index 000000000..c3ee30746
307 --- /dev/null
308 +++ b/bin/install-qa-check.d/90bad-bin-owner
309 @@ -0,0 +1,48 @@
310 +# Copyright 1999-2018 Gentoo Foundation
311 +# Distributed under the terms of the GNU General Public License v2
312 +
313 +bad_bin_owner_check() {
314 + # Warn about globally-installed executables (in /bin, /usr/bin, /sbin,
315 + # /usr/sbin, or /opt/bin) that are owned by a nonzero UID.
316 +
317 + # This check doesn't work on non-root prefix installations at
318 + # the moment, because every executable therein is owned by a
319 + # nonzero UID.
320 + [[ "${EUID}" -ne "0" || "${PORTAGE_INST_UID}" -ne "0" ]] && return
321 +
322 + local d f found=()
323 +
324 + for d in "${ED%/}/opt/bin" "${ED%/}/bin" "${ED%/}/usr/bin" \
325 + "${ED%/}/sbin" "${ED%/}/usr/sbin"; do
326 + [[ -d "${d}" ]] || continue
327 +
328 + # Read the results of the "find" command into the "found" bash array.
329 + #
330 + # Use -L to catch symlinks whose targets are owned by a non-root user,
331 + # even though it won't catch ABSOLUTE symlinks until the package
332 + # is RE-installed (the first time around, the target won't exist).
333 + #
334 + # We do want to list non-superuser setuid executables, because
335 + # they can be exploited. The owner can simply wipe the setuid
336 + # bit, and then alter the contents of the file. The superuser
337 + # will then have a time bomb in his $PATH.
338 + while read -r -d '' f; do
339 + found+=( "${f}" )
340 + done < <(find -L "${d}" \
341 + -maxdepth 1 \
342 + -type f \
343 + ! -uid 0 \
344 + -print0)
345 + done
346 +
347 + if [[ ${found[@]} ]]; then
348 + eqawarn "system executables owned by nonzero uid:"
349 + for f in "${found[@]}"; do
350 + # Strip off the leading destdir before outputting the path.
351 + eqawarn " ${f#${D%/}}"
352 + done
353 + fi
354 +}
355 +
356 +bad_bin_owner_check
357 +:
358
359 diff --git a/cnf/make.conf.example b/cnf/make.conf.example
360 index 6de7a903f..ad290c7e6 100644
361 --- a/cnf/make.conf.example
362 +++ b/cnf/make.conf.example
363 @@ -13,7 +13,7 @@
364 # very extensive set of USE variables described in our USE variable HOWTO at
365 # https://wiki.gentoo.org/wiki/Handbook:X86/Working/USE
366 #
367 -# The available list of use flags with descriptions is in your portage tree.
368 +# The available list of use flags with descriptions is in the ebuild repository.
369 # Use 'less' to view them: --> less /usr/portage/profiles/use.desc <--
370 #
371 # 'ufed' is an ncurses/dialog interface available in portage to make handling
372 @@ -102,7 +102,7 @@
373 # the application being installed.
374 #PORTAGE_TMPDIR=@PORTAGE_EPREFIX@/var/tmp
375 #
376 -# PORTDIR is the location of the portage tree. This is the repository
377 +# PORTDIR is the location of the ebuild repository. This is the repository
378 # for all profile information as well as all ebuilds. If you change
379 # this, you must update your /etc/portage/make.profile symlink accordingly.
380 # ***Warning***
381 @@ -193,7 +193,7 @@
382 # Synchronizing Portage
383 # =====================
384 #
385 -# Each of these settings affects how Gentoo synchronizes your Portage tree.
386 +# Each of these settings affects how Gentoo synchronizes the ebuild repository.
387 # Synchronization is handled by rsync and these settings allow some control
388 # over how it is done.
389 #
390 @@ -224,7 +224,7 @@
391 #SYNC="rsync://rsync.gentoo.org/gentoo-portage"
392 #
393 # PORTAGE_RSYNC_RETRIES sets the number of times portage will attempt to retrieve
394 -# a current portage tree before it exits with an error. This allows
395 +# a current ebuild repository before it exits with an error. This allows
396 # for a more successful retrieval without user intervention most times.
397 # If set to a negative number, then retry until all possible addresses are
398 # exhausted.
399
400 diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
401 index a9856de94..b2410065e 100644
402 --- a/lib/_emerge/actions.py
403 +++ b/lib/_emerge/actions.py
404 @@ -986,7 +986,7 @@ def calc_depclean(settings, trees, ldpath_mtimes,
405 msg.append("")
406 msg.extend(textwrap.wrap(
407 "Also, note that it may be necessary to manually uninstall " + \
408 - "packages that no longer exist in the portage tree, since " + \
409 + "packages that no longer exist in the repository, since " + \
410 "it may not be possible to satisfy their dependencies.", 65
411 ))
412 if action == "prune":
413
414 diff --git a/lib/_emerge/create_world_atom.py b/lib/_emerge/create_world_atom.py
415 index 947f8088a..c5e1f58be 100644
416 --- a/lib/_emerge/create_world_atom.py
417 +++ b/lib/_emerge/create_world_atom.py
418 @@ -67,7 +67,7 @@ def create_world_atom(pkg, args_set, root_config, before_install=False):
419 # unknown value, so just record an unslotted atom.
420 #
421 # 2) SLOT comes from an installed package and there is no
422 - # matching SLOT in the portage tree.
423 + # matching SLOT in the ebuild repository.
424 #
425 # Make sure that the slot atom is available in either the
426 # portdb or the vardb, since otherwise the user certainly
427
428 diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
429 index cb8afa903..44d3eff6c 100644
430 --- a/lib/_emerge/depgraph.py
431 +++ b/lib/_emerge/depgraph.py
432 @@ -4007,7 +4007,7 @@ class depgraph(object):
433 pkgdir = os.path.dirname(ebuild_path)
434 tree_root = os.path.dirname(os.path.dirname(pkgdir))
435 cp = pkgdir[len(tree_root)+1:]
436 - error_msg = ("\n\n!!! '%s' is not in a valid portage tree "
437 + error_msg = ("\n\n!!! '%s' is not in a valid ebuild repository "
438 "hierarchy or does not exist\n") % x
439 if not portage.isvalidatom(cp):
440 writemsg(error_msg, noiselevel=-1)
441
442 diff --git a/lib/_emerge/search.py b/lib/_emerge/search.py
443 index eb52b2ca3..be639dfa3 100644
444 --- a/lib/_emerge/search.py
445 +++ b/lib/_emerge/search.py
446 @@ -104,7 +104,7 @@ class search(object):
447 for db in self._dbs:
448 if db is not self._portdb:
449 # We don't want findname to return anything
450 - # unless it's an ebuild in a portage tree.
451 + # unless it's an ebuild in a repository.
452 # Otherwise, it's already built and we don't
453 # care about it.
454 continue
455
456 diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
457 index f9cc1a87c..db9c3854d 100644
458 --- a/lib/portage/__init__.py
459 +++ b/lib/portage/__init__.py
460 @@ -134,7 +134,7 @@ except ImportError as e:
461 sys.stderr.write("!!! Failed to complete portage imports. There are internal modules for\n")
462 sys.stderr.write("!!! portage and failure here indicates that you have a problem with your\n")
463 sys.stderr.write("!!! installation of portage. Please try a rescue portage located in the\n")
464 - sys.stderr.write("!!! portage tree under '/usr/portage/sys-apps/portage/files/' (default).\n")
465 + sys.stderr.write("!!! ebuild repository under '/usr/portage/sys-apps/portage/files/' (default).\n")
466 sys.stderr.write("!!! There is a README.RESCUE file that details the steps required to perform\n")
467 sys.stderr.write("!!! a recovery of portage.\n")
468 sys.stderr.write(" "+str(e)+"\n\n")
469
470 diff --git a/lib/portage/_global_updates.py b/lib/portage/_global_updates.py
471 index 81ee484ee..6a595da56 100644
472 --- a/lib/portage/_global_updates.py
473 +++ b/lib/portage/_global_updates.py
474 @@ -22,7 +22,7 @@ def _global_updates(trees, prev_mtimes, quiet=False, if_mtime_changed=True):
475 This simply returns if ROOT != "/" (when len(trees) != 1). If ROOT != "/"
476 then the user should instead use emaint --fix movebin and/or moveinst.
477
478 - @param trees: A dictionary containing portage trees.
479 + @param trees: A dictionary containing package databases.
480 @type trees: dict
481 @param prev_mtimes: A dictionary containing mtimes of files located in
482 $PORTDIR/profiles/updates/.
483
484 diff --git a/lib/portage/dbapi/porttree.py b/lib/portage/dbapi/porttree.py
485 index 677452273..aa8b50a57 100644
486 --- a/lib/portage/dbapi/porttree.py
487 +++ b/lib/portage/dbapi/porttree.py
488 @@ -239,7 +239,7 @@ class portdbapi(dbapi):
489 # This is used as sanity check for aux_get(). If there is no
490 # root eclass dir, we assume that PORTDIR is invalid or
491 # missing. This check allows aux_get() to detect a missing
492 - # portage tree and return early by raising a KeyError.
493 + # repository and return early by raising a KeyError.
494 self._have_root_eclass_dir = os.path.isdir(
495 os.path.join(self.settings.repositories.mainRepoLocation(), "eclass"))
496
497
498 diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
499 index 943ba4ff8..645e8e313 100644
500 --- a/lib/portage/dbapi/vartree.py
501 +++ b/lib/portage/dbapi/vartree.py
502 @@ -2189,7 +2189,7 @@ class dblink(object):
503 "removal phases to execute successfully. "
504 "The ebuild will be "
505 "sourced and the eclasses "
506 - "from the current portage tree will be used "
507 + "from the current ebuild repository will be used "
508 "when necessary. Removal of "
509 "the ebuild file will cause the "
510 "pkg_prerm() and pkg_postrm() removal "
511
512 diff --git a/lib/portage/emaint/modules/merges/merges.py b/lib/portage/emaint/modules/merges/merges.py
513 index 416a725ff..e166150f4 100644
514 --- a/lib/portage/emaint/modules/merges/merges.py
515 +++ b/lib/portage/emaint/modules/merges/merges.py
516 @@ -182,7 +182,7 @@ class MergesHandler(object):
517 % pkg_atom)
518 if not portdb.cpv_exists(pkg_name):
519 pkg_invalid_entries.add(
520 - "'%s' does not exist in the portage tree." % pkg_name)
521 + "'%s' does not exist in the ebuild repository." % pkg_name)
522 pkg_atoms.add(pkg_atom)
523
524
525
526 diff --git a/lib/portage/metadata.py b/lib/portage/metadata.py
527 index 1abec5a89..ffb7672d3 100644
528 --- a/lib/portage/metadata.py
529 +++ b/lib/portage/metadata.py
530 @@ -50,8 +50,10 @@ def action_metadata(settings, portdb, myopts, porttrees=None):
531 src_db = portdb._create_pregen_cache(path)
532
533 if src_db is not None:
534 - porttrees_data.append(TreeData(portdb.auxdb[path],
535 - portdb.repositories.get_repo_for_location(path).eclass_db, path, src_db))
536 + eclass_db = portdb.repositories.get_repo_for_location(path).eclass_db
537 + # Update eclass data which may be stale after sync.
538 + eclass_db.update_eclasses()
539 + porttrees_data.append(TreeData(portdb.auxdb[path], eclass_db, path, src_db))
540
541 porttrees = [tree_data.path for tree_data in porttrees_data]
542
543
544 diff --git a/lib/portage/news.py b/lib/portage/news.py
545 index d4f1429b2..8397646b0 100644
546 --- a/lib/portage/news.py
547 +++ b/lib/portage/news.py
548 @@ -393,7 +393,7 @@ def count_unread_news(portdb, vardb, repos=None, update=True):
549 By default, this will scan all repos and check for new items that have
550 appeared since the last scan.
551
552 - @param portdb: a portage tree database
553 + @param portdb: an ebuild database
554 @type portdb: pordbapi
555 @param vardb: an installed package database
556 @type vardb: vardbapi
557
558 diff --git a/lib/portage/package/ebuild/config.py b/lib/portage/package/ebuild/config.py
559 index 2eb766d0f..c00a51345 100644
560 --- a/lib/portage/package/ebuild/config.py
561 +++ b/lib/portage/package/ebuild/config.py
562 @@ -1130,7 +1130,7 @@ class config(object):
563 archlist = self.archlist()
564 if not archlist:
565 writemsg(_("--- 'profiles/arch.list' is empty or "
566 - "not available. Empty portage tree?\n"), noiselevel=1)
567 + "not available. Empty ebuild repository?\n"), noiselevel=1)
568 else:
569 for group in groups:
570 if group not in archlist and \
571
572 diff --git a/lib/portage/sync/modules/rsync/rsync.py b/lib/portage/sync/modules/rsync/rsync.py
573 index 56e38631e..e0748794a 100644
574 --- a/lib/portage/sync/modules/rsync/rsync.py
575 +++ b/lib/portage/sync/modules/rsync/rsync.py
576 @@ -352,7 +352,7 @@ class RsyncSync(NewBase):
577 if (retries==0):
578 if "--ask" in opts:
579 uq = UserQuery(opts)
580 - if uq.query("Do you want to sync your Portage tree " + \
581 + if uq.query("Do you want to sync your ebuild repository " + \
582 "with the mirror at\n" + blue(dosyncuri) + bold("?"),
583 enter_invalid) == "No":
584 print()
585
586 diff --git a/lib/portage/tests/util/futures/asyncio/test_subprocess_exec.py b/lib/portage/tests/util/futures/asyncio/test_subprocess_exec.py
587 index 5a812ba6a..61646cb92 100644
588 --- a/lib/portage/tests/util/futures/asyncio/test_subprocess_exec.py
589 +++ b/lib/portage/tests/util/futures/asyncio/test_subprocess_exec.py
590 @@ -3,61 +3,16 @@
591
592 import os
593 import subprocess
594 -
595 -try:
596 - from asyncio import create_subprocess_exec
597 -except ImportError:
598 - create_subprocess_exec = None
599 +import sys
600
601 from portage.process import find_binary
602 from portage.tests import TestCase
603 from portage.util._eventloop.global_event_loop import global_event_loop
604 from portage.util.futures import asyncio
605 -from portage.util.futures.executor.fork import ForkExecutor
606 +from portage.util.futures._asyncio import create_subprocess_exec
607 +from portage.util.futures._asyncio.streams import _reader as reader
608 +from portage.util.futures.compat_coroutine import coroutine, coroutine_return
609 from portage.util.futures.unix_events import DefaultEventLoopPolicy
610 -from _emerge.PipeReader import PipeReader
611 -
612 -
613 -def reader(input_file, loop=None):
614 - """
615 - Asynchronously read a binary input file.
616 -
617 - @param input_file: binary input file
618 - @type input_file: file
619 - @param loop: event loop
620 - @type loop: EventLoop
621 - @return: bytes
622 - @rtype: asyncio.Future (or compatible)
623 - """
624 - loop = asyncio._wrap_loop(loop)
625 - future = loop.create_future()
626 - _Reader(future, input_file, loop)
627 - return future
628 -
629 -
630 -class _Reader(object):
631 - def __init__(self, future, input_file, loop):
632 - self._future = future
633 - self._pipe_reader = PipeReader(
634 - input_files={'input_file':input_file}, scheduler=loop)
635 -
636 - self._future.add_done_callback(self._cancel_callback)
637 - self._pipe_reader.addExitListener(self._eof)
638 - self._pipe_reader.start()
639 -
640 - def _cancel_callback(self, future):
641 - if future.cancelled():
642 - self._cancel()
643 -
644 - def _eof(self, pipe_reader):
645 - self._pipe_reader = None
646 - self._future.set_result(pipe_reader.getvalue())
647 -
648 - def _cancel(self):
649 - if self._pipe_reader is not None and self._pipe_reader.poll() is None:
650 - self._pipe_reader.removeExitListener(self._eof)
651 - self._pipe_reader.cancel()
652 - self._pipe_reader = None
653
654
655 class SubprocessExecTestCase(TestCase):
656 @@ -76,99 +31,66 @@ class SubprocessExecTestCase(TestCase):
657 self.assertFalse(global_event_loop().is_closed())
658
659 def testEcho(self):
660 - if create_subprocess_exec is None:
661 - self.skipTest('create_subprocess_exec not implemented for python2')
662 -
663 args_tuple = (b'hello', b'world')
664 echo_binary = find_binary("echo")
665 self.assertNotEqual(echo_binary, None)
666 echo_binary = echo_binary.encode()
667
668 - # Use os.pipe(), since this loop does not implement the
669 - # ReadTransport necessary for subprocess.PIPE support.
670 - stdout_pr, stdout_pw = os.pipe()
671 - stdout_pr = os.fdopen(stdout_pr, 'rb', 0)
672 - stdout_pw = os.fdopen(stdout_pw, 'wb', 0)
673 - files = [stdout_pr, stdout_pw]
674 -
675 def test(loop):
676 - output = None
677 - try:
678 - with open(os.devnull, 'rb', 0) as devnull:
679 - proc = loop.run_until_complete(
680 - create_subprocess_exec(
681 - echo_binary, *args_tuple,
682 - stdin=devnull, stdout=stdout_pw, stderr=stdout_pw))
683 -
684 - # This belongs exclusively to the subprocess now.
685 - stdout_pw.close()
686 -
687 - output = asyncio.ensure_future(
688 - reader(stdout_pr, loop=loop), loop=loop)
689 -
690 - self.assertEqual(
691 - loop.run_until_complete(proc.wait()), os.EX_OK)
692 - self.assertEqual(
693 - tuple(loop.run_until_complete(output).split()), args_tuple)
694 - finally:
695 - if output is not None and not output.done():
696 - output.cancel()
697 - for f in files:
698 - f.close()
699 + @coroutine
700 + def test_coroutine(loop=None):
701
702 - self._run_test(test)
703 + proc = (yield create_subprocess_exec(echo_binary, *args_tuple,
704 + stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
705 + loop=loop))
706
707 - def testCat(self):
708 - if create_subprocess_exec is None:
709 - self.skipTest('create_subprocess_exec not implemented for python2')
710 + out, err = (yield proc.communicate())
711 + self.assertEqual(tuple(out.split()), args_tuple)
712 + self.assertEqual(proc.returncode, os.EX_OK)
713
714 - stdin_data = b'hello world'
715 - cat_binary = find_binary("cat")
716 - self.assertNotEqual(cat_binary, None)
717 - cat_binary = cat_binary.encode()
718 + proc = (yield create_subprocess_exec(
719 + 'bash', '-c', 'echo foo; echo bar 1>&2;',
720 + stdout=subprocess.PIPE, stderr=subprocess.PIPE,
721 + loop=loop))
722
723 - # Use os.pipe(), since this loop does not implement the
724 - # ReadTransport necessary for subprocess.PIPE support.
725 - stdout_pr, stdout_pw = os.pipe()
726 - stdout_pr = os.fdopen(stdout_pr, 'rb', 0)
727 - stdout_pw = os.fdopen(stdout_pw, 'wb', 0)
728 + out, err = (yield proc.communicate())
729 + self.assertEqual(out, b'foo\n')
730 + self.assertEqual(err, b'bar\n')
731 + self.assertEqual(proc.returncode, os.EX_OK)
732
733 - stdin_pr, stdin_pw = os.pipe()
734 - stdin_pr = os.fdopen(stdin_pr, 'rb', 0)
735 - stdin_pw = os.fdopen(stdin_pw, 'wb', 0)
736 + proc = (yield create_subprocess_exec(
737 + 'bash', '-c', 'echo foo; echo bar 1>&2;',
738 + stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
739 + loop=loop))
740
741 - files = [stdout_pr, stdout_pw, stdin_pr, stdin_pw]
742 + out, err = (yield proc.communicate())
743 + self.assertEqual(out, b'foo\nbar\n')
744 + self.assertEqual(err, None)
745 + self.assertEqual(proc.returncode, os.EX_OK)
746
747 - def test(loop):
748 - output = None
749 - try:
750 - proc = loop.run_until_complete(
751 - create_subprocess_exec(
752 - cat_binary,
753 - stdin=stdin_pr, stdout=stdout_pw, stderr=stdout_pw))
754 + coroutine_return('success')
755
756 - # These belong exclusively to the subprocess now.
757 - stdout_pw.close()
758 - stdin_pr.close()
759 + self.assertEqual('success',
760 + loop.run_until_complete(test_coroutine(loop=loop)))
761
762 - output = asyncio.ensure_future(
763 - reader(stdout_pr, loop=loop), loop=loop)
764 + self._run_test(test)
765
766 - with ForkExecutor(loop=loop) as executor:
767 - writer = asyncio.ensure_future(loop.run_in_executor(
768 - executor, stdin_pw.write, stdin_data), loop=loop)
769 + def testCat(self):
770 + stdin_data = b'hello world'
771 + cat_binary = find_binary("cat")
772 + self.assertNotEqual(cat_binary, None)
773 + cat_binary = cat_binary.encode()
774
775 - # This belongs exclusively to the writer now.
776 - stdin_pw.close()
777 - loop.run_until_complete(writer)
778 + def test(loop):
779 + proc = loop.run_until_complete(
780 + create_subprocess_exec(cat_binary,
781 + stdin=subprocess.PIPE, stdout=subprocess.PIPE,
782 + loop=loop))
783 +
784 + out, err = loop.run_until_complete(proc.communicate(input=stdin_data))
785
786 - self.assertEqual(loop.run_until_complete(proc.wait()), os.EX_OK)
787 - self.assertEqual(loop.run_until_complete(output), stdin_data)
788 - finally:
789 - if output is not None and not output.done():
790 - output.cancel()
791 - for f in files:
792 - f.close()
793 + self.assertEqual(loop.run_until_complete(proc.wait()), os.EX_OK)
794 + self.assertEqual(out, stdin_data)
795
796 self._run_test(test)
797
798 @@ -178,8 +100,8 @@ class SubprocessExecTestCase(TestCase):
799 requires an AbstractEventLoop.connect_read_pipe implementation
800 (and a ReadTransport implementation for it to return).
801 """
802 - if create_subprocess_exec is None:
803 - self.skipTest('create_subprocess_exec not implemented for python2')
804 + if sys.version_info.major < 3:
805 + self.skipTest('ReadTransport not implemented for python2')
806
807 args_tuple = (b'hello', b'world')
808 echo_binary = find_binary("echo")
809 @@ -192,7 +114,8 @@ class SubprocessExecTestCase(TestCase):
810 create_subprocess_exec(
811 echo_binary, *args_tuple,
812 stdin=devnull,
813 - stdout=subprocess.PIPE, stderr=subprocess.STDOUT))
814 + stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
815 + loop=loop))
816
817 self.assertEqual(
818 tuple(loop.run_until_complete(proc.stdout.read()).split()),
819 @@ -207,8 +130,8 @@ class SubprocessExecTestCase(TestCase):
820 requires an AbstractEventLoop.connect_write_pipe implementation
821 (and a WriteTransport implementation for it to return).
822 """
823 - if create_subprocess_exec is None:
824 - self.skipTest('create_subprocess_exec not implemented for python2')
825 + if sys.version_info.major < 3:
826 + self.skipTest('WriteTransport not implemented for python2')
827
828 stdin_data = b'hello world'
829 cat_binary = find_binary("cat")
830 @@ -220,7 +143,8 @@ class SubprocessExecTestCase(TestCase):
831 create_subprocess_exec(
832 cat_binary,
833 stdin=subprocess.PIPE,
834 - stdout=subprocess.PIPE, stderr=subprocess.STDOUT))
835 + stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
836 + loop=loop))
837
838 # This buffers data when necessary to avoid blocking.
839 proc.stdin.write(stdin_data)
840
841 diff --git a/lib/portage/util/compression_probe.py b/lib/portage/util/compression_probe.py
842 index 29d0eedff..90880b1cd 100644
843 --- a/lib/portage/util/compression_probe.py
844 +++ b/lib/portage/util/compression_probe.py
845 @@ -45,7 +45,7 @@ _compressors = {
846 },
847 "zstd": {
848 "compress": "zstd ${BINPKG_COMPRESS_FLAGS}",
849 - "decompress": "zstd -d",
850 + "decompress": "zstd -d --long=31",
851 "package": "app-arch/zstd",
852 },
853 }
854
855 diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
856 index acfd59396..faab98e47 100644
857 --- a/lib/portage/util/futures/_asyncio/__init__.py
858 +++ b/lib/portage/util/futures/_asyncio/__init__.py
859 @@ -20,6 +20,9 @@ __all__ = (
860 'wait',
861 )
862
863 +import subprocess
864 +import sys
865 +
866 try:
867 import asyncio as _real_asyncio
868 except ImportError:
869 @@ -45,6 +48,7 @@ from portage.util.futures.futures import (
870 InvalidStateError,
871 TimeoutError,
872 )
873 +from portage.util.futures._asyncio.process import _Process
874 from portage.util.futures._asyncio.tasks import (
875 ALL_COMPLETED,
876 FIRST_COMPLETED,
877 @@ -105,6 +109,49 @@ def set_child_watcher(watcher):
878 return get_event_loop_policy().set_child_watcher(watcher)
879
880
881 +# Python 3.4 and later implement PEP 446, which makes newly
882 +# created file descriptors non-inheritable by default.
883 +_close_fds_default = sys.version_info < (3, 4)
884 +
885 +
886 +def create_subprocess_exec(*args, **kwargs):
887 + """
888 + Create a subprocess.
889 +
890 + @param args: program and arguments
891 + @type args: str
892 + @param stdin: stdin file descriptor
893 + @type stdin: file or int
894 + @param stdout: stdout file descriptor
895 + @type stdout: file or int
896 + @param stderr: stderr file descriptor
897 + @type stderr: file or int
898 + @param close_fds: close file descriptors
899 + @type close_fds: bool
900 + @param loop: asyncio.AbstractEventLoop (or compatible)
901 + @type loop: event loop
902 + @type kwargs: varies
903 + @param kwargs: subprocess.Popen parameters
904 + @rtype: asyncio.Future (or compatible)
905 + @return: subset of asyncio.subprocess.Process interface
906 + """
907 + loop = _wrap_loop(kwargs.pop('loop', None))
908 + kwargs.setdefault('close_fds', _close_fds_default)
909 + if _asyncio_enabled and isinstance(loop, _AsyncioEventLoop):
910 + # Use the real asyncio loop and create_subprocess_exec.
911 + return _real_asyncio.create_subprocess_exec(*args, loop=loop._loop, **kwargs)
912 +
913 + result = loop.create_future()
914 +
915 + result.set_result(_Process(subprocess.Popen(
916 + args,
917 + stdin=kwargs.pop('stdin', None),
918 + stdout=kwargs.pop('stdout', None),
919 + stderr=kwargs.pop('stderr', None), **kwargs), loop))
920 +
921 + return result
922 +
923 +
924 class Task(Future):
925 """
926 Schedule the execution of a coroutine: wrap it in a future. A task
927 @@ -127,6 +174,12 @@ def ensure_future(coro_or_future, loop=None):
928 @rtype: asyncio.Future (or compatible)
929 @return: an instance of Future
930 """
931 + loop = _wrap_loop(loop)
932 + if _asyncio_enabled and isinstance(loop, _AsyncioEventLoop):
933 + # Use the real asyncio loop and ensure_future.
934 + return _real_asyncio.ensure_future(
935 + coro_or_future, loop=loop._loop)
936 +
937 if isinstance(coro_or_future, Future):
938 return coro_or_future
939 raise NotImplementedError
940
941 diff --git a/lib/portage/util/futures/_asyncio/process.py b/lib/portage/util/futures/_asyncio/process.py
942 new file mode 100644
943 index 000000000..020164c9b
944 --- /dev/null
945 +++ b/lib/portage/util/futures/_asyncio/process.py
946 @@ -0,0 +1,107 @@
947 +# Copyright 2018 Gentoo Foundation
948 +# Distributed under the terms of the GNU General Public License v2
949 +
950 +import portage
951 +portage.proxy.lazyimport.lazyimport(globals(),
952 + 'portage.util.futures:asyncio',
953 +)
954 +from portage.util.futures._asyncio.streams import _reader, _writer
955 +from portage.util.futures.compat_coroutine import coroutine, coroutine_return
956 +
957 +
958 +class _Process(object):
959 + """
960 + Emulate a subset of the asyncio.subprocess.Process interface,
961 + for python2.
962 + """
963 + def __init__(self, proc, loop):
964 + """
965 + @param proc: process instance
966 + @type proc: subprocess.Popen
967 + @param loop: asyncio.AbstractEventLoop (or compatible)
968 + @type loop: event loop
969 + """
970 + self._proc = proc
971 + self._loop = loop
972 + self.terminate = proc.terminate
973 + self.kill = proc.kill
974 + self.send_signal = proc.send_signal
975 + self.pid = proc.pid
976 + self._waiters = []
977 + loop._asyncio_child_watcher.\
978 + add_child_handler(self.pid, self._proc_exit)
979 +
980 + @property
981 + def returncode(self):
982 + return self._proc.returncode
983 +
984 + @coroutine
985 + def communicate(self, input=None):
986 + """
987 + Read data from stdout and stderr, until end-of-file is reached.
988 + Wait for process to terminate.
989 +
990 + @param input: stdin content to write
991 + @type input: bytes
992 + @return: tuple (stdout_data, stderr_data)
993 + @rtype: asyncio.Future (or compatible)
994 + """
995 + futures = []
996 + for input_file in (self._proc.stdout, self._proc.stderr):
997 + if input_file is None:
998 + future = self._loop.create_future()
999 + future.set_result(None)
1000 + else:
1001 + future = _reader(input_file, loop=self._loop)
1002 + futures.append(future)
1003 +
1004 + writer = None
1005 + if input is not None:
1006 + if self._proc.stdin is None:
1007 + raise TypeError('communicate: expected file or int, got {}'.format(type(self._proc.stdin)))
1008 + writer = asyncio.ensure_future(_writer(self._proc.stdin, input), loop=self._loop)
1009 +
1010 + try:
1011 + yield asyncio.wait(futures + [self.wait()], loop=self._loop)
1012 + finally:
1013 + if writer is not None:
1014 + if writer.done():
1015 + # Consume expected exceptions.
1016 + try:
1017 + writer.result()
1018 + except EnvironmentError:
1019 + # This is normal if the other end of the pipe was closed.
1020 + pass
1021 + else:
1022 + writer.cancel()
1023 +
1024 + coroutine_return(tuple(future.result() for future in futures))
1025 +
1026 + def wait(self):
1027 + """
1028 + Wait for child process to terminate. Set and return returncode attribute.
1029 +
1030 + @return: returncode
1031 + @rtype: asyncio.Future (or compatible)
1032 + """
1033 + waiter = self._loop.create_future()
1034 + if self.returncode is None:
1035 + self._waiters.append(waiter)
1036 + waiter.add_done_callback(self._waiter_cancel)
1037 + else:
1038 + waiter.set_result(self.returncode)
1039 + return waiter
1040 +
1041 + def _waiter_cancel(self, waiter):
1042 + if waiter.cancelled():
1043 + try:
1044 + self._waiters.remove(waiter)
1045 + except ValueError:
1046 + pass
1047 +
1048 + def _proc_exit(self, pid, returncode):
1049 + self._proc.returncode = returncode
1050 + waiters = self._waiters
1051 + self._waiters = []
1052 + for waiter in waiters:
1053 + waiter.set_result(returncode)
1054
1055 diff --git a/lib/portage/util/futures/_asyncio/streams.py b/lib/portage/util/futures/_asyncio/streams.py
1056 new file mode 100644
1057 index 000000000..650a16491
1058 --- /dev/null
1059 +++ b/lib/portage/util/futures/_asyncio/streams.py
1060 @@ -0,0 +1,96 @@
1061 +# Copyright 2018 Gentoo Foundation
1062 +# Distributed under the terms of the GNU General Public License v2
1063 +
1064 +import errno
1065 +import os
1066 +
1067 +import portage
1068 +portage.proxy.lazyimport.lazyimport(globals(),
1069 + '_emerge.PipeReader:PipeReader',
1070 + 'portage.util.futures:asyncio',
1071 + 'portage.util.futures.unix_events:_set_nonblocking',
1072 +)
1073 +from portage.util.futures.compat_coroutine import coroutine
1074 +
1075 +
1076 +def _reader(input_file, loop=None):
1077 + """
1078 + Asynchronously read a binary input file, and close it when
1079 + it reaches EOF.
1080 +
1081 + @param input_file: binary input file descriptor
1082 + @type input_file: file or int
1083 + @param loop: asyncio.AbstractEventLoop (or compatible)
1084 + @type loop: event loop
1085 + @return: bytes
1086 + @rtype: asyncio.Future (or compatible)
1087 + """
1088 + loop = asyncio._wrap_loop(loop)
1089 + future = loop.create_future()
1090 + _Reader(future, input_file, loop)
1091 + return future
1092 +
1093 +
1094 +class _Reader(object):
1095 + def __init__(self, future, input_file, loop):
1096 + self._future = future
1097 + self._pipe_reader = PipeReader(
1098 + input_files={'input_file':input_file}, scheduler=loop)
1099 +
1100 + self._future.add_done_callback(self._cancel_callback)
1101 + self._pipe_reader.addExitListener(self._eof)
1102 + self._pipe_reader.start()
1103 +
1104 + def _cancel_callback(self, future):
1105 + if future.cancelled():
1106 + self._cancel()
1107 +
1108 + def _eof(self, pipe_reader):
1109 + self._pipe_reader = None
1110 + self._future.set_result(pipe_reader.getvalue())
1111 +
1112 + def _cancel(self):
1113 + if self._pipe_reader is not None and self._pipe_reader.poll() is None:
1114 + self._pipe_reader.removeExitListener(self._eof)
1115 + self._pipe_reader.cancel()
1116 + self._pipe_reader = None
1117 +
1118 +
1119 +@coroutine
1120 +def _writer(output_file, content, loop=None):
1121 + """
1122 + Asynchronously write bytes to output file, and close it when
1123 + done. If an EnvironmentError other than EAGAIN is encountered,
1124 + which typically indicates that the other end of the pipe has
1125 + close, the error is raised. This function is a coroutine.
1126 +
1127 + @param output_file: output file descriptor
1128 + @type output_file: file or int
1129 + @param content: content to write
1130 + @type content: bytes
1131 + @param loop: asyncio.AbstractEventLoop (or compatible)
1132 + @type loop: event loop
1133 + """
1134 + fd = output_file if isinstance(output_file, int) else output_file.fileno()
1135 + _set_nonblocking(fd)
1136 + loop = asyncio._wrap_loop(loop)
1137 + try:
1138 + while content:
1139 + waiter = loop.create_future()
1140 + loop.add_writer(fd, lambda: waiter.set_result(None))
1141 + try:
1142 + yield waiter
1143 + while content:
1144 + try:
1145 + content = content[os.write(fd, content):]
1146 + except EnvironmentError as e:
1147 + if e.errno == errno.EAGAIN:
1148 + break
1149 + else:
1150 + raise
1151 + finally:
1152 + loop.remove_writer(fd)
1153 + except GeneratorExit:
1154 + raise
1155 + finally:
1156 + os.close(output_file) if isinstance(output_file, int) else output_file.close()
1157
1158 diff --git a/lib/portage/util/futures/compat_coroutine.py b/lib/portage/util/futures/compat_coroutine.py
1159 index 17400b74d..59fdc31b6 100644
1160 --- a/lib/portage/util/futures/compat_coroutine.py
1161 +++ b/lib/portage/util/futures/compat_coroutine.py
1162 @@ -1,9 +1,13 @@
1163 # Copyright 2018 Gentoo Foundation
1164 # Distributed under the terms of the GNU General Public License v2
1165
1166 -from portage.util.futures import asyncio
1167 import functools
1168
1169 +import portage
1170 +portage.proxy.lazyimport.lazyimport(globals(),
1171 + 'portage.util.futures:asyncio',
1172 +)
1173 +
1174
1175 def coroutine(generator_func):
1176 """
1177
1178 diff --git a/man/emerge.1 b/man/emerge.1
1179 index 7003cf29e..af553fcb5 100644
1180 --- a/man/emerge.1
1181 +++ b/man/emerge.1
1182 @@ -22,7 +22,7 @@ emerge \- Command\-line interface to the Portage system
1183 \fBemerge\fR is the definitive command\-line interface to the Portage
1184 system. It is primarily used for installing packages, and \fBemerge\fR
1185 can automatically handle any dependencies that the desired package has.
1186 -\fBemerge\fR can also update the \fBportage tree\fR, making new and
1187 +\fBemerge\fR can also update the \fBebuild repository\fR, making new and
1188 updated packages available. \fBemerge\fR gracefully handles updating
1189 installed packages to newer releases as well. It handles both source
1190 and binary packages, and it can be used to create binary packages for
1191 @@ -198,7 +198,7 @@ the emerge output of the next \-\-depclean run carefully! Use
1192 .TP
1193 .BR \-\-regen
1194 Causes portage to check and update the dependency cache of all ebuilds in the
1195 -portage tree. The cache is used to speed up searches and the building of
1196 +repository. The cache is used to speed up searches and the building of
1197 dependency trees. This command is not recommended for rsync users as rsync
1198 updates the cache using server\-side caches. If you do not know the
1199 differences between a 'rsync user' and some other user, then you are a 'rsync
1200 @@ -227,7 +227,7 @@ explicitly discarded by running `emaint \-\-fix cleanresume` (see
1201 \fBemaint\fR(1)).
1202 .TP
1203 .BR \-\-search ", " \-s
1204 -Searches for matches of the supplied string in the portage tree.
1205 +Searches for matches of the supplied string in the ebuild repository.
1206 By default emerge uses a case-insensitive simple search, but you can
1207 enable a regular expression search by prefixing the search string with %.
1208 For example, \fBemerge \-\-search "%^kde"\fR searches for any package whose
1209 @@ -264,7 +264,7 @@ more information about sync operations.
1210
1211 \fBNOTE:\fR
1212 The \fBemerge\-webrsync\fR program will download the entire
1213 -portage tree as a tarball, which is much faster than emerge
1214 +ebuild repository as a tarball, which is much faster than emerge
1215 \-\-sync for first time syncs.
1216
1217 .TP
1218 @@ -1230,7 +1230,7 @@ You should almost always precede any package install or update attempt with a
1219 \fB\-\-pretend\fR install or update. This lets you see how much will be
1220 done, and shows you any blocking packages that you will have to rectify.
1221 This goes doubly so for the \fBsystem\fR and \fBworld\fR sets, which can
1222 -update a large number of packages if the portage tree has been particularly
1223 +update a large number of packages if the ebuild repository has been particularly
1224 active.
1225 .LP
1226 You also want to typically use \fB\-\-update\fR, which ignores packages that
1227
1228 diff --git a/man/make.conf.5 b/man/make.conf.5
1229 index 2c7b0fc17..127a62e44 100644
1230 --- a/man/make.conf.5
1231 +++ b/man/make.conf.5
1232 @@ -1107,7 +1107,7 @@ Defaults to /usr/portage/rpm.
1233 .TP
1234 \fBSYNC\fR = \fI[RSYNC]\fR
1235 Insert your preferred rsync mirror here. This rsync server
1236 -is used to sync the local portage tree when `emerge \-\-sync` is run.
1237 +is used to sync the local ebuild repository when `emerge \-\-sync` is run.
1238
1239 Note that the \fBSYNC\fR variable is now deprecated, and instead the
1240 sync\-type and sync\-uri attributes in repos.conf should be used. See
1241
1242 diff --git a/man/portage.5 b/man/portage.5
1243 index cd9d5036d..c3c610a6c 100644
1244 --- a/man/portage.5
1245 +++ b/man/portage.5
1246 @@ -214,7 +214,7 @@ More reading:
1247 .TP
1248 \fB/etc/portage/make.profile/\fR or \fB/etc/make.profile/\fR
1249 This is usually just a symlink to the correct profile in
1250 -\fB/usr/portage/profiles/\fR. Since it is part of the portage tree, it
1251 +\fB/usr/portage/profiles/\fR. Since it is part of the ebuild repository, it
1252 may easily be updated/regenerated by running `emerge \-\-sync`. It defines
1253 what a profile is (usually arch specific stuff). If you need a custom
1254 profile, then you should make your own \fBmake.profile\fR
1255 @@ -1666,13 +1666,13 @@ again.
1256 All installed package information is recorded here. If portage thinks you have
1257 a package installed, it is usually because it is listed here.
1258
1259 -The format follows somewhat closely that of the portage tree. There is a
1260 +The format follows somewhat closely that of the ebuild repository. There is a
1261 directory for each category and a package-version subdirectory for each package
1262 you have installed.
1263
1264 Inside each package directory are misc files that describe the installed
1265 contents of the package as well as build time information (so that the package
1266 -can be unmerged without needing the portage tree).
1267 +can be unmerged without needing the ebuild repository).
1268
1269 The exact file contents and format are not described here again so that things
1270 can be changed quickly. Generally though there is one file per environment
1271
1272 diff --git a/repoman/README b/repoman/README
1273 index 5e78842c9..f0976e841 100644
1274 --- a/repoman/README
1275 +++ b/repoman/README
1276 @@ -3,7 +3,7 @@ About Portage
1277
1278 Portage is a package management system based on ports collections. The
1279 Package Manager Specification Project (PMS) standardises and documents
1280 -the behaviour of Portage so that the Portage tree can be used by other
1281 +the behaviour of Portage so that ebuild repositories can be used by other
1282 package managers.
1283
1284
1285
1286 diff --git a/repoman/lib/repoman/__init__.py b/repoman/lib/repoman/__init__.py
1287 index 89779b95c..301b34309 100644
1288 --- a/repoman/lib/repoman/__init__.py
1289 +++ b/repoman/lib/repoman/__init__.py
1290 @@ -14,7 +14,7 @@ except ImportError as e:
1291 sys.stderr.write("!!! Failed to complete portage imports. There are internal modules for\n")
1292 sys.stderr.write("!!! portage and failure here indicates that you have a problem with your\n")
1293 sys.stderr.write("!!! installation of portage. Please try a rescue portage located in the\n")
1294 - sys.stderr.write("!!! portage tree under '/usr/portage/sys-apps/portage/files/' (default).\n")
1295 + sys.stderr.write("!!! ebuild repository under '/usr/portage/sys-apps/portage/files/' (default).\n")
1296 sys.stderr.write("!!! There is a README.RESCUE file that details the steps required to perform\n")
1297 sys.stderr.write("!!! a recovery of portage.\n")
1298 sys.stderr.write(" "+str(e)+"\n\n")
1299
1300 diff --git a/repoman/lib/repoman/checks/herds/__init__.py b/repoman/lib/repoman/checks/herds/__init__.py
1301 deleted file mode 100644
1302 index e69de29bb..000000000
1303
1304 diff --git a/repoman/lib/repoman/checks/herds/herdbase.py b/repoman/lib/repoman/checks/herds/herdbase.py
1305 deleted file mode 100644
1306 index ebe6a19b4..000000000
1307 --- a/repoman/lib/repoman/checks/herds/herdbase.py
1308 +++ /dev/null
1309 @@ -1,135 +0,0 @@
1310 -# -*- coding: utf-8 -*-
1311 -# repoman: Herd database analysis
1312 -# Copyright 2010-2013 Gentoo Foundation
1313 -# Distributed under the terms of the GNU General Public License v2 or later
1314 -
1315 -from __future__ import print_function, unicode_literals
1316 -
1317 -import errno
1318 -import xml.etree.ElementTree
1319 -try:
1320 - from xml.parsers.expat import ExpatError
1321 -except (SystemExit, KeyboardInterrupt):
1322 - raise
1323 -except (ImportError, SystemError, RuntimeError, Exception):
1324 - # broken or missing xml support
1325 - # https://bugs.python.org/issue14988
1326 - # This means that python is built without xml support.
1327 - # We tolerate global scope import failures for optional
1328 - # modules, so that ImportModulesTestCase can succeed (or
1329 - # possibly alert us about unexpected import failures).
1330 - pass
1331 -
1332 -from portage import _encodings, _unicode_encode
1333 -from portage.exception import FileNotFound, ParseError, PermissionDenied
1334 -from portage import os
1335 -
1336 -from repoman.errors import err
1337 -
1338 -__all__ = [
1339 - "make_herd_base", "get_herd_base"
1340 -]
1341 -
1342 -
1343 -def _make_email(nick_name):
1344 - if not nick_name.endswith('@gentoo.org'):
1345 - nick_name = nick_name + '@gentoo.org'
1346 - return nick_name
1347 -
1348 -
1349 -class HerdBase(object):
1350 - def __init__(self, herd_to_emails, all_emails):
1351 - self.herd_to_emails = herd_to_emails
1352 - self.all_emails = all_emails
1353 -
1354 - def known_herd(self, herd_name):
1355 - return herd_name in self.herd_to_emails
1356 -
1357 - def known_maintainer(self, nick_name):
1358 - return _make_email(nick_name) in self.all_emails
1359 -
1360 - def maintainer_in_herd(self, nick_name, herd_name):
1361 - return _make_email(nick_name) in self.herd_to_emails[herd_name]
1362 -
1363 -
1364 -class _HerdsTreeBuilder(xml.etree.ElementTree.TreeBuilder):
1365 - """
1366 - Implements doctype() as required to avoid deprecation warnings with
1367 - >=python-2.7.
1368 - """
1369 - def doctype(self, name, pubid, system):
1370 - pass
1371 -
1372 -
1373 -def make_herd_base(filename):
1374 - herd_to_emails = dict()
1375 - all_emails = set()
1376 -
1377 - try:
1378 - xml_tree = xml.etree.ElementTree.parse(
1379 - _unicode_encode(
1380 - filename, encoding=_encodings['fs'], errors='strict'),
1381 - parser=xml.etree.ElementTree.XMLParser(
1382 - target=_HerdsTreeBuilder()))
1383 - except ExpatError as e:
1384 - raise ParseError("metadata.xml: %s" % (e,))
1385 - except EnvironmentError as e:
1386 - func_call = "open('%s')" % filename
1387 - if e.errno == errno.EACCES:
1388 - raise PermissionDenied(func_call)
1389 - elif e.errno == errno.ENOENT:
1390 - raise FileNotFound(filename)
1391 - raise
1392 -
1393 - herds = xml_tree.findall('herd')
1394 - for h in herds:
1395 - _herd_name = h.find('name')
1396 - if _herd_name is None:
1397 - continue
1398 - herd_name = _herd_name.text.strip()
1399 - del _herd_name
1400 -
1401 - maintainers = h.findall('maintainer')
1402 - herd_emails = set()
1403 - for m in maintainers:
1404 - _m_email = m.find('email')
1405 - if _m_email is None:
1406 - continue
1407 - m_email = _m_email.text.strip()
1408 -
1409 - herd_emails.add(m_email)
1410 - all_emails.add(m_email)
1411 -
1412 - herd_to_emails[herd_name] = herd_emails
1413 -
1414 - return HerdBase(herd_to_emails, all_emails)
1415 -
1416 -
1417 -def get_herd_base(repoman_settings):
1418 - try:
1419 - herd_base = make_herd_base(
1420 - os.path.join(repoman_settings["PORTDIR"], "metadata/herds.xml"))
1421 - except (EnvironmentError, ParseError, PermissionDenied) as e:
1422 - err(str(e))
1423 - except FileNotFound:
1424 - # TODO: Download as we do for metadata.dtd, but add a way to
1425 - # disable for non-gentoo repoman users who may not have herds.
1426 - herd_base = None
1427 - return herd_base
1428 -
1429 -
1430 -if __name__ == '__main__':
1431 - h = make_herd_base('/usr/portage/metadata/herds.xml')
1432 -
1433 - assert(h.known_herd('sound'))
1434 - assert(not h.known_herd('media-sound'))
1435 -
1436 - assert(h.known_maintainer('sping'))
1437 - assert(h.known_maintainer('sping@g.o'))
1438 - assert(not h.known_maintainer('portage'))
1439 -
1440 - assert(h.maintainer_in_herd('zmedico@g.o', 'tools-portage'))
1441 - assert(not h.maintainer_in_herd('pva@g.o', 'tools-portage'))
1442 -
1443 - import pprint
1444 - pprint.pprint(h.herd_to_emails)
1445
1446 diff --git a/repoman/lib/repoman/checks/herds/metadata.py b/repoman/lib/repoman/checks/herds/metadata.py
1447 deleted file mode 100644
1448 index b4a433ed7..000000000
1449 --- a/repoman/lib/repoman/checks/herds/metadata.py
1450 +++ /dev/null
1451 @@ -1,26 +0,0 @@
1452 -# -*- coding:utf-8 -*-
1453 -
1454 -
1455 -class UnknownHerdsError(ValueError):
1456 - def __init__(self, herd_names):
1457 - _plural = len(herd_names) != 1
1458 - super(UnknownHerdsError, self).__init__(
1459 - 'Unknown %s %s' % (
1460 - _plural and 'herds' or 'herd',
1461 - ','.join('"%s"' % e for e in herd_names)))
1462 -
1463 -
1464 -def check_metadata_herds(xml_tree, herd_base):
1465 - herd_nodes = xml_tree.findall('herd')
1466 - unknown_herds = [
1467 - name for name in (
1468 - e.text.strip() for e in herd_nodes if e.text is not None)
1469 - if not herd_base.known_herd(name)]
1470 -
1471 - if unknown_herds:
1472 - raise UnknownHerdsError(unknown_herds)
1473 -
1474 -
1475 -def check_metadata(xml_tree, herd_base):
1476 - if herd_base is not None:
1477 - check_metadata_herds(xml_tree, herd_base)
1478
1479 diff --git a/repoman/lib/repoman/utilities.py b/repoman/lib/repoman/utilities.py
1480 index 1272f3fb6..790d5e516 100644
1481 --- a/repoman/lib/repoman/utilities.py
1482 +++ b/repoman/lib/repoman/utilities.py
1483 @@ -256,7 +256,7 @@ def FindPortdir(settings):
1484 pwd = _unicode_decode(os.environ.get('PWD', ''), encoding=_encodings['fs'])
1485 if pwd and pwd != location and os.path.realpath(pwd) == location:
1486 # getcwd() returns the canonical path but that makes it hard for repoman to
1487 - # orient itself if the user has symlinks in their portage tree structure.
1488 + # orient itself if the user has symlinks in their repository structure.
1489 # We use os.environ["PWD"], if available, to get the non-canonical path of
1490 # the current working directory (from the shell).
1491 location = pwd
1492
1493 diff --git a/repoman/man/repoman.1 b/repoman/man/repoman.1
1494 index dea17c3b4..766146f57 100644
1495 --- a/repoman/man/repoman.1
1496 +++ b/repoman/man/repoman.1
1497 @@ -1,7 +1,7 @@
1498 .TH "REPOMAN" "1" "Mar 2018" "Repoman VERSION" "Repoman"
1499 .SH NAME
1500 repoman \- Gentoo's program to enforce a minimal level of quality assurance in
1501 -packages added to the portage tree
1502 +packages added to the ebuild repository
1503 .SH SYNOPSIS
1504 \fBrepoman\fR [\fIoption\fR] [\fImode\fR]
1505 .SH DESCRIPTION
1506
1507 diff --git a/setup.py b/setup.py
1508 index 52cda62d0..7cb4e2eb7 100755
1509 --- a/setup.py
1510 +++ b/setup.py
1511 @@ -662,7 +662,7 @@ class build_ext(_build_ext):
1512
1513 setup(
1514 name = 'portage',
1515 - version = '2.3.44',
1516 + version = '2.3.45',
1517 url = 'https://wiki.gentoo.org/wiki/Project:Portage',
1518 author = 'Gentoo Portage Development Team',
1519 author_email = 'dev-portage@g.o',