Gentoo Archives: gentoo-portage-dev

From: Zac Medico <zmedico@g.o>
To: gentoo-portage-dev@l.g.o
Cc: Zac Medico <zmedico@g.o>
Subject: [gentoo-portage-dev] [PATCH 0/4] Add sync-rcu support for rsync (bug 662070)
Date: Mon, 06 Aug 2018 07:43:50
Message-Id: 20180806074033.30318-1-zmedico@gentoo.org
1 Add a boolean sync-rcu repos.conf setting that behaves as follows:
2
3 sync-rcu = yes|no
4
5 Enable read-copy-update (RCU) behavior for sync operations. The
6 current latest immutable version of a repository will be referenced
7 by a symlink found where the repository would normally be located
8 (see the location setting). Repository consumers should resolve
9 the cannonical path of this symlink before attempt to access
10 the repository, and all operations should be read-only, since
11 the repository is considered immutable. Updates occur by atomic
12 replacement of the symlink, which causes new consumers to use the
13 new immutable version, while any earlier consumers continue to
14 use the cannonical path that was resolved earlier. This option
15 requires sync-allow-hardlinks and sync-rcu-store-dir options to
16 be enabled, and currently also requires that sync-type is set
17 to rsync. This option is disabled by default, since the symlink
18 usage would require special handling for scenarios involving bind
19 mounts and chroots.
20
21 sync-rcu-store-dir
22
23 Directory path reserved for sync-rcu storage. This directory must
24 have a unique value for each repository (do not set it in the
25 DEFAULT section). This directory must not contain any other files
26 or directories aside from those that are created automatically
27 when sync-rcu is enabled.
28
29 sync-rcu-spare-snapshots = 1
30
31 Number of spare snapshots for sync-rcu to retain with expired
32 ttl. This protects the previous latest snapshot from being removed
33 immediately after a new version becomes available, since it might
34 still be used by running processes.
35
36 sync-rcu-ttl-days = 7
37
38 Number of days for sync-rcu to retain previous immutable snapshots
39 of a repository. After the ttl of a particular snapshot has
40 expired, it will be remove automatically (the latest snapshot
41 is exempt, and sync-rcu-spare-snapshots configures the number of
42 previous snapshots that are exempt). If the ttl is set too low,
43 then a snapshot could expire while it is in use by a running
44 process.
45
46 Zac Medico (4):
47 Implement asyncio.iscoroutinefunction for compat_coroutine
48 Add _sync_decorator module
49 rsync: split out repo storage framework
50 Add sync-rcu support for rsync (bug 662070)
51
52 lib/portage/repository/config.py | 36 ++-
53 lib/portage/repository/storage/__init__.py | 0
54 .../repository/storage/hardlink_quarantine.py | 95 ++++++++
55 lib/portage/repository/storage/hardlink_rcu.py | 251 +++++++++++++++++++++
56 lib/portage/repository/storage/inplace.py | 49 ++++
57 lib/portage/repository/storage/interface.py | 87 +++++++
58 lib/portage/sync/controller.py | 1 +
59 lib/portage/sync/modules/rsync/rsync.py | 85 ++-----
60 lib/portage/sync/syncbase.py | 33 +++
61 .../tests/util/futures/test_compat_coroutine.py | 14 ++
62 lib/portage/util/futures/_asyncio/__init__.py | 14 ++
63 lib/portage/util/futures/_sync_decorator.py | 54 +++++
64 lib/portage/util/futures/compat_coroutine.py | 12 +
65 man/portage.5 | 35 +++
66 14 files changed, 700 insertions(+), 66 deletions(-)
67 create mode 100644 lib/portage/repository/storage/__init__.py
68 create mode 100644 lib/portage/repository/storage/hardlink_quarantine.py
69 create mode 100644 lib/portage/repository/storage/hardlink_rcu.py
70 create mode 100644 lib/portage/repository/storage/inplace.py
71 create mode 100644 lib/portage/repository/storage/interface.py
72 create mode 100644 lib/portage/util/futures/_sync_decorator.py
73
74 --
75 2.16.4

Replies