1 |
On Mon, 19 Apr 2010 17:31:28 +0200, Michał Górny <gentoo@××××××××××.pl> |
2 |
wrote: |
3 |
> Hello, |
4 |
|
5 |
Hello, overall I am confused about what you are asking or proposing. Some |
6 |
comments inline. |
7 |
|
8 |
> |
9 |
> Another important topic related to the git migration is how rsync tree |
10 |
> is supposed to be created. Here is how I see it. |
11 |
> |
12 |
> I. GIT repository side. |
13 |
> |
14 |
> The only related mechanism at the GIT repository would be a post-commit |
15 |
> hook, sending a signal to the rsync server. The signal would only |
16 |
> notify the rsync side that the repository needs to be pulled, without |
17 |
> further information. |
18 |
|
19 |
On the rsync staging server, it should/will be time interval based. |
20 |
|
21 |
> |
22 |
> II. rsync server side. |
23 |
> |
24 |
> The main part of the server layout is a hybrid git-rsync tree. It |
25 |
> starts as a GIT repository checkout, which is being filled with missing |
26 |
> data (i.e. ebuild checksums) by rsync scripting. |
27 |
|
28 |
I feel like this should be solved by a new manifest proposal not in the |
29 |
git->rsync generation. Eg. MetaManifests or ThinManifests. |
30 |
|
31 |
> After receiving the signal, the rsync scripting pulls from |
32 |
> the repository and updates the working copy. Afterwards, it determines |
33 |
> which files have changed (using diffstat?) and calls repoman (or other |
34 |
> script) to regenerate the full Manifests for the related directories. |
35 |
|
36 |
repoman seems like it would be too expensive for this. |
37 |
|
38 |
> That's the simplest scenario. In fact, as our working tree would |
39 |
> contain Manifests changed for rsync users, most of the merges would |
40 |
> result in conflicts -- which should be resolved through always using |
41 |
> the remote version (as our own is going to be recreated anyway). |
42 |
> |
43 |
> The problem is that git doesn't seem to like having conflicts with |
44 |
> local uncommited changes. Is there a way to force it to replace them |
45 |
> with the remote versions? |
46 |
> |
47 |
> There would be probably some more problems related to file removal too. |
48 |
|
49 |
Where would the file removal problems be? rsync (emerge --sync) will |
50 |
handle this like normal. |
51 |
|
52 |
> |
53 |
> Another useful thing would be atomization of the rsync updates to avoid |
54 |
> the possibility of downloading Thin Manifests by end user. The simplest |
55 |
> solution for that would be to keep two distinct trees, with working |
56 |
> tree being rsynced onto public tree after the updates are done. |
57 |
|
58 |
This is handled on the rsync staging host as well. All downstream mirrors |
59 |
will get an "atomic tree" from the staged copy. |
60 |
|
61 |
> |
62 |
> But that requires keeping two copies of the tree, and doesn't provide |
63 |
> full atomization anyway (it's still possible that end-user downloads |
64 |
> partially copied directory, i.e. with new ebuilds and Manifest not yet |
65 |
> updated). |
66 |
> |
67 |
> Thus, it'd be best to have the atomization (i.e. commit-update) |
68 |
> behaviour on filesystem level. If that's not possible, we could still |
69 |
> use poor man's solution similar to VGA page switching - two switching |
70 |
> rsync trees, one being used by end-users and the other being updated. |
71 |
|
72 |
In short, let's work out a defined Manifest proposal before worrying about |
73 |
the rsync generation for mirrors. I think that this proposal is just |
74 |
re-highlighting a known issue. Issue being, current Manifests are not |
75 |
ideal. |
76 |
|
77 |
-Jeremy |