| 1 |
On Mon, 19 Apr 2010 17:31:28 +0200, Michał Górny <gentoo@××××××××××.pl> |
| 2 |
wrote: |
| 3 |
> Hello, |
| 4 |
|
| 5 |
Hello, overall I am confused about what you are asking or proposing. Some |
| 6 |
comments inline. |
| 7 |
|
| 8 |
> |
| 9 |
> Another important topic related to the git migration is how rsync tree |
| 10 |
> is supposed to be created. Here is how I see it. |
| 11 |
> |
| 12 |
> I. GIT repository side. |
| 13 |
> |
| 14 |
> The only related mechanism at the GIT repository would be a post-commit |
| 15 |
> hook, sending a signal to the rsync server. The signal would only |
| 16 |
> notify the rsync side that the repository needs to be pulled, without |
| 17 |
> further information. |
| 18 |
|
| 19 |
On the rsync staging server, it should/will be time interval based. |
| 20 |
|
| 21 |
> |
| 22 |
> II. rsync server side. |
| 23 |
> |
| 24 |
> The main part of the server layout is a hybrid git-rsync tree. It |
| 25 |
> starts as a GIT repository checkout, which is being filled with missing |
| 26 |
> data (i.e. ebuild checksums) by rsync scripting. |
| 27 |
|
| 28 |
I feel like this should be solved by a new manifest proposal not in the |
| 29 |
git->rsync generation. Eg. MetaManifests or ThinManifests. |
| 30 |
|
| 31 |
> After receiving the signal, the rsync scripting pulls from |
| 32 |
> the repository and updates the working copy. Afterwards, it determines |
| 33 |
> which files have changed (using diffstat?) and calls repoman (or other |
| 34 |
> script) to regenerate the full Manifests for the related directories. |
| 35 |
|
| 36 |
repoman seems like it would be too expensive for this. |
| 37 |
|
| 38 |
> That's the simplest scenario. In fact, as our working tree would |
| 39 |
> contain Manifests changed for rsync users, most of the merges would |
| 40 |
> result in conflicts -- which should be resolved through always using |
| 41 |
> the remote version (as our own is going to be recreated anyway). |
| 42 |
> |
| 43 |
> The problem is that git doesn't seem to like having conflicts with |
| 44 |
> local uncommited changes. Is there a way to force it to replace them |
| 45 |
> with the remote versions? |
| 46 |
> |
| 47 |
> There would be probably some more problems related to file removal too. |
| 48 |
|
| 49 |
Where would the file removal problems be? rsync (emerge --sync) will |
| 50 |
handle this like normal. |
| 51 |
|
| 52 |
> |
| 53 |
> Another useful thing would be atomization of the rsync updates to avoid |
| 54 |
> the possibility of downloading Thin Manifests by end user. The simplest |
| 55 |
> solution for that would be to keep two distinct trees, with working |
| 56 |
> tree being rsynced onto public tree after the updates are done. |
| 57 |
|
| 58 |
This is handled on the rsync staging host as well. All downstream mirrors |
| 59 |
will get an "atomic tree" from the staged copy. |
| 60 |
|
| 61 |
> |
| 62 |
> But that requires keeping two copies of the tree, and doesn't provide |
| 63 |
> full atomization anyway (it's still possible that end-user downloads |
| 64 |
> partially copied directory, i.e. with new ebuilds and Manifest not yet |
| 65 |
> updated). |
| 66 |
> |
| 67 |
> Thus, it'd be best to have the atomization (i.e. commit-update) |
| 68 |
> behaviour on filesystem level. If that's not possible, we could still |
| 69 |
> use poor man's solution similar to VGA page switching - two switching |
| 70 |
> rsync trees, one being used by end-users and the other being updated. |
| 71 |
|
| 72 |
In short, let's work out a defined Manifest proposal before worrying about |
| 73 |
the rsync generation for mirrors. I think that this proposal is just |
| 74 |
re-highlighting a known issue. Issue being, current Manifests are not |
| 75 |
ideal. |
| 76 |
|
| 77 |
-Jeremy |
Archives