1 |
On Thu, Jul 02, 2015 at 06:05:10PM -0400, Rich Freeman wrote: |
2 |
> On Thu, Jul 2, 2015 at 5:39 PM, Robin H. Johnson <robbat2@g.o> wrote: |
3 |
> > 2015/08/08 15:00 UTC - Freeze |
4 |
> > 2015/08/11 - History repo available to graft |
5 |
> I'm not sure if you already had plans for creating the history, but |
6 |
> I've generally been able to turn them around in 12 hours without |
7 |
> really trying too hard. |
8 |
I expect verification of the conversion to take longer than the runs, |
9 |
and that we're going to find we miss something at least once, so it'll |
10 |
be 2 or 3 runs before we get output we're happy with. |
11 |
|
12 |
> I have it all running in a container/chroot |
13 |
> where I mount the cvsroot so I could probably get it running on EC2 or |
14 |
> elsewhere easily enough if we're in a real hurry. |
15 |
Can you publish the container? Letting other people be able to replicate |
16 |
the result here would be a benefit, and we can easily shuffle it as |
17 |
well. |
18 |
|
19 |
> If I had a ton of RAM and store everything in tmpfs then I suspect |
20 |
> that it could be done faster - |
21 |
How much RAM & CPU have you been running it with? I was planning on |
22 |
64GB+ tmpfs and 12+ dedicated cores. |
23 |
|
24 |
> ferringb's design does each package category in a thread so the |
25 |
> largest categories become the limiter if you have enough cores (it |
26 |
> does them in reverse order by size so the longest ones do start |
27 |
> first). |
28 |
Yes, that was the design I settled on with him. |
29 |
|
30 |
> At some point the fetching of cvsroots and pushing of git |
31 |
> repos becomes limiting - for github the push takes tens of minutes, |
32 |
> and the repository takes much longer to show up, though I'm sure that |
33 |
> could all be done faster on our infra, either via git push or letting |
34 |
> infra clone from a bundle. |
35 |
I did a test push of a full conversion previously, to measure the |
36 |
performance, and it was pretty much bandwidth-limited for us (at |
37 |
more than 100Mbit inbound) not by anything else. |
38 |
|
39 |
> With git replace we don't need the history to go live, and this also |
40 |
> means that if we find a mistake in the history we can fix it and issue |
41 |
> a new history. So, there isn't really a rush. But, if you don't |
42 |
> already have the history conversion taken care of or need it faster, I |
43 |
> can take care of it. |
44 |
The history is needed within a few days if we go with the changelog |
45 |
generation step; but is NOT in the critical path for developers being |
46 |
able to carry on committing. |
47 |
|
48 |
> The most recent conversion is at: |
49 |
> https://github.com/gentoo/gentoo-gitmig-20150614 |
50 |
Can you publish the container you mention, rather than just the results? |
51 |
|
52 |
-- |
53 |
Robin Hugh Johnson |
54 |
Gentoo Linux: Developer, Infrastructure Lead |
55 |
E-Mail : robbat2@g.o |
56 |
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 |