1 |
On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon <alan.mckinnon@×××××.com> wrote: |
2 |
> On 02/10/2013 19:37, Paul Hartman wrote: |
3 |
>> On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon <alan.mckinnon@×××××.com> wrote: |
4 |
>>> On 01/10/2013 17:17, Greg Turner wrote: |
5 |
>>>> Rsync mirrors don't grow on trees, man. People pay good money to |
6 |
>>>> provide that service to us. You should seriously be embarrassed to |
7 |
>>>> have posted this. |
8 |
>>> |
9 |
>>> Really? |
10 |
>>> |
11 |
>>> |
12 |
>>> Then you can all use mine with the greatest of pleasure: |
13 |
>>> |
14 |
>>> SYNC="rsync://ftp.is.co.za/gentoo-portage" |
15 |
>>> |
16 |
>>> |
17 |
>>> I have the NetOps team BEGGING me weekly to try and generate more |
18 |
>>> traffic out of our network going international. The in-out ratio on the |
19 |
>>> peering links is seriously screwed and they badly want something to even |
20 |
>>> it out a bit :-) |
21 |
>> |
22 |
>> Challenge accepted. :) |
23 |
>> |
24 |
>> Here's my sync summary (I'm in the USA): |
25 |
>> |
26 |
>> Number of files: 174305 |
27 |
>> Number of files transferred: 50913 |
28 |
> ^^^^^ |
29 |
>> Total file size: 305.99M bytes |
30 |
>> Total transferred file size: 73.98M bytes |
31 |
>> Literal data: 73.98M bytes |
32 |
>> Matched data: 0 bytes |
33 |
>> File list size: 4.31M |
34 |
>> File list generation time: 343.526 seconds |
35 |
> ^^^^^^^^^^^^^^^ |
36 |
>> File list transfer time: 0.000 seconds |
37 |
>> Total bytes sent: 1.12M |
38 |
>> Total bytes received: 41.37M |
39 |
>> |
40 |
>> sent 1.12M bytes received 41.37M bytes 64.33K bytes/sec |
41 |
>> total size is 305.99M speedup is 7.20 |
42 |
> |
43 |
> |
44 |
> You don't sync very often, right? |
45 |
|
46 |
I usually sync manually daily or every other day if I'm busy and don't |
47 |
get a chance. I assumed there was some mass change to ebuild headers |
48 |
or license text or something which caused everything in the tree to |
49 |
get touched this week. |
50 |
|
51 |
My local portage tree is on a fast SSD in an 8-core box with 32GB of |
52 |
RAM and a 100mbit internet connection, so the bottleneck hopefully is |
53 |
not on my side of the transaction. ;) |
54 |
|
55 |
Let's do some more trials. Between yesterday and today, I have synced |
56 |
with my normal mirror, but I'm syncing with your server again now: |
57 |
|
58 |
Number of files: 174410 |
59 |
Number of files transferred: 17372 |
60 |
Total file size: 306.28M bytes |
61 |
Total transferred file size: 22.32M bytes |
62 |
Literal data: 22.32M bytes |
63 |
Matched data: 0 bytes |
64 |
File list size: 4.31M |
65 |
File list generation time: 379.920 seconds |
66 |
File list transfer time: 0.000 seconds |
67 |
Total bytes sent: 382.35K |
68 |
Total bytes received: 15.71M |
69 |
|
70 |
sent 382.35K bytes received 15.71M bytes 29.33K bytes/sec |
71 |
total size is 306.28M speedup is 19.04 |
72 |
|
73 |
Now I'm immediately doing another sync, first deleting timestamp.chk |
74 |
to force it to sync again. There should be zero files to transfer |
75 |
(except the timestamp file). |
76 |
|
77 |
Number of files: 174410 |
78 |
Number of files transferred: 1 |
79 |
Total file size: 306.28M bytes |
80 |
Total transferred file size: 32 bytes |
81 |
Literal data: 32 bytes |
82 |
Matched data: 0 bytes |
83 |
File list size: 4.31M |
84 |
File list generation time: 28.612 seconds |
85 |
File list transfer time: 0.000 seconds |
86 |
Total bytes sent: 183 |
87 |
Total bytes received: 4.31M |
88 |
|
89 |
sent 183 bytes received 4.31M bytes 128.75K bytes/sec |
90 |
total size is 306.28M speedup is 71.01 |
91 |
|
92 |
|
93 |
Now I'm switching back to my beloved mirror.steadfast.net and running |
94 |
another sync. |
95 |
|
96 |
Number of files: 174409 |
97 |
Number of files transferred: 17364 |
98 |
Total file size: 306.30M bytes |
99 |
Total transferred file size: 21.74M bytes |
100 |
Literal data: 21.74M bytes |
101 |
Matched data: 0 bytes |
102 |
File list size: 4.39M |
103 |
File list generation time: 0.001 seconds |
104 |
File list transfer time: 0.000 seconds |
105 |
Total bytes sent: 355.23K |
106 |
Total bytes received: 15.67M |
107 |
|
108 |
sent 355.23K bytes received 15.67M bytes 191.93K bytes/sec |
109 |
total size is 306.30M speedup is 19.11 |
110 |
|
111 |
|
112 |
Interestingly it transferred almost the same number of files as my |
113 |
first sync with yours. Comparing timestamps, your server's latest |
114 |
update is about 5 hours older than Steadfast's, so things must be |
115 |
changing frequently in portage these days! 17k changes in 5 hours... |
116 |
|
117 |
My ping to your server is 300ms, my ping to steadfast is 18ms. I don't |
118 |
know anything about how rsync works behind the curtain, if a higher |
119 |
latency would cause the file list generation to be slower, or if that |
120 |
is a measurement of server performance or something else. |
121 |
|
122 |
Total sync times from my log: |
123 |
|
124 |
1380814364: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage |
125 |
1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage |
126 |
(first sync, 17k files updated, 552 seconds) |
127 |
|
128 |
1380815150: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage |
129 |
1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage |
130 |
(sync with no updates except timestamp.chk, 38 seconds) |
131 |
|
132 |
1380815292: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage |
133 |
1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage |
134 |
(re-sync with steadfast, 17k files updated, 83 seconds) |
135 |
|
136 |
1380816062: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage |
137 |
1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage |
138 |
(sync with no updates except timestamp.chk, 12 seconds) |
139 |
|
140 |
|
141 |
HTH and thanks for the mirror :) |
142 |
Paul |