1 |
On 03/10/2013 18:57, Paul Hartman wrote: |
2 |
> On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon <alan.mckinnon@×××××.com> wrote: |
3 |
>> On 02/10/2013 19:37, Paul Hartman wrote: |
4 |
>>> On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon <alan.mckinnon@×××××.com> wrote: |
5 |
>>>> On 01/10/2013 17:17, Greg Turner wrote: |
6 |
|
7 |
|
8 |
[snip] |
9 |
|
10 |
|
11 |
>> You don't sync very often, right? |
12 |
> |
13 |
> I usually sync manually daily or every other day if I'm busy and don't |
14 |
> get a chance. I assumed there was some mass change to ebuild headers |
15 |
> or license text or something which caused everything in the tree to |
16 |
> get touched this week. |
17 |
> |
18 |
> My local portage tree is on a fast SSD in an 8-core box with 32GB of |
19 |
> RAM and a 100mbit internet connection, so the bottleneck hopefully is |
20 |
> not on my side of the transaction. ;) |
21 |
> |
22 |
> Let's do some more trials. Between yesterday and today, I have synced |
23 |
> with my normal mirror, but I'm syncing with your server again now: |
24 |
> |
25 |
> Number of files: 174410 |
26 |
> Number of files transferred: 17372 |
27 |
> Total file size: 306.28M bytes |
28 |
> Total transferred file size: 22.32M bytes |
29 |
> Literal data: 22.32M bytes |
30 |
> Matched data: 0 bytes |
31 |
> File list size: 4.31M |
32 |
> File list generation time: 379.920 seconds |
33 |
> File list transfer time: 0.000 seconds |
34 |
> Total bytes sent: 382.35K |
35 |
> Total bytes received: 15.71M |
36 |
> |
37 |
> sent 382.35K bytes received 15.71M bytes 29.33K bytes/sec |
38 |
> total size is 306.28M speedup is 19.04 |
39 |
> |
40 |
> Now I'm immediately doing another sync, first deleting timestamp.chk |
41 |
> to force it to sync again. There should be zero files to transfer |
42 |
> (except the timestamp file). |
43 |
> |
44 |
> Number of files: 174410 |
45 |
> Number of files transferred: 1 |
46 |
> Total file size: 306.28M bytes |
47 |
> Total transferred file size: 32 bytes |
48 |
> Literal data: 32 bytes |
49 |
> Matched data: 0 bytes |
50 |
> File list size: 4.31M |
51 |
> File list generation time: 28.612 seconds |
52 |
> File list transfer time: 0.000 seconds |
53 |
> Total bytes sent: 183 |
54 |
> Total bytes received: 4.31M |
55 |
> |
56 |
> sent 183 bytes received 4.31M bytes 128.75K bytes/sec |
57 |
> total size is 306.28M speedup is 71.01 |
58 |
> |
59 |
> |
60 |
> Now I'm switching back to my beloved mirror.steadfast.net and running |
61 |
> another sync. |
62 |
> |
63 |
> Number of files: 174409 |
64 |
> Number of files transferred: 17364 |
65 |
> Total file size: 306.30M bytes |
66 |
> Total transferred file size: 21.74M bytes |
67 |
> Literal data: 21.74M bytes |
68 |
> Matched data: 0 bytes |
69 |
> File list size: 4.39M |
70 |
> File list generation time: 0.001 seconds |
71 |
> File list transfer time: 0.000 seconds |
72 |
> Total bytes sent: 355.23K |
73 |
> Total bytes received: 15.67M |
74 |
> |
75 |
> sent 355.23K bytes received 15.67M bytes 191.93K bytes/sec |
76 |
> total size is 306.30M speedup is 19.11 |
77 |
> |
78 |
> |
79 |
> Interestingly it transferred almost the same number of files as my |
80 |
> first sync with yours. Comparing timestamps, your server's latest |
81 |
> update is about 5 hours older than Steadfast's, so things must be |
82 |
> changing frequently in portage these days! 17k changes in 5 hours... |
83 |
|
84 |
|
85 |
My guess is the metadata. I'll have to do some checks on that mirror, |
86 |
IIRC it syncs every 6 hours, most likely steadfast syncs more often. |
87 |
|
88 |
One of the things about running a mirror is, it's very much set it up |
89 |
once and forget all about it evermore. Which is great and all, but users |
90 |
tend to spot problems long before the sysadmins do :-) |
91 |
|
92 |
|
93 |
> |
94 |
> My ping to your server is 300ms, my ping to steadfast is 18ms. I don't |
95 |
> know anything about how rsync works behind the curtain, if a higher |
96 |
> latency would cause the file list generation to be slower, or if that |
97 |
> is a measurement of server performance or something else. |
98 |
|
99 |
|
100 |
I don't think latency is much of a factor but let me re-read some FAQs |
101 |
before commenting further. |
102 |
|
103 |
300ms is totally normal from here to eu, uk and us - we're in deepest |
104 |
darkest Africa where hyenas prowl the streets[1] - and all traffic goes |
105 |
over undersea cable with *lots* of repeaters |
106 |
|
107 |
|
108 |
> |
109 |
> Total sync times from my log: |
110 |
> |
111 |
> 1380814364: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage |
112 |
> 1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage |
113 |
> (first sync, 17k files updated, 552 seconds) |
114 |
> |
115 |
> 1380815150: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage |
116 |
> 1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage |
117 |
> (sync with no updates except timestamp.chk, 38 seconds) |
118 |
> |
119 |
> 1380815292: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage |
120 |
> 1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage |
121 |
> (re-sync with steadfast, 17k files updated, 83 seconds) |
122 |
> |
123 |
> 1380816062: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage |
124 |
> 1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage |
125 |
> (sync with no updates except timestamp.chk, 12 seconds) |
126 |
> |
127 |
> |
128 |
> HTH and thanks for the mirror :) |
129 |
> Paul |
130 |
> |
131 |
|
132 |
[1] Literally. True's bob, I kid you not. A 6 month old brown hyena this |
133 |
week wandered into the suburb where I live - it must have got separated |
134 |
from it's mother and walked 15 miles in the dark to get here |
135 |
|
136 |
|
137 |
|
138 |
-- |
139 |
Alan McKinnon |
140 |
alan.mckinnon@×××××.com |