1 |
On 06/11/2014 10:34 AM, Stefan G. Weichinger wrote: |
2 |
> Am 11.06.2014 11:19, schrieb thegeezer: |
3 |
> |
4 |
>> Hi Stefan, |
5 |
>> block size / stripe size mismatches only really penalise random io, if |
6 |
>> you are trying to use dd and have slow speeds this would suggest |
7 |
>> something else is awry. |
8 |
>> I don't know the c600 rad chip personally, but in trying to google it it |
9 |
>> appears to be a motherboard based raid device? is it a real raid or |
10 |
>> fakeraid? |
11 |
>> |
12 |
>> I'm a little confused over your setup to help. i'm sorry if there is |
13 |
>> duplication but it would be useful to have all info in one hit rather |
14 |
>> than trying to piece it together from all your messages. |
15 |
> OK, will do ... |
16 |
> |
17 |
> |
18 |
>> 1. please can you list your hardware raid config. I'm looking for the |
19 |
>> physical disk sizes, the virtual disks and their raid types. do you have |
20 |
>> cache enabled on the raid card, is there a background scrub or anything |
21 |
>> like that running? do you have active seek/ prefetch configured ? |
22 |
>> parity size being 50% of total size is just odd to me - but i guess |
23 |
>> these are mirrors ? but it says raid-level3 --- just odd, most setups |
24 |
>> use raid0(not raid) raid1(mirror) raid5(parity stripe) raid6 (double |
25 |
>> parity stripe) or combinations, like 50.. raid3 is allocating a single |
26 |
>> disk to parity but is very rarely used. |
27 |
> Basically 3 RAID-6 hw-raids over 6 SAS hdds. |
28 |
|
29 |
OK so i'm confused again. RAID6 requires minimum of 4 drives. |
30 |
if you have 3 raid6's then you would need 12 drives (coffee hasn't quite |
31 |
activated in me yet so my maths may not be right) |
32 |
or do you have essentially the first part of each of the six drives be |
33 |
virtual disk 1, the second part of each of the six drives virtual disk 2 |
34 |
and the third part be virtual disk 3 -- if this is the case bear in mind |
35 |
that the slowest part of the disk is the end of the disk -- so you are |
36 |
essentially hobbling your virtual disk3 but only a little, instead of |
37 |
being around 150MB/sec it might run at 80. |
38 |
|
39 |
you might also like to try a simple test of the following (yes lvs count |
40 |
as block devices) |
41 |
# hdparm -t /dev/sda |
42 |
# hdparm -t /dev/sdb |
43 |
# hdparm -t /dev/sdc |
44 |
# hdparm -t /dev/vg01/winserver_disk0 |
45 |
# hdparm -t /dev/vg01/amhold |
46 |
|
47 |
> I don't know where this RAID-3 term comes from -> |
48 |
> |
49 |
> # megacli -LDInfo -Lall -aALL |
50 |
> |
51 |
> |
52 |
> Adapter 0 -- Virtual Drive Information: |
53 |
> Virtual Drive: 0 (Target Id: 0) |
54 |
> Name :root |
55 |
> RAID Level : Primary-6, Secondary-3, RAID Level Qualifier-3 |
56 |
> Size : 500.0 GB |
57 |
> Sector Size : 512 |
58 |
> Is VD emulated : No |
59 |
> Parity Size : 250.0 GB |
60 |
> State : Optimal |
61 |
> Strip Size : 256 KB |
62 |
> Number Of Drives : 6 |
63 |
> Span Depth : 1 |
64 |
> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if |
65 |
> Bad BBU |
66 |
> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if |
67 |
> Bad BBU |
68 |
> Default Access Policy: Read/Write |
69 |
> Current Access Policy: Read/Write |
70 |
> Disk Cache Policy : Disabled |
71 |
> Encryption Type : None |
72 |
> Bad Blocks Exist: No |
73 |
> Is VD Cached: No |
74 |
> |
75 |
> |
76 |
> Virtual Drive: 1 (Target Id: 1) |
77 |
> Name :swap |
78 |
> RAID Level : Primary-6, Secondary-3, RAID Level Qualifier-3 |
79 |
> Size : 8.0 GB |
80 |
> Sector Size : 512 |
81 |
> Is VD emulated : No |
82 |
> Parity Size : 4.0 GB |
83 |
> State : Optimal |
84 |
> Strip Size : 64 KB |
85 |
> Number Of Drives : 6 |
86 |
> Span Depth : 1 |
87 |
> Default Cache Policy: WriteBack, ReadAheadNone, Cached, No Write Cache |
88 |
> if Bad BBU |
89 |
> Current Cache Policy: WriteBack, ReadAheadNone, Cached, No Write Cache |
90 |
> if Bad BBU |
91 |
> Default Access Policy: Read/Write |
92 |
> Current Access Policy: Read/Write |
93 |
> Disk Cache Policy : Disabled |
94 |
> Encryption Type : None |
95 |
> Bad Blocks Exist: No |
96 |
> Is VD Cached: No |
97 |
> |
98 |
> |
99 |
> Virtual Drive: 2 (Target Id: 2) |
100 |
> Name :lvm |
101 |
> RAID Level : Primary-6, Secondary-3, RAID Level Qualifier-3 |
102 |
> Size : 1.321 TB |
103 |
> Sector Size : 512 |
104 |
> Is VD emulated : No |
105 |
> Parity Size : 676.5 GB |
106 |
> State : Optimal |
107 |
> Strip Size : 64 KB |
108 |
> Number Of Drives : 6 |
109 |
> Span Depth : 1 |
110 |
> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if |
111 |
> Bad BBU |
112 |
> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if |
113 |
> Bad BBU |
114 |
> Default Access Policy: Read/Write |
115 |
> Current Access Policy: Read/Write |
116 |
> Disk Cache Policy : Disabled |
117 |
> Encryption Type : None |
118 |
> Bad Blocks Exist: No |
119 |
> Is VD Cached: No |
120 |
> |
121 |
> |
122 |
> |
123 |
> |
124 |
>> 2. how many other devices are actively doing IO? do you have any other |
125 |
>> raid cards/io cards of note that might be clashign on the board. |
126 |
> The Intel C600 Controller seems to only run the LTO-4-drive in the |
127 |
> server while the |
128 |
> |
129 |
> LSI Logic / Symbios Logic MegaRAID SAS 2108 |
130 |
> |
131 |
> runs the 6 hard disks. |
132 |
> |
133 |
> # lspci |
134 |
> 00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07) |
135 |
> 00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express |
136 |
> Root Port 1a (rev 07) |
137 |
> 00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express |
138 |
> Root Port 1b (rev 07) |
139 |
> 00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express |
140 |
> Root Port 3a in PCI Express Mode (rev 07) |
141 |
> 00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address |
142 |
> Map, VTd_Misc, System Management (rev 07) |
143 |
> 00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control |
144 |
> Status and Global Errors (rev 07) |
145 |
> 00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07) |
146 |
> 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI |
147 |
> Express Virtual Root Port (rev 05) |
148 |
> 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 |
149 |
> Enhanced Host Controller #2 (rev 05) |
150 |
> 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI |
151 |
> Express Root Port 1 (rev b5) |
152 |
> 00:1c.4 PCI bridge: Intel Corporation C600/X79 series chipset PCI |
153 |
> Express Root Port 5 (rev b5) |
154 |
> 00:1c.5 PCI bridge: Intel Corporation C600/X79 series chipset PCI |
155 |
> Express Root Port 6 (rev b5) |
156 |
> 00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI |
157 |
> Express Root Port 8 (rev b5) |
158 |
> 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 |
159 |
> Enhanced Host Controller #1 (rev 05) |
160 |
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) |
161 |
> 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC |
162 |
> Controller (rev 05) |
163 |
> 00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset |
164 |
> 6-Port SATA AHCI Controller (rev 05) |
165 |
> 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host |
166 |
> Controller (rev 05) |
167 |
> 02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 |
168 |
> [Liberator] (rev 05) |
169 |
> 04:00.0 Serial Attached SCSI controller: Intel Corporation C604/X79 |
170 |
> series chipset 4-Port SATA/SAS Storage Control Unit (rev 05) |
171 |
> 04:00.3 SMBus: Intel Corporation C600/X79 series chipset SMBus |
172 |
> Controller 0 (rev 05) |
173 |
> 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network |
174 |
> Connection |
175 |
> 07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network |
176 |
> Connection |
177 |
> 08:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA |
178 |
> G200e [Pilot] ServerEngines (SEP1) (rev 05) |
179 |
> ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 |
180 |
> (rev 07) |
181 |
> ff:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link |
182 |
> Reut 0 (rev 07) |
183 |
> ff:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link |
184 |
> Reut 0 (rev 07) |
185 |
> ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 |
186 |
> (rev 07) |
187 |
> ff:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link |
188 |
> Reut 1 (rev 07) |
189 |
> ff:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link |
190 |
> Reut 1 (rev 07) |
191 |
> ff:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power |
192 |
> Control Unit 0 (rev 07) |
193 |
> ff:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power |
194 |
> Control Unit 1 (rev 07) |
195 |
> ff:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power |
196 |
> Control Unit 2 (rev 07) |
197 |
> ff:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power |
198 |
> Control Unit 3 (rev 07) |
199 |
> ff:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt |
200 |
> Control Registers (rev 07) |
201 |
> ff:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore |
202 |
> and Scratchpad Configuration Registers (rev 07) |
203 |
> ff:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast |
204 |
> Register 0 (rev 07) |
205 |
> ff:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast |
206 |
> Register 0 (rev 07) |
207 |
> ff:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
208 |
> Memory Controller System Address Decoder 0 (rev 07) |
209 |
> ff:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System |
210 |
> Address Decoder (rev 07) |
211 |
> ff:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast |
212 |
> Register 0 (rev 07) |
213 |
> ff:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast |
214 |
> Register 0 (rev 07) |
215 |
> ff:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
216 |
> Memory Controller System Address Decoder 1 (rev 07) |
217 |
> ff:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor |
218 |
> Home Agent (rev 07) |
219 |
> ff:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 |
220 |
> Processor Home Agent Performance Monitoring (rev 07) |
221 |
> ff:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
222 |
> Memory Controller Registers (rev 07) |
223 |
> ff:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
224 |
> Memory Controller RAS Registers (rev 07) |
225 |
> ff:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
226 |
> Memory Controller Target Address Decoder 0 (rev 07) |
227 |
> ff:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
228 |
> Memory Controller Target Address Decoder 1 (rev 07) |
229 |
> ff:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
230 |
> Memory Controller Target Address Decoder 2 (rev 07) |
231 |
> ff:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
232 |
> Memory Controller Target Address Decoder 3 (rev 07) |
233 |
> ff:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
234 |
> Memory Controller Target Address Decoder 4 (rev 07) |
235 |
> ff:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
236 |
> Memory Controller Channel 0-3 Thermal Control 0 (rev 07) |
237 |
> ff:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
238 |
> Memory Controller Channel 0-3 Thermal Control 1 (rev 07) |
239 |
> ff:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
240 |
> Memory Controller ERROR Registers 0 (rev 07) |
241 |
> ff:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
242 |
> Memory Controller ERROR Registers 1 (rev 07) |
243 |
> ff:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
244 |
> Memory Controller Channel 0-3 Thermal Control 3 (rev 07) |
245 |
> ff:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
246 |
> Memory Controller ERROR Registers 2 (rev 07) |
247 |
> ff:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated |
248 |
> Memory Controller ERROR Registers 3 (rev 07) |
249 |
> ff:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07) |
250 |
> ff:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07) |
251 |
> ff:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to |
252 |
> PCI Express Performance Monitor (rev 07) |
253 |
> ff:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 |
254 |
> QuickPath Interconnect Agent Ring Registers (rev 07) |
255 |
> ff:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to |
256 |
> QuickPath Interconnect Link 0 Performance Monitor (rev 07) |
257 |
> ff:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to |
258 |
> QuickPath Interconnect Link 1 Performance Monitor (rev 07) |
259 |
> |
260 |
|
261 |
i notice the core i7 only now. have you disabled turbo boost in the bios ? |
262 |
this is great for a desktop but awful for a server as it disables all |
263 |
those extra cores for a single busy thread |
264 |
|
265 |
>> 3. do you have active I/O when doing your performance tests? if you have |
266 |
>> several virtual machines running depending on what they are doing they |
267 |
>> will crucify your access. |
268 |
> My testing was done with all the VMs stopped! |
269 |
eek |
270 |
> |
271 |
> |
272 |
>> 4. are you using any type of CGroups ? |
273 |
> Yes. systemd needs them. |
274 |
> I attach the kernel-config. |
275 |
ok it would be useful to know how the storage cgroups are configured, as |
276 |
this will also hobble things. |
277 |
cgroups are a great way of limiting or guaranteeing performance. by |
278 |
default i believe systemd will aim for user interactivity, but you want |
279 |
to change that to be more balanced. |
280 |
maybe some else can suggest how best to configure systemd cgroups. |
281 |
meanwhile can you |
282 |
# tree /sys/fs/cgroup/ |
283 |
|
284 |
>> 5. i'm also confused over your LVM config. please can you send the |
285 |
>> output of "vgs" "pvs" and "lvs -a -o +devices" |
286 |
> /dev/sdc is the third hardware raid array listed above and used as PV |
287 |
> for the VG vg01 -> |
288 |
> |
289 |
> # pvs |
290 |
> PV VG Fmt Attr PSize PFree |
291 |
> /dev/sdc vg01 lvm2 a-- 1,32t 708,86g |
292 |
> |
293 |
> # vgs |
294 |
> VG #PV #LV #SN Attr VSize VFree |
295 |
> vg01 1 2 0 wz--n- 1,32t 708,86g |
296 |
> |
297 |
> # lvs -a -o +devices |
298 |
> LV VG Attr LSize Pool Origin Data% Move Log |
299 |
> Cpy%Sync Convert Devices |
300 |
> amhold vg01 -wi-ao---- 400,00g |
301 |
> /dev/sdc(0) |
302 |
> winserver_disk0 vg01 -wi-ao---- 244,14g |
303 |
> /dev/sdc(102400) |
304 |
> |
305 |
> |
306 |
>> 6. please also send the output of "mount" |
307 |
> # mount |
308 |
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) |
309 |
> sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) |
310 |
> devtmpfs on /dev type devtmpfs |
311 |
> (rw,nosuid,size=8203540k,nr_inodes=2050885,mode=755) |
312 |
> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) |
313 |
> devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620) |
314 |
> tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) |
315 |
> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) |
316 |
> cgroup on /sys/fs/cgroup/systemd type cgroup |
317 |
> (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) |
318 |
> cgroup on /sys/fs/cgroup/cpuset type cgroup |
319 |
> (rw,nosuid,nodev,noexec,relatime,cpuset) |
320 |
> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup |
321 |
> (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu) |
322 |
> cgroup on /sys/fs/cgroup/devices type cgroup |
323 |
> (rw,nosuid,nodev,noexec,relatime,devices) |
324 |
> /dev/sda on / type btrfs (rw,noatime,space_cache) |
325 |
> mqueue on /dev/mqueue type mqueue (rw,relatime) |
326 |
> tmpfs on /tmp type tmpfs (rw) |
327 |
> tmpfs on /var/tmp/portage type tmpfs (rw,relatime) |
328 |
> /dev/mapper/vg01-amhold on /mnt/amhold type xfs |
329 |
> (rw,noatime,attr2,inode64,noquota) |
330 |
> tmpfs on /run/user/0 type tmpfs |
331 |
> (rw,nosuid,nodev,relatime,size=1641324k,mode=700) |
332 |
> /dev/sda on /mnt/btrfs_windows type btrfs (rw,noatime,space_cache) |
333 |
|
334 |
a little odd that you have no partitions but otherwise minor. |
335 |
|
336 |
|
337 |
> |
338 |
> the last line is meant as a target directory for dumping the content of |
339 |
> the LV winserver_disk0 ... it is a btrfs subvolume mounted with |
340 |
> compression turned OFF. |
341 |
> |
342 |
|
343 |
if you consider that your drives are on the same raidset, they are all |
344 |
essentially "one disk". |
345 |
copying from one disk to the same disk you are halving the speed it will |
346 |
work at. |
347 |
|
348 |
|
349 |
> |
350 |
>> 7. do you have atop or iotop that you can use to monitor performance - |
351 |
>> specifically we are looking for disk ios per device and disk latency per |
352 |
>> device. both before and during you are trying to run your backup. |
353 |
> I will install atop and check ... |
354 |
> |
355 |
>> this should give us a better idea of where the problems lay. |
356 |
> Thanks so far! |
357 |
> |
358 |
> Stefan |
359 |
> |
360 |
|
361 |
just some extra thoughts |