Re: [gentoo-user] Intel(R) C600 SAS Controller

[Stefan G. Weichinger]

Am 11.06.2014 11:19, schrieb thegeezer:

> Hi Stefan,
> block size / stripe size mismatches only really penalise random io, if
> you are trying to use dd and have slow speeds this would suggest
> something else is awry.
> I don't know the c600 rad chip personally, but in trying to google it it
> appears to be a motherboard based raid device?  is it a real raid or
> fakeraid?
>
> I'm a little confused over your setup to help. i'm sorry if there is
> duplication but it would be useful to have all info in one hit rather
> than trying to piece it together from all your messages.

OK, will do ...


> 1. please can you list your hardware raid config.  I'm looking for the
> physical disk sizes, the virtual disks and their raid types. do you have
> cache enabled on the raid card, is there a background scrub or anything
> like that running? do you have active seek/ prefetch configured ?
> parity size being 50% of total size is just odd to me - but i guess
> these are mirrors ?  but it says raid-level3 --- just odd, most setups
> use raid0(not raid) raid1(mirror) raid5(parity stripe) raid6 (double
> parity stripe) or combinations, like 50.. raid3 is allocating a single
> disk to parity but is very rarely used.

Basically 3 RAID-6 hw-raids over 6 SAS hdds.

I don't know where this RAID-3 term comes from ->

#  megacli -LDInfo -Lall -aALL


Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :root
RAID Level          : Primary-6, Secondary-3, RAID Level Qualifier-3
Size                : 500.0 GB
Sector Size         : 512
Is VD emulated      : No
Parity Size         : 250.0 GB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives    : 6
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 1 (Target Id: 1)
Name                :swap
RAID Level          : Primary-6, Secondary-3, RAID Level Qualifier-3
Size                : 8.0 GB
Sector Size         : 512
Is VD emulated      : No
Parity Size         : 4.0 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 6
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAheadNone, Cached, No Write Cache
if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Cached, No Write Cache
if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 2 (Target Id: 2)
Name                :lvm
RAID Level          : Primary-6, Secondary-3, RAID Level Qualifier-3
Size                : 1.321 TB
Sector Size         : 512
Is VD emulated      : No
Parity Size         : 676.5 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 6
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No




> 2. how many other devices are actively doing IO? do you have any other
> raid cards/io cards of note that might be clashign on the board.

The Intel C600 Controller seems to only run the LTO-4-drive in the
server while the

LSI Logic / Symbios Logic MegaRAID SAS 2108

runs the 6 hard disks.

# lspci
00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express
Root Port 1a (rev 07)
00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express
Root Port 1b (rev 07)
00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express
Root Port 3a in PCI Express Mode (rev 07)
00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address
Map, VTd_Misc, System Management (rev 07)
00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control
Status and Global Errors (rev 07)
00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07)
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI
Express Virtual Root Port (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2
Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI
Express Root Port 1 (rev b5)
00:1c.4 PCI bridge: Intel Corporation C600/X79 series chipset PCI
Express Root Port 5 (rev b5)
00:1c.5 PCI bridge: Intel Corporation C600/X79 series chipset PCI
Express Root Port 6 (rev b5)
00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI
Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2
Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC
Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset
6-Port SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host
Controller (rev 05)
02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108
[Liberator] (rev 05)
04:00.0 Serial Attached SCSI controller: Intel Corporation C604/X79
series chipset 4-Port SATA/SAS Storage Control Unit (rev 05)
04:00.3 SMBus: Intel Corporation C600/X79 series chipset SMBus
Controller 0 (rev 05)
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
08:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA
G200e [Pilot] ServerEngines (SEP1) (rev 05)
ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0
(rev 07)
ff:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link
Reut 0 (rev 07)
ff:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link
Reut 0 (rev 07)
ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1
(rev 07)
ff:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link
Reut 1 (rev 07)
ff:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link
Reut 1 (rev 07)
ff:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power
Control Unit 0 (rev 07)
ff:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power
Control Unit 1 (rev 07)
ff:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power
Control Unit 2 (rev 07)
ff:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power
Control Unit 3 (rev 07)
ff:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt
Control Registers (rev 07)
ff:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore
and Scratchpad Configuration Registers (rev 07)
ff:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast
Register 0 (rev 07)
ff:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast
Register 0 (rev 07)
ff:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller System Address Decoder 0 (rev 07)
ff:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System
Address Decoder (rev 07)
ff:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast
Register 0 (rev 07)
ff:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast
Register 0 (rev 07)
ff:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller System Address Decoder 1 (rev 07)
ff:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor
Home Agent (rev 07)
ff:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7
Processor Home Agent Performance Monitoring (rev 07)
ff:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller Registers (rev 07)
ff:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller RAS Registers (rev 07)
ff:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller Target Address Decoder 0 (rev 07)
ff:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller Target Address Decoder 1 (rev 07)
ff:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller Target Address Decoder 2 (rev 07)
ff:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller Target Address Decoder 3 (rev 07)
ff:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller Target Address Decoder 4 (rev 07)
ff:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
ff:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
ff:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller ERROR Registers 0 (rev 07)
ff:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller ERROR Registers 1 (rev 07)
ff:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
ff:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller ERROR Registers 2 (rev 07)
ff:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated
Memory Controller ERROR Registers 3 (rev 07)
ff:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07)
ff:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07)
ff:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to
PCI Express Performance Monitor (rev 07)
ff:13.4 Performance counters: Intel Corporation Xeon E5/Core i7
QuickPath Interconnect Agent Ring Registers (rev 07)
ff:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to
QuickPath Interconnect Link 0 Performance Monitor (rev 07)
ff:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to
QuickPath Interconnect Link 1 Performance Monitor (rev 07)


> 3. do you have active I/O when doing your performance tests? if you have
> several virtual machines running depending on what they are doing they
> will crucify your access.

My testing was done with all the VMs stopped!


> 4. are you using any type of CGroups ?

Yes. systemd needs them.
I attach the kernel-config.

> 5. i'm also confused over your LVM config.  please can you send the
> output of "vgs" "pvs" and "lvs -a -o +devices"

/dev/sdc is the third hardware raid array listed above and used as PV
for the VG vg01 ->

# pvs
  PV         VG   Fmt  Attr PSize PFree
  /dev/sdc   vg01 lvm2 a--  1,32t 708,86g

# vgs
  VG   #PV #LV #SN Attr   VSize VFree
  vg01   1   2   0 wz--n- 1,32t 708,86g

# lvs -a -o +devices
  LV              VG   Attr       LSize   Pool Origin Data%  Move Log
Cpy%Sync Convert Devices
  amhold          vg01 -wi-ao---- 400,00g
               /dev/sdc(0)
  winserver_disk0 vg01 -wi-ao---- 244,14g
               /dev/sdc(102400)


> 6. please also send the output of "mount"

# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs
(rw,nosuid,size=8203540k,nr_inodes=2050885,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup
(rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/devices type cgroup
(rw,nosuid,nodev,noexec,relatime,devices)
/dev/sda on / type btrfs (rw,noatime,space_cache)
mqueue on /dev/mqueue type mqueue (rw,relatime)
tmpfs on /tmp type tmpfs (rw)
tmpfs on /var/tmp/portage type tmpfs (rw,relatime)
/dev/mapper/vg01-amhold on /mnt/amhold type xfs
(rw,noatime,attr2,inode64,noquota)
tmpfs on /run/user/0 type tmpfs
(rw,nosuid,nodev,relatime,size=1641324k,mode=700)
/dev/sda on /mnt/btrfs_windows type btrfs (rw,noatime,space_cache)


the last line is meant as a target directory for dumping the content of
the LV winserver_disk0  ... it is a btrfs subvolume mounted with
compression turned OFF.



> 7. do you have atop or iotop that you can use to monitor performance -
> specifically we are looking for disk ios per device and disk latency per
> device.  both before and during you are trying to run your backup.

I will install atop and check ...

> this should give us a better idea of where the problems lay.

Thanks so far!

Stefan