Gentoo Archives: gentoo-doc-cvs

From: Joshua Saddler <nightmorph@××××××××××××.org>
To: gentoo-doc-cvs@l.g.o
Subject: [gentoo-doc-cvs] cvs commit: metadoc.xml multipath.xml
Date: Wed, 10 Sep 2008 23:14:38
Message-Id: E1KdYtO-00024f-KK@stork.gentoo.org
1 nightmorph 08/09/10 23:14:34
2
3 Modified: metadoc.xml
4 Added: multipath.xml
5 Log:
6 Added new multipath guide from bug 233527. thanks to tsunam and all the guys from liquidustech who assembled the thing. and to rane for some xml cleanups. added to the sysadmin_specific documentation category.
7
8 Revision Changes Path
9 1.213 xml/htdocs/doc/en/metadoc.xml
10
11 file : http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/metadoc.xml?rev=1.213&view=markup
12 plain: http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/metadoc.xml?rev=1.213&content-type=text/plain
13 diff : http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/metadoc.xml?r1=1.212&r2=1.213
14
15 Index: metadoc.xml
16 ===================================================================
17 RCS file: /var/cvsroot/gentoo/xml/htdocs/doc/en/metadoc.xml,v
18 retrieving revision 1.212
19 retrieving revision 1.213
20 diff -u -r1.212 -r1.213
21 --- metadoc.xml 2 May 2008 08:04:22 -0000 1.212
22 +++ metadoc.xml 10 Sep 2008 23:14:34 -0000 1.213
23 @@ -1,8 +1,8 @@
24 <?xml version="1.0" encoding="UTF-8"?>
25 <!DOCTYPE metadoc SYSTEM "/dtd/metadoc.dtd">
26 -<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/metadoc.xml,v 1.212 2008/05/02 08:04:22 nightmorph Exp $ -->
27 +<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/metadoc.xml,v 1.213 2008/09/10 23:14:34 nightmorph Exp $ -->
28 <metadoc lang="en">
29 - <version>1.134</version>
30 + <version>1.135</version>
31 <members>
32 <lead>neysx</lead>
33 <member>cam</member>
34 @@ -410,6 +410,7 @@
35 <file id="embedded-handbook">/proj/en/base/embedded/handbook/index.xml</file>
36 <file id="texlive-migration-guide">/proj/en/tex/texlive-migration-guide.xml</file>
37 <file id="openrc-migration">/doc/en/openrc-migration.xml</file>
38 + <file id="multipath">/doc/en/multipath.xml</file>
39 </files>
40 <docs>
41 <doc fileid="name-logo">
42 @@ -885,6 +886,9 @@
43 <memberof>sysadmin_specific</memberof>
44 <memberof>upgrade</memberof>
45 </doc>
46 + <doc fileid="multipath">
47 + <memberof>sysadmin_specific</memberof>
48 + </doc>
49 <doc fileid="new-dev-training">
50 <memberof>project_devrel</memberof>
51 </doc>
52
53
54
55 1.1 xml/htdocs/doc/en/multipath.xml
56
57 file : http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/multipath.xml?rev=1.1&view=markup
58 plain: http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/multipath.xml?rev=1.1&content-type=text/plain
59
60 Index: multipath.xml
61 ===================================================================
62 <?xml version='1.0' encoding="UTF-8"?>
63 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/multipath.xml,v 1.1 2008/09/10 23:14:34 nightmorph Exp $ -->
64 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
65
66 <guide>
67 <title>Multipathing for Gentoo</title>
68
69 <author title="Author">
70 <mail link="tsunam"/>
71 </author>
72 <author title="Author">
73 <mail link="matthew.summers@××××××××××××.com">Matthew Summers</mail>
74 </author>
75 <author title="Author">
76 <mail link="richard.anderson@××××××××××××.com">Richard Anderson</mail>
77 </author>
78 <author title="Author">
79 <mail link="steve.rucker@××××××××××××.com">Steve Rucker</mail>
80 </author>
81 <author title="Editor">
82 <mail link="nightmorph"/>
83 </author>
84
85 <abstract>
86 This document teaches you how to set up multipathing services for data storage.
87 </abstract>
88
89 <!-- The content of this document is licensed under the CC-BY-SA license -->
90 <!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
91 <license/>
92
93 <version>1</version>
94 <date>2008-09-10</date>
95
96 <chapter>
97 <title>Introduction</title>
98 <section>
99 <body>
100
101 <p>
102 Multipathing services, generally deployed in enterprise environments, provide a
103 means for high performance, load-balanced, and fault-tolerant data storage
104 either locally or via a storage area network (SAN). Multipathing facilitates a
105 single storage device to be transparently accessed across one or more paths.
106 For example, if there are two connections from a server Host Bus Adapter (HBA)
107 to two Fibre Channel switches and then to a SAN, when the HBA module loads and
108 scans the bus, it will read four paths to the SAN: the paths from the server HBA
109 to and from each Fibre Channel switch and at the storage device. Taking
110 advantage of this situation, Multipath allows you to make use of each path
111 simultaneously or independently to ensure a constant and reliable connection to
112 the data in storage. Multipath serves as a failover for all connections points
113 in the event of losing one path making critical data always available due to
114 redundancy in the design and implementation.
115 </p>
116
117 <p>
118 In the most basic sense, multipathing is made of two distinct parts:
119 <c>device-mapper</c> and <c>multipath-tools</c>. <b>Device Mapper</b> is the
120 first key element of this application. Administrators are probably familiar with
121 Device Mapper from LVM, EVMS, dm-crypt, or in this case, Multipath. In short,
122 working within the kernel space Device Mapper takes one block device such as
123 <path>/dev/sda</path> (as all SAN based targets will be some type of SCSI
124 device) and maps it to another device.
125 </p>
126
127 <p>
128 On a lower level, Device Mapper creates a virtual block device accepting all of
129 the commands of a regular block device, but passes on the actual data to the
130 real block device. As previously stated, the mapping process is all handled in
131 the kernel space and not in user space.
132 </p>
133
134 <p>
135 <b>Multipath Tools</b> is a set of userspace tools that interacts with the
136 Device Mapper tools and creates structures for device handling, implementing I/O
137 multipathing at the OS level. In a typical SAN environment, you will have
138 multiple paths to the same storage device: a fiber card (or two) on your server
139 that connects to a switch which then connects to the actual storage itself (as
140 in the scenario discussed above). So administrators could possibly see the same
141 device one to four times in such a situation (each card will see the LUN twice,
142 once for each path it has available to it). Thus, a single drive could be
143 recognized as <path>sda</path>, <path>sdb</path>, <path>sdc</path>, and
144 <path>sdd</path>. If you were to mount <path>/dev/sda</path> to
145 <path>/san1</path>, for instance, you would be going over the singular path from
146 one fiber card to a switch and then to a port on the same storage device. If any
147 of those points were to fail, you would lose your storage device suddenly and
148 have to unmount and remount with another device (<path>sdb</path>).
149 </p>
150
151 <p>
152 Consequently, this scenario is not ideal as you are only using one out of the
153 four possible paths. This is where the combination of Multipath tools and Device
154 Mapper are beneficial. As already explained, Device Mapper creates virtual block
155 devices and then passes information to the real block devices.
156 </p>
157
158 </body>
159 </section>
160 </chapter>
161
162 <chapter>
163 <title>Installation and Configuration</title>
164 <section>
165 <title>Installation and Tools</title>
166 <body>
167
168 <p>
169 Tou need to emerge <c>multipath-tools</c> and <c>sg3_utils</c>. On the disk, you
170 want to find the <c>wwid</c>. You can use <c>sq_vpd</c> (provided by
171 <c>sg3_utils</c>) to do this.
172 </p>
173
174 <pre caption="Installing multipath-tools and initial configuration">
175 # <i>emerge multipath-tools sg3_utils</i>
176 <comment>(Replace /dev/DEVICE with your disk to find its wwid)</comment>
177 # <i>/usr/bin/sq_vpd ?page=di /dev/DEVICE</i>
178 </pre>
179
180 <p>
181 Where DEVICE is the sd device, the ID will come back with a <c>0x6</c>. Replace
182 <c>0x</c> with <c>3</c>, and you will have the proper ID that you'll put into
183 the multipath <c>wwid</c> in <path>/etc/multipath.conf</path>. More on this in
184 the next chapter.
185 </p>
186
187 </body>
188 </section>
189 <section>
190 <title>Configuring Gentoo for multipathing</title>
191 <body>
192
193 <p>
194 To configure Gentoo for multipath, your kernel needs the following settings:
195 </p>
196
197 <pre caption="Adding multipath support">
198 Device Drivers --->
199 SCSI device support --->
200 &lt;*&gt; SCSI target support
201 &lt;*&gt; SCSI disk support
202 [*] Probe all LUNs on each SCSI device
203 [*] Multiple devices driver support (RAID and LVM) --->
204 &lt;*&gt; Multipath I/O support
205 &lt;*&gt; Device mapper support
206 &lt;*&gt; Multipath target
207 <comment>(Select your device from the list)</comment>)
208 &lt;*&gt; EMC CX/AX multipath support
209 &lt;*&gt; LSI/Engenio RDAC multipath support
210 &lt;*&gt; HP MSA multipath support
211 </pre>
212
213 <note>
214 <c>scsi_id</c> is done by targets. IDE drives have two spots to which you can
215 connect. An administrator has the ability to set a drive as a master and another
216 drive as a slave or set to autoselect by changing the dip switches. scsi_id is
217 similar. Each drive or Logical Unit Number (LUN) has a unique ID, which ranges
218 from 0 to 254. A device that has ID 0 will be discovered before a device that
219 has, for example, ID 120, because it performs a LIP (a scan of the SCSI bus for
220 devices that respond) that starts from 0 and works its way upwards.
221 </note>
222
223 <p>
224 In the kernel menu config, make sure CONFIG_SCSI_MULTI_LUN=y is set to ensure
225 the SCSI subsystem is able to probe all Logical Unit Numbers (LUNs) (This is
226 recommended as you'll stop scanning after ID 0 if you have a device on an ID of
227 <c>0</c> but not <c>1</c> and then on an ID of <c>2</c>. Simply, you'll get your
228 device for ID <c>0</c> but not <c>2</c>.) or whichever device you need for SCSI,
229 such as a QLogic 2400 card, which is in the SCSI low-level drivers area.
230 </p>
231
232 <p>
233 For a better understanding, consider the following scenarios:
234 </p>
235
236 <p>
237 There are three drives with IDs of 0,1,2. Without the "probe all LUNs" setting,
238 you will see IDs 0,1,2 as sda,sdb,sdc - all devices are seen. If you delete the
239 ID 1 drive. IDs 0,2 will still be seen. It might seem to make sense that you
240 would see sda and sdb now (sdc would move to sdb as there is no device to fill
241 it up). However, if you don't probe all LUNs, it will perform in the following
242 manner:
243 </p>
244
245 <p>
246 Scenario 1: Without "probe all LUNs", the scan will start and ID 0 will be seen.
247 ID 0 will be set to sda and then move to find ID 1. If ID 1 is not detected,
248 scanning will stop and be considered complete having perceived to have scanned
249 all devices even if there is a device on ID 2 or any other subsequent ID. Reboot
250 for scenario two.
251 </p>
252
253 <p>
254 Scenario 2: If you have "probe all LUNs", the scan will start and detect ID 0.
255 This ID will be assigned sda and will continue to detect the next device. If ID
256 1 is not detected, scanning will continue to find more devices. ID 2 will be
257 located and assigned to be sdb. If no devices (IDs) are detected beyond that,
258 scanning will be considered complete.
259 </p>
260
261 <note>
262 Although it seems that it is unfeasible or even unnecessary to have devices
263 spaced many LUNs apart, to account for all options it is necessary to still
264 probe all LUNs. An administrator will encounter many reasons (business or
265 personal) for such a setup. Therefore, the second scenario would be optimal to
266 ensure that all devices are recognized and assigned an ID in the multipath setup
267 process.
268 </note>
269
270 <p>
271 So, once you probe all LUNs, all devices will be recognized and assigned an ID
272 in Multipath.
273 </p>
274
275 </body>
276 </section>
277 </chapter>
278
279 <chapter>
280 <title>Architectural Overview</title>
281 <section>
282 <body>
283
284 <p>
285 As part of Multipath Tools, there are priority groups filled with the devices
286 mentioned earlier. After you have configured <c>multipath-tools</c> and started
287 it with <c>/etc/init.d/multipath start</c>, you can list the groups via
288 <c>multipath -l</c>. The output will look like the following:
289 </p>
290
291 <pre caption="multipath -l output">
292 EVA_SAN (3600508b4001044ee00013000031e0000)
293 [size=300 GB][features="1 queue_if_no_path"][hwhandler="0"]
294 \_ round-robin 0 [active]
295 \_ 0:0:0:1 sda 8:0 [active]
296 \_ round-robin 0 [enabled]
297 \_ 0:0:1:1 sdb 8:16 [active]
298
299 EVA_SAN2 (3600508b4001044ee0001300003880000)
300 [size=300 GB][features="1 queue_if_no_path"][hwhandler="0"]
301 \_ round-robin 0 [active]
302 \_ 0:0:0:2 sdc 8:32 [active]
303 \_ round-robin 0 [enabled]
304 \_ 0:0:1:2 sdd 8:48 [active]
305 </pre>
306
307 <p>
308 By default, it will pick the first priority group (the first top round-robin for
309 the EVA_SAN2, for instance, being <path>sdc</path>). In this instance, due to
310 round robin it will bounce back and forth. But if one path was to fail, it would
311 push all information to the other path and continue. Only if all the devices in
312 a path fail will it actually fail and go to the secondary priority group.
313 </p>
314
315 </body>
316 </section>
317 <section>
318 <title>Typical Configuration</title>
319 <body>
320
321 <p>
322 A typical Multipath configuration looks like the following:
323 </p>
324
325 <pre caption="A typical /etc/multipath.conf file">
326 defaults {
327 udev_dir /dev
328 polling_interval 15
329 selector "round-robin 0"
330 path_grouping_policy group_by_prio
331 failback 5
332 path_checker tur
333 prio_callout "/sbin/mpath_prio_tpc /dev/%n"
334 rr_min_io 100
335 rr_weight uniform
336 no_path_retry queue
337 user_friendly_names yes
338 }
339 blacklist {
340 devnode cciss
341 devnode fd
342 devnode hd
343 devnode md
344 devnode sr
345 devnode scd
346 devnode st
347 devnode ram
348 devnode raw
349 devnode loop
350 devnode sda
351 }
352
353 multipaths {
354 multipath {
355 wwid
356 <comment>(To find your wwid, please use /usr/bin/sq_vpd ?page=di /dev/DEVICE.
357 The address will be a 0x6. Remove the 0x and replace it with 3.)</comment>
358 alias DB_SAN
359 }
360 devices {
361 device {
362 <comment>(White spacing is important on these two items to match the vendor specifications.)</comment>
363 "IBM "
364 "1815 FAStT "
365 }
366 }
367 }
368 </pre>
369
370 <impo>
371 On your devices, it is best to <c>cat</c>
372 <path>/sys/block/sd(device)/device/model</path> and <c>cat</c>
373 <path>/sys/block/device/sd(device)/device/vendor</path>, placing both directly
374 into your devices section in <path>/etc/multipath.conf</path>. You might not
375 always see the white spacing, and it's part of the name in this case. One reason
376 for the device section is that not every vendor's string is in the kernel
377 convention and naming, and the string, as such, is not always detected as
378 required.
379 </impo>
380
381 <p>
382 A typical multipath configuration utilizing an EVA_SAN where the device
383 information is in the kernel information regarding SAN hardware detection would
384 look like:
385 </p>
386
387 <pre caption="EVA_SAN configuration">
388 multipaths {
389 multipath {
390 wwid 3600508b4001044ee00013000031e0000
391 alias EVA_SAN
392 }
393 multipath {
394 wwid 3600508b4001044ee0001300003880000
395 alias EVA_SAN2
396 }
397 }
398 </pre>
399
400 </body>
401 </section>
402 </chapter>
403
404 <chapter>
405 <title>Setting Up Your Own Configuration</title>
406 <section>
407 <body>
408
409 <p>
410 The multipath configuration is fairly simple to accomplish because the only file
411 that needs modification is <path>/etc/multipath.conf</path>.
412 </p>
413
414 <p>
415 To begin, set the <b>polling interview</b> to how often (in seconds) path checks
416 will be performed to ensure that the path is alive and healthy.
417 </p>
418
419 <p>
420 <b>selector</b> will be set at <c>"round-robin 0"</c>.
421 </p>
422
423 <note>
424 This round-robin value is the only selector value that will be used in this
425 configuration.
426 </note>
427
428 <p>
429 <b>prio_callout</b>: This one can be quite important, and there are a number of
430 different priorities for different devices, such as:
431 </p>
432
433 <ul>
434 <li>mpath_prio_alua</li>
435 <li>mpath_prio_emc</li>
436 <li>mpath_prio_hds_modular</li>
437 <li>mpath_prio_netapp</li>
438 <li>mpath_prio_tpc</li>
439 </ul>
440
441 <note>
442 For most people, <c>mpath_prio_tpc</c> will suffice as it's a conservative
443 checker. Other devices like <c>mpath_prio_netapp</c> have special functionality
444 for priority grouping, such as netapps.
445 </note>
446
447 <p>
448 <b>path_grouping_policy</b> has a few different options: failover, multibus,
449 group_by_prio. <c>Failover</c> will only have one disk per priority group.
450 <c>Multibus</c> will put all devices into one priority group.
451 <c>Group_by_prio</c> is done by a "priority value." So routes that have the same
452 priority value will be grouped together, the priority values being determined by
453 the callout.
454 </p>
455
456 <p>
457 <b>no_path_retry</b> is set to <c>queue</c> as most people don't want data to
458 fail to send at all. So, if all paths fail, for instance, the I/Os will queue up
459 until the device returns and then sends everything again. Depending on your
460 transfer, this can cause load issues.
461 </p>
462
463 <p>
464 <b>rr_min_io</b> are the number of I/Os to do per path before switching to the
465 next I/Os in the same group. If <path>sda</path> and <path>sdb</path> were in
466 the same group, rr_min_io would do 100 I/Os to <path>sda</path> then do 100 to
467 <path>sdb</path>, bouncing back and forth. This is a setting to tweak for each
468 instance to maximize performance because the data load and size of
469 transfers/request vary by company. The default in the case is <c>1000</c>, but
470 some may prefer a smaller number in order to switch ports more often, when
471 possible.
472 </p>
473
474 <p>
475 <b>user_friendly_names</b> make it easier to see which device you are working
476 with. For example, if you set user_friendly_names to <c>no</c>, then you'll see
477 WWID instead of EVA_SAN for your device.
478 </p>
479
480 </body>
481 </section>
482 </chapter>
483 </guide>