1 |
nightmorph 08/09/10 23:14:34 |
2 |
|
3 |
Modified: metadoc.xml |
4 |
Added: multipath.xml |
5 |
Log: |
6 |
Added new multipath guide from bug 233527. thanks to tsunam and all the guys from liquidustech who assembled the thing. and to rane for some xml cleanups. added to the sysadmin_specific documentation category. |
7 |
|
8 |
Revision Changes Path |
9 |
1.213 xml/htdocs/doc/en/metadoc.xml |
10 |
|
11 |
file : http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/metadoc.xml?rev=1.213&view=markup |
12 |
plain: http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/metadoc.xml?rev=1.213&content-type=text/plain |
13 |
diff : http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/metadoc.xml?r1=1.212&r2=1.213 |
14 |
|
15 |
Index: metadoc.xml |
16 |
=================================================================== |
17 |
RCS file: /var/cvsroot/gentoo/xml/htdocs/doc/en/metadoc.xml,v |
18 |
retrieving revision 1.212 |
19 |
retrieving revision 1.213 |
20 |
diff -u -r1.212 -r1.213 |
21 |
--- metadoc.xml 2 May 2008 08:04:22 -0000 1.212 |
22 |
+++ metadoc.xml 10 Sep 2008 23:14:34 -0000 1.213 |
23 |
@@ -1,8 +1,8 @@ |
24 |
<?xml version="1.0" encoding="UTF-8"?> |
25 |
<!DOCTYPE metadoc SYSTEM "/dtd/metadoc.dtd"> |
26 |
-<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/metadoc.xml,v 1.212 2008/05/02 08:04:22 nightmorph Exp $ --> |
27 |
+<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/metadoc.xml,v 1.213 2008/09/10 23:14:34 nightmorph Exp $ --> |
28 |
<metadoc lang="en"> |
29 |
- <version>1.134</version> |
30 |
+ <version>1.135</version> |
31 |
<members> |
32 |
<lead>neysx</lead> |
33 |
<member>cam</member> |
34 |
@@ -410,6 +410,7 @@ |
35 |
<file id="embedded-handbook">/proj/en/base/embedded/handbook/index.xml</file> |
36 |
<file id="texlive-migration-guide">/proj/en/tex/texlive-migration-guide.xml</file> |
37 |
<file id="openrc-migration">/doc/en/openrc-migration.xml</file> |
38 |
+ <file id="multipath">/doc/en/multipath.xml</file> |
39 |
</files> |
40 |
<docs> |
41 |
<doc fileid="name-logo"> |
42 |
@@ -885,6 +886,9 @@ |
43 |
<memberof>sysadmin_specific</memberof> |
44 |
<memberof>upgrade</memberof> |
45 |
</doc> |
46 |
+ <doc fileid="multipath"> |
47 |
+ <memberof>sysadmin_specific</memberof> |
48 |
+ </doc> |
49 |
<doc fileid="new-dev-training"> |
50 |
<memberof>project_devrel</memberof> |
51 |
</doc> |
52 |
|
53 |
|
54 |
|
55 |
1.1 xml/htdocs/doc/en/multipath.xml |
56 |
|
57 |
file : http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/multipath.xml?rev=1.1&view=markup |
58 |
plain: http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/doc/en/multipath.xml?rev=1.1&content-type=text/plain |
59 |
|
60 |
Index: multipath.xml |
61 |
=================================================================== |
62 |
<?xml version='1.0' encoding="UTF-8"?> |
63 |
<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/multipath.xml,v 1.1 2008/09/10 23:14:34 nightmorph Exp $ --> |
64 |
<!DOCTYPE guide SYSTEM "/dtd/guide.dtd"> |
65 |
|
66 |
<guide> |
67 |
<title>Multipathing for Gentoo</title> |
68 |
|
69 |
<author title="Author"> |
70 |
<mail link="tsunam"/> |
71 |
</author> |
72 |
<author title="Author"> |
73 |
<mail link="matthew.summers@××××××××××××.com">Matthew Summers</mail> |
74 |
</author> |
75 |
<author title="Author"> |
76 |
<mail link="richard.anderson@××××××××××××.com">Richard Anderson</mail> |
77 |
</author> |
78 |
<author title="Author"> |
79 |
<mail link="steve.rucker@××××××××××××.com">Steve Rucker</mail> |
80 |
</author> |
81 |
<author title="Editor"> |
82 |
<mail link="nightmorph"/> |
83 |
</author> |
84 |
|
85 |
<abstract> |
86 |
This document teaches you how to set up multipathing services for data storage. |
87 |
</abstract> |
88 |
|
89 |
<!-- The content of this document is licensed under the CC-BY-SA license --> |
90 |
<!-- See http://creativecommons.org/licenses/by-sa/2.5 --> |
91 |
<license/> |
92 |
|
93 |
<version>1</version> |
94 |
<date>2008-09-10</date> |
95 |
|
96 |
<chapter> |
97 |
<title>Introduction</title> |
98 |
<section> |
99 |
<body> |
100 |
|
101 |
<p> |
102 |
Multipathing services, generally deployed in enterprise environments, provide a |
103 |
means for high performance, load-balanced, and fault-tolerant data storage |
104 |
either locally or via a storage area network (SAN). Multipathing facilitates a |
105 |
single storage device to be transparently accessed across one or more paths. |
106 |
For example, if there are two connections from a server Host Bus Adapter (HBA) |
107 |
to two Fibre Channel switches and then to a SAN, when the HBA module loads and |
108 |
scans the bus, it will read four paths to the SAN: the paths from the server HBA |
109 |
to and from each Fibre Channel switch and at the storage device. Taking |
110 |
advantage of this situation, Multipath allows you to make use of each path |
111 |
simultaneously or independently to ensure a constant and reliable connection to |
112 |
the data in storage. Multipath serves as a failover for all connections points |
113 |
in the event of losing one path making critical data always available due to |
114 |
redundancy in the design and implementation. |
115 |
</p> |
116 |
|
117 |
<p> |
118 |
In the most basic sense, multipathing is made of two distinct parts: |
119 |
<c>device-mapper</c> and <c>multipath-tools</c>. <b>Device Mapper</b> is the |
120 |
first key element of this application. Administrators are probably familiar with |
121 |
Device Mapper from LVM, EVMS, dm-crypt, or in this case, Multipath. In short, |
122 |
working within the kernel space Device Mapper takes one block device such as |
123 |
<path>/dev/sda</path> (as all SAN based targets will be some type of SCSI |
124 |
device) and maps it to another device. |
125 |
</p> |
126 |
|
127 |
<p> |
128 |
On a lower level, Device Mapper creates a virtual block device accepting all of |
129 |
the commands of a regular block device, but passes on the actual data to the |
130 |
real block device. As previously stated, the mapping process is all handled in |
131 |
the kernel space and not in user space. |
132 |
</p> |
133 |
|
134 |
<p> |
135 |
<b>Multipath Tools</b> is a set of userspace tools that interacts with the |
136 |
Device Mapper tools and creates structures for device handling, implementing I/O |
137 |
multipathing at the OS level. In a typical SAN environment, you will have |
138 |
multiple paths to the same storage device: a fiber card (or two) on your server |
139 |
that connects to a switch which then connects to the actual storage itself (as |
140 |
in the scenario discussed above). So administrators could possibly see the same |
141 |
device one to four times in such a situation (each card will see the LUN twice, |
142 |
once for each path it has available to it). Thus, a single drive could be |
143 |
recognized as <path>sda</path>, <path>sdb</path>, <path>sdc</path>, and |
144 |
<path>sdd</path>. If you were to mount <path>/dev/sda</path> to |
145 |
<path>/san1</path>, for instance, you would be going over the singular path from |
146 |
one fiber card to a switch and then to a port on the same storage device. If any |
147 |
of those points were to fail, you would lose your storage device suddenly and |
148 |
have to unmount and remount with another device (<path>sdb</path>). |
149 |
</p> |
150 |
|
151 |
<p> |
152 |
Consequently, this scenario is not ideal as you are only using one out of the |
153 |
four possible paths. This is where the combination of Multipath tools and Device |
154 |
Mapper are beneficial. As already explained, Device Mapper creates virtual block |
155 |
devices and then passes information to the real block devices. |
156 |
</p> |
157 |
|
158 |
</body> |
159 |
</section> |
160 |
</chapter> |
161 |
|
162 |
<chapter> |
163 |
<title>Installation and Configuration</title> |
164 |
<section> |
165 |
<title>Installation and Tools</title> |
166 |
<body> |
167 |
|
168 |
<p> |
169 |
Tou need to emerge <c>multipath-tools</c> and <c>sg3_utils</c>. On the disk, you |
170 |
want to find the <c>wwid</c>. You can use <c>sq_vpd</c> (provided by |
171 |
<c>sg3_utils</c>) to do this. |
172 |
</p> |
173 |
|
174 |
<pre caption="Installing multipath-tools and initial configuration"> |
175 |
# <i>emerge multipath-tools sg3_utils</i> |
176 |
<comment>(Replace /dev/DEVICE with your disk to find its wwid)</comment> |
177 |
# <i>/usr/bin/sq_vpd ?page=di /dev/DEVICE</i> |
178 |
</pre> |
179 |
|
180 |
<p> |
181 |
Where DEVICE is the sd device, the ID will come back with a <c>0x6</c>. Replace |
182 |
<c>0x</c> with <c>3</c>, and you will have the proper ID that you'll put into |
183 |
the multipath <c>wwid</c> in <path>/etc/multipath.conf</path>. More on this in |
184 |
the next chapter. |
185 |
</p> |
186 |
|
187 |
</body> |
188 |
</section> |
189 |
<section> |
190 |
<title>Configuring Gentoo for multipathing</title> |
191 |
<body> |
192 |
|
193 |
<p> |
194 |
To configure Gentoo for multipath, your kernel needs the following settings: |
195 |
</p> |
196 |
|
197 |
<pre caption="Adding multipath support"> |
198 |
Device Drivers ---> |
199 |
SCSI device support ---> |
200 |
<*> SCSI target support |
201 |
<*> SCSI disk support |
202 |
[*] Probe all LUNs on each SCSI device |
203 |
[*] Multiple devices driver support (RAID and LVM) ---> |
204 |
<*> Multipath I/O support |
205 |
<*> Device mapper support |
206 |
<*> Multipath target |
207 |
<comment>(Select your device from the list)</comment>) |
208 |
<*> EMC CX/AX multipath support |
209 |
<*> LSI/Engenio RDAC multipath support |
210 |
<*> HP MSA multipath support |
211 |
</pre> |
212 |
|
213 |
<note> |
214 |
<c>scsi_id</c> is done by targets. IDE drives have two spots to which you can |
215 |
connect. An administrator has the ability to set a drive as a master and another |
216 |
drive as a slave or set to autoselect by changing the dip switches. scsi_id is |
217 |
similar. Each drive or Logical Unit Number (LUN) has a unique ID, which ranges |
218 |
from 0 to 254. A device that has ID 0 will be discovered before a device that |
219 |
has, for example, ID 120, because it performs a LIP (a scan of the SCSI bus for |
220 |
devices that respond) that starts from 0 and works its way upwards. |
221 |
</note> |
222 |
|
223 |
<p> |
224 |
In the kernel menu config, make sure CONFIG_SCSI_MULTI_LUN=y is set to ensure |
225 |
the SCSI subsystem is able to probe all Logical Unit Numbers (LUNs) (This is |
226 |
recommended as you'll stop scanning after ID 0 if you have a device on an ID of |
227 |
<c>0</c> but not <c>1</c> and then on an ID of <c>2</c>. Simply, you'll get your |
228 |
device for ID <c>0</c> but not <c>2</c>.) or whichever device you need for SCSI, |
229 |
such as a QLogic 2400 card, which is in the SCSI low-level drivers area. |
230 |
</p> |
231 |
|
232 |
<p> |
233 |
For a better understanding, consider the following scenarios: |
234 |
</p> |
235 |
|
236 |
<p> |
237 |
There are three drives with IDs of 0,1,2. Without the "probe all LUNs" setting, |
238 |
you will see IDs 0,1,2 as sda,sdb,sdc - all devices are seen. If you delete the |
239 |
ID 1 drive. IDs 0,2 will still be seen. It might seem to make sense that you |
240 |
would see sda and sdb now (sdc would move to sdb as there is no device to fill |
241 |
it up). However, if you don't probe all LUNs, it will perform in the following |
242 |
manner: |
243 |
</p> |
244 |
|
245 |
<p> |
246 |
Scenario 1: Without "probe all LUNs", the scan will start and ID 0 will be seen. |
247 |
ID 0 will be set to sda and then move to find ID 1. If ID 1 is not detected, |
248 |
scanning will stop and be considered complete having perceived to have scanned |
249 |
all devices even if there is a device on ID 2 or any other subsequent ID. Reboot |
250 |
for scenario two. |
251 |
</p> |
252 |
|
253 |
<p> |
254 |
Scenario 2: If you have "probe all LUNs", the scan will start and detect ID 0. |
255 |
This ID will be assigned sda and will continue to detect the next device. If ID |
256 |
1 is not detected, scanning will continue to find more devices. ID 2 will be |
257 |
located and assigned to be sdb. If no devices (IDs) are detected beyond that, |
258 |
scanning will be considered complete. |
259 |
</p> |
260 |
|
261 |
<note> |
262 |
Although it seems that it is unfeasible or even unnecessary to have devices |
263 |
spaced many LUNs apart, to account for all options it is necessary to still |
264 |
probe all LUNs. An administrator will encounter many reasons (business or |
265 |
personal) for such a setup. Therefore, the second scenario would be optimal to |
266 |
ensure that all devices are recognized and assigned an ID in the multipath setup |
267 |
process. |
268 |
</note> |
269 |
|
270 |
<p> |
271 |
So, once you probe all LUNs, all devices will be recognized and assigned an ID |
272 |
in Multipath. |
273 |
</p> |
274 |
|
275 |
</body> |
276 |
</section> |
277 |
</chapter> |
278 |
|
279 |
<chapter> |
280 |
<title>Architectural Overview</title> |
281 |
<section> |
282 |
<body> |
283 |
|
284 |
<p> |
285 |
As part of Multipath Tools, there are priority groups filled with the devices |
286 |
mentioned earlier. After you have configured <c>multipath-tools</c> and started |
287 |
it with <c>/etc/init.d/multipath start</c>, you can list the groups via |
288 |
<c>multipath -l</c>. The output will look like the following: |
289 |
</p> |
290 |
|
291 |
<pre caption="multipath -l output"> |
292 |
EVA_SAN (3600508b4001044ee00013000031e0000) |
293 |
[size=300 GB][features="1 queue_if_no_path"][hwhandler="0"] |
294 |
\_ round-robin 0 [active] |
295 |
\_ 0:0:0:1 sda 8:0 [active] |
296 |
\_ round-robin 0 [enabled] |
297 |
\_ 0:0:1:1 sdb 8:16 [active] |
298 |
|
299 |
EVA_SAN2 (3600508b4001044ee0001300003880000) |
300 |
[size=300 GB][features="1 queue_if_no_path"][hwhandler="0"] |
301 |
\_ round-robin 0 [active] |
302 |
\_ 0:0:0:2 sdc 8:32 [active] |
303 |
\_ round-robin 0 [enabled] |
304 |
\_ 0:0:1:2 sdd 8:48 [active] |
305 |
</pre> |
306 |
|
307 |
<p> |
308 |
By default, it will pick the first priority group (the first top round-robin for |
309 |
the EVA_SAN2, for instance, being <path>sdc</path>). In this instance, due to |
310 |
round robin it will bounce back and forth. But if one path was to fail, it would |
311 |
push all information to the other path and continue. Only if all the devices in |
312 |
a path fail will it actually fail and go to the secondary priority group. |
313 |
</p> |
314 |
|
315 |
</body> |
316 |
</section> |
317 |
<section> |
318 |
<title>Typical Configuration</title> |
319 |
<body> |
320 |
|
321 |
<p> |
322 |
A typical Multipath configuration looks like the following: |
323 |
</p> |
324 |
|
325 |
<pre caption="A typical /etc/multipath.conf file"> |
326 |
defaults { |
327 |
udev_dir /dev |
328 |
polling_interval 15 |
329 |
selector "round-robin 0" |
330 |
path_grouping_policy group_by_prio |
331 |
failback 5 |
332 |
path_checker tur |
333 |
prio_callout "/sbin/mpath_prio_tpc /dev/%n" |
334 |
rr_min_io 100 |
335 |
rr_weight uniform |
336 |
no_path_retry queue |
337 |
user_friendly_names yes |
338 |
} |
339 |
blacklist { |
340 |
devnode cciss |
341 |
devnode fd |
342 |
devnode hd |
343 |
devnode md |
344 |
devnode sr |
345 |
devnode scd |
346 |
devnode st |
347 |
devnode ram |
348 |
devnode raw |
349 |
devnode loop |
350 |
devnode sda |
351 |
} |
352 |
|
353 |
multipaths { |
354 |
multipath { |
355 |
wwid |
356 |
<comment>(To find your wwid, please use /usr/bin/sq_vpd ?page=di /dev/DEVICE. |
357 |
The address will be a 0x6. Remove the 0x and replace it with 3.)</comment> |
358 |
alias DB_SAN |
359 |
} |
360 |
devices { |
361 |
device { |
362 |
<comment>(White spacing is important on these two items to match the vendor specifications.)</comment> |
363 |
"IBM " |
364 |
"1815 FAStT " |
365 |
} |
366 |
} |
367 |
} |
368 |
</pre> |
369 |
|
370 |
<impo> |
371 |
On your devices, it is best to <c>cat</c> |
372 |
<path>/sys/block/sd(device)/device/model</path> and <c>cat</c> |
373 |
<path>/sys/block/device/sd(device)/device/vendor</path>, placing both directly |
374 |
into your devices section in <path>/etc/multipath.conf</path>. You might not |
375 |
always see the white spacing, and it's part of the name in this case. One reason |
376 |
for the device section is that not every vendor's string is in the kernel |
377 |
convention and naming, and the string, as such, is not always detected as |
378 |
required. |
379 |
</impo> |
380 |
|
381 |
<p> |
382 |
A typical multipath configuration utilizing an EVA_SAN where the device |
383 |
information is in the kernel information regarding SAN hardware detection would |
384 |
look like: |
385 |
</p> |
386 |
|
387 |
<pre caption="EVA_SAN configuration"> |
388 |
multipaths { |
389 |
multipath { |
390 |
wwid 3600508b4001044ee00013000031e0000 |
391 |
alias EVA_SAN |
392 |
} |
393 |
multipath { |
394 |
wwid 3600508b4001044ee0001300003880000 |
395 |
alias EVA_SAN2 |
396 |
} |
397 |
} |
398 |
</pre> |
399 |
|
400 |
</body> |
401 |
</section> |
402 |
</chapter> |
403 |
|
404 |
<chapter> |
405 |
<title>Setting Up Your Own Configuration</title> |
406 |
<section> |
407 |
<body> |
408 |
|
409 |
<p> |
410 |
The multipath configuration is fairly simple to accomplish because the only file |
411 |
that needs modification is <path>/etc/multipath.conf</path>. |
412 |
</p> |
413 |
|
414 |
<p> |
415 |
To begin, set the <b>polling interview</b> to how often (in seconds) path checks |
416 |
will be performed to ensure that the path is alive and healthy. |
417 |
</p> |
418 |
|
419 |
<p> |
420 |
<b>selector</b> will be set at <c>"round-robin 0"</c>. |
421 |
</p> |
422 |
|
423 |
<note> |
424 |
This round-robin value is the only selector value that will be used in this |
425 |
configuration. |
426 |
</note> |
427 |
|
428 |
<p> |
429 |
<b>prio_callout</b>: This one can be quite important, and there are a number of |
430 |
different priorities for different devices, such as: |
431 |
</p> |
432 |
|
433 |
<ul> |
434 |
<li>mpath_prio_alua</li> |
435 |
<li>mpath_prio_emc</li> |
436 |
<li>mpath_prio_hds_modular</li> |
437 |
<li>mpath_prio_netapp</li> |
438 |
<li>mpath_prio_tpc</li> |
439 |
</ul> |
440 |
|
441 |
<note> |
442 |
For most people, <c>mpath_prio_tpc</c> will suffice as it's a conservative |
443 |
checker. Other devices like <c>mpath_prio_netapp</c> have special functionality |
444 |
for priority grouping, such as netapps. |
445 |
</note> |
446 |
|
447 |
<p> |
448 |
<b>path_grouping_policy</b> has a few different options: failover, multibus, |
449 |
group_by_prio. <c>Failover</c> will only have one disk per priority group. |
450 |
<c>Multibus</c> will put all devices into one priority group. |
451 |
<c>Group_by_prio</c> is done by a "priority value." So routes that have the same |
452 |
priority value will be grouped together, the priority values being determined by |
453 |
the callout. |
454 |
</p> |
455 |
|
456 |
<p> |
457 |
<b>no_path_retry</b> is set to <c>queue</c> as most people don't want data to |
458 |
fail to send at all. So, if all paths fail, for instance, the I/Os will queue up |
459 |
until the device returns and then sends everything again. Depending on your |
460 |
transfer, this can cause load issues. |
461 |
</p> |
462 |
|
463 |
<p> |
464 |
<b>rr_min_io</b> are the number of I/Os to do per path before switching to the |
465 |
next I/Os in the same group. If <path>sda</path> and <path>sdb</path> were in |
466 |
the same group, rr_min_io would do 100 I/Os to <path>sda</path> then do 100 to |
467 |
<path>sdb</path>, bouncing back and forth. This is a setting to tweak for each |
468 |
instance to maximize performance because the data load and size of |
469 |
transfers/request vary by company. The default in the case is <c>1000</c>, but |
470 |
some may prefer a smaller number in order to switch ports more often, when |
471 |
possible. |
472 |
</p> |
473 |
|
474 |
<p> |
475 |
<b>user_friendly_names</b> make it easier to see which device you are working |
476 |
with. For example, if you set user_friendly_names to <c>no</c>, then you'll see |
477 |
WWID instead of EVA_SAN for your device. |
478 |
</p> |
479 |
|
480 |
</body> |
481 |
</section> |
482 |
</chapter> |
483 |
</guide> |