Re: [gentoo-cluster] openib, no /dev/infiniband

[Brian Budge]

Hi Bryan -

I don't seem to have a 40-ib.rules in any of my /etc/udev/rules.d on any
node.

My /sys/class/infiniband directory contains mthca0, which contains:
> ls -la /sys/class/infiniband/mthca0/
total 0
drwxr-xr-x 3 root root    0 Jan  2 20:54 .
drwxr-xr-x 3 root root    0 Jan  2 20:54 ..
-r--r--r-- 1 root root 4096 Jan  2 21:07 board_id
lrwxrwxrwx 1 root root    0 Jan  3 00:01 device ->
../../../devices/pci0000:20/0000:20:0a.0/0000:21:00.0
-r--r--r-- 1 root root 4096 Jan  2 21:07 fw_ver
-r--r--r-- 1 root root 4096 Jan  2 21:07 hca_type
-r--r--r-- 1 root root 4096 Jan  2 21:07 hw_rev
-rw-r--r-- 1 root root 4096 Jan  2 21:07 node_desc
-r--r--r-- 1 root root 4096 Jan  2 21:07 node_guid
-r--r--r-- 1 root root 4096 Jan  2 21:06 node_type
drwxr-xr-x 3 root root    0 Jan  2 21:07 ports
lrwxrwxrwx 1 root root    0 Jan  3 00:01 subsystem ->
../../../class/infiniband
-r--r--r-- 1 root root 4096 Jan  2 21:07 sys_image_guid
--w------- 1 root root 4096 Jan  2 20:54 uevent

I don't have any ib modules loaded at all on any node.  All of my kernel
modules are built into the kernel:

CONFIG_INFINIBAND=y
CONFIG_INFINIBAND_USER_MAD=y
CONFIG_INFINIBAND_USER_ACCESS=y
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_MTHCA=y
CONFIG_INFINIBAND_MTHCA_DEBUG=y
# CONFIG_INFINIBAND_IPATH is not set
CONFIG_INFINIBAND_AMSO1100=y
# CONFIG_INFINIBAND_AMSO1100_DEBUG is not set
CONFIG_MLX4_INFINIBAND=y
CONFIG_INFINIBAND_IPOIB=y
# CONFIG_INFINIBAND_IPOIB_CM is not set
CONFIG_INFINIBAND_IPOIB_DEBUG=y
# CONFIG_INFINIBAND_IPOIB_DEBUG_DATA is not set
# CONFIG_INFINIBAND_SRP is not set
# CONFIG_INFINIBAND_ISER is not set


Thanks,
  Brian

On Jan 2, 2008 2:11 PM, Bryan Green <bryan.d.green@××××.gov> wrote:

> "Brian Budge" writes:
> >
> > Hi all -
> >
> > I'm new to infiniband and still getting my feet wet.  I am admining a
> very
> > small cluster of 5 nodes, and have recently installed infiniband HCAs.
>  I
> > have the infiniband modules built into the kernel, and I am using the
> > openib-userspace package in the gentoo-science overlay.
> >
> > The strange thing with my situation is that I have infiniband working
> with
> > openmpi on 4 of my 5 nodes, but the 5th one is a mystery.
> >
> > All 4 working nodes have a /dev/infiniband directory that look roughly
> like
> > this:
> >
> > crw-rw---- 1 root root 231,  64 Dec 31 09:13 issm0
> > crw-rw-rw- 1 root root 231, 224 Dec 31 09:13 ucm0
> > crw-rw---- 1 root root 231,   0 Dec 31 09:13 umad0
> > crw-rw-rw- 1 root root 231, 192 Dec 31 09:13 uverbs0
> >
> >
> > But the 5th node doesn't, which could indicate the problem (it isn't
> > completely the problem, as I tried making those nodes myself to match,
> but
> > it doesn't help).  I'm just not sure what the difference is, because I
> > installed them all the same way, they all have the same hardware, and
> they
> > are all running the same kernel.
>
> The '/dev/infiniband' subdir is created by the udev rules in
> '/etc/udev/rules.d/40-ib.rules'
>
> Does the '/sys/class/infiniband' directory exist?
> If so, what does it contain?  What loaded modules with an 'ib_' prefix
> does
> lsmod report?
>
> -bryan
>
> --
> gentoo-cluster@g.o mailing list
>
>