Issue with using ASMLIB with Multipath Disks

Issue with using ASMLIB with Multipath Disks

During Grid infrastructure installation root.sh ran successfully in first root and it errored in the second node with the following message as it could not recognize the voting disk from second node.

Error

====

[cssd(17275)]CRS-1709:Lease acquisition failed for node vsnlpmdb02 because no voting file has been configured; Details at (:CSSNM00031:) in /orabin/app/11.2.0/grid/log/vsnlpmdb02/cssd/ocssd.log

Explanation of fix

=============

ASM cannot handle seeing the same disk twice. If it does, it will cause an error. A single disk can appear three times in a multipath configuration:

    The first path to the disk

    The second path to the disk

    The multipath access point

Here's an example: say a system has one local disk, /dev/sda, and one disk attached via external storage. The machine has two connections, or paths, to that external storage. The Linux SCSI driver will see both paths. They will appear as /dev/sdb and /dev/sdc. The system may access sdb or sdc, the access ends up in the same place.

If multipathing is enabled, there will be a multipath disk, for example /dev/multipatha, that can access both paths. That is, any I/O to multipatha can use either path. If a system were using the sdb path, and that cable is unplugged, the system will get an error. But the multipath disk will know to switch to the sdc path.

Most system software will be unaware of the multipath configuration. It can use any of the paths, sdb, sdc, or multipatha, and it won't know the difference. ASMLib is the same way. The default configuration does not care which path it uses.

ASMLib will choose only one of the paths, because ASM cannot handle seeing the same disk twice. This solves the first issue. ASM only sees one path, and it is happy. Here is the second issue: which path does ASM see? In its default configuration, ASMLib will choose the first path it finds. This is the first path as reported by Linux. Depending on your storage driver, it could be the multipath, or it could be one of the single paths.

The system administrator wants ASMLib to always use the multipath disk. What's the point of having it if Oracle is not using it? There is no way, however, for ASMLib to know what a multipath looks like. It must be told via its configuration in the following way.

Solution

======

Scan Multipath Disks First

====================

The system administrator configures ASMLib to scan the multipath disks first. In the ASMLib configuration file the ORACLEASM_SCANORDER variable is edited to look like so:

      ORACLEASM_SCANORDER="<Multipath disk>"

During a scan, ASMLib first tries all disks that start with "multipath". The multipath device /dev/multipatha certainly matches. It is scanned first. Next, ASMLib tries all disks that start with "sd". This is all the SCSI disks.

The local disk /dev/sda will be scanned, but it is not an ASM disk. The single path disks /dev/sdb and /dev/sdc are also scanned. They are ASM disks, but ASMLib will see that it already has a path to that disk. It will ignore them. Finally, ASMLib will scan any other disks that did not match either prefix.

Excluding Single Path Disks

======================

The system administrator configures ASMLib to ignore the single path disks. In the ASMLib configuration, he edits the ORACLEASM_SCANEXCLUDE variable to look like so:

      ORACLEASM_SCANEXCLUDE="<Single Path disk>"

So we had changed the configuration file /etc/sysconfig/oracleasm-_dev_oracleasm in the below manner and created the candidate disks and we were able to run root.sh successfully on both the nodes. Here dm is the multipath disk prefix and sd is the single disk prefix.

                            

ORACLEASM_SCANORDER="dm"

ORACLEASM_SCANEXCLUDE="sd"