Loading...
Loading...
In rare cases, if InfiniBand architecture is in use, adding an unconfigured Isilon node to an InfiniBand switch can cause the new node to become an SM master. When this occurs, logical IDs (LIDs) may be redistributed across the fabric. If a node receives a new LID value, the backend port (ib0 or ib1) will flap, taking on the new LID value. In logs, you can see the interfaces flap at the moment the customer or CE has connected the new node to the fabric. 2021-09-30T10:57:11-04:00 <0.5> isilon-1 /boot/kernel.amd64/kernel: ib0: link state changed to DOWN 2021-09-30T10:57:11-04:00 <0.5> isilon-2 /boot/kernel.amd64/kernel: ib0: link state changed to DOWN 2021-09-30T10:57:11-04:00 <0.5> isilon-3 /boot/kernel.amd64/kernel: ib0: link state changed to DOWN 2021-09-30T10:57:11-04:00 <0.5> isilon-4 /boot/kernel.amd64/kernel: ib0: link state changed to DOWN 2021-09-30T10:57:11-04:00 <0.5> isilon-5 /boot/kernel.amd64/kernel: ib0: link state changed to DOWN 2021-09-30T10:57:11-04:00 <0.5> isilon-6 /boot/kernel.amd64/kernel: ib0: link state changed to DOWN The problem with this behavior is if the customer or CE connects both backend fabrics (int-a and int-b) and then boots the unconfigured node, both int-a and int-b could flap simultaneously causing nodes to split depending on which nodes receive a new LID value. NOTE: We cannot determine which node receives a new LID and which node's InfiniBand interfaces will flap. All node ports can flap or only a subset of nodes can flap, the symptom can vary. This has resulted in a short-term DU as both paths become unavailable for a short time.
Opensm is a process that is run on OneFS nodes running InfiniBand architecture. All nodes run two opensm processes, one for each internal fabric (int-a and int-b). While this runs on every node, only one node is the elected the opensm master. The primary provides update tables to the switch and assigns LIDs to all devices on the fabric. There are two deciding factors on which node is elected the primary: The opensm priority The lowest value hardware address (lladdr) of the InfiniBand (HCA) port By default all nodes have an equal priority level. Thus, the deciding factor in most typical scenarios is the lowest hardware address of the HCA port. If a newly unconfigured node is introduced to the fabric and this node has a lower hardware address value, it may become the new opensm primary. When this happens, there is a potential chance that LIDs may become reassigned in the fabric. The reason this does not occur with nodes existing in the cluster is because they are already assigned LIDs and have no need to redistribute LIDs (they are also considered cluster aware).
There are two options in avoiding this issue in the future. When connecting unconfigured nodes to a fabric, connect each fabric on separate occasions. For example, connect int-a first and wait ten minutes for the master to change and links to flap. After ten minutes, connect int-b to the fabric and allow the opensm primary to change. Select a subset of nodes to become the master by following KB https://www.dell.com/support/kbdoc/en-us/000158790. New nodes will not become the opensm primary as their priority is set to a default value. Those nodes will be the opensm primary as long as they remain in the cluster. The priority does not change when a node is added to the cluster group, it can only be changed manually and will persist through reboots.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.