Loading...
Loading...
Isilon On-Cluster Analysis is reporting output similar to the following: Network Pools - Parallel Upgrade Support FAILCRITICAL: A parallel upgrade runs the risk of making one or more external networks temporarily unavailable. Affected pools: groupnet0.subnet0.pool1, groupnet0.subnet1.pool1, groupnet0.subnet1.pool2
This is by design as the cluster has disjoint external networks where there is a possible risk that all nodes in the impacted network pools reboot simultaneously. In which case, clients cannot access the cluster through the SmartConnect Zone Name (SCZN) associated with that network pool.
Before going through the resolution, it is good to understand how parallel upgrades work. Important definitions: Diskpool: Collection of disks distributed among a subset of cluster nodes. Protection Levels/Policies: All files on the /ifs file system are assigned a protection level. OneFS stripes or mirrors the content of files across nodes and drives of a single diskpool. A file with a protection level of 2d+1n guarantees that the file will still be available if any two drives, or one node in the enclosing diskpool goes down. Neighborhoods: Sets of nodes that wholly contain one or more diskpools. A subset of nodes assigned to a diskpool may overlap with a subset of nodes that are assigned to another neighborhood. There is no overlapping of diskpools between neighborhoods. Diskpools DB API and reservations The Diskpools database API has a reservation feature that allows a client application such as the Upgrade Framework to remove a set of nodes (ie reboot) or drives from the cluster without breaching protection on any diskpool. Disjoint external networks : A cluster has disjoint external networks if there a set of nodes with external interfaces that do not overlap across all diskpools Parallel Upgrades -- Top level overview: A parallel upgrade installs the new operating system on all nodes and then reboots a subset of nodes simultaneously, up to one node in each diskpool. Each node attempts to make a reservation for their turn to reboot until all nodes are upgraded. Node subsets and reservations are based on diskpool and node availability. During a parallel upgrade, node subsets that are not being rebooted remain online and can continue serving clients. Check definition: The cluster has disjoint external networks ( KB example: groupnet0.subnet0.pool1, groupnet0.subnet1.pool1, groupnet0.subnet1.pool2) During the parallel upgrade, there is a possible DU risk on those network pools as all nodes in the affected network pools could reboot at a same time (i.e, they could all take a diskpools reservation to reboot simultaneously). If that occurs, access to the affected network pool's SCZN will not be possible. Notes regarding the resolution: As every cluster has a different network configuration, the solution is not the same for all clusters. To avoid having a possible DU on the disjoint network pool during the parallel upgrade, a network configuration change on the cluster and the external environment will be needed which usually requires several approvals on the customer's end. This KB article will not include a direct resolution to the IOCA check failure due to the above points. However, the general guideline is that at least two nodes’ external member interfaces in the affected network pools exist on the same neighborhood/failure domain to avoid a disjoint network/DU risk to that SmartConnect network pool (i.e, add nodes from the same neighborhood/failure domain to the affected network pool). The KB article provides commands on how to determine the cluster's neighborhood/failure domains and the member nodes in affected network pools. Resolution: Use the below command determine the member nodes in affected network pools ( Note: Replace the "<network-pool 1 ID> | <network-pool 2 ID> | <network-pool 3 ID>" with the affected network pool IDs from the IOCA check details and separate between them using the '|' character ). # isi network pools list -v | egrep -i 'ID: |ifaces' | egrep -A1 '<network-pool 1 ID> | <network-pool 2 ID> | <network-pool 3 ID>' Run the below command to determine the cluster's neighborhoods ( Note : Replace the <IOCA script location> and <New OneFS version> with their correct values ) # perl <IOCA script location> -o <New OneFS version>,parallel -e -r "checkNetworkParallelUpgrade" Cross reference the outputs of the above commands to determine which changes needs to be done to the cluster's network pools. A quick fix would be to add all clusters nodes to the impacted network pools but again, that is usually not easy as the external network environment might not allow this change. After the needed changes are applied, rerun the IOCA check again. If unable to determine the needed changes, open a case with Support for further assistance. Example below to determine the needed changes in a test scenario: 1) Determine member nodes in affected network pools: # isi network pools list -v | grep -i 'ID: |ifaces' | egrep -A1 'groupnet0.subnet0.pool1| groupnet0.subnet1.pool1| groupnet0.subnet1.pool2' ID: groupnet0.subnet0.pool1 Ifaces: 1:10gige-agg-1, 2:10gige-agg-1, 4:10gige-agg-1, 3:10gige-agg-1 ID: groupnet0.subnet1.pool1 Ifaces: 37:10gige-agg-1, 38:10gige-agg-1, 39:10gige-agg-1, 40:10gige-agg-1 ID: groupnet0.subnet1.pool2 Ifaces: 37:10gige-agg-1, 38:10gige-agg-1, 39:10gige-agg-1, 40:10gige-agg-12) Run the IOCA check to determine cluster neighborhoods: # perl IOCA -o 9.2.1.5,parallel -e -r "checkNetworkParallelUpgrade" Isilon On-Cluster Analysis 0.1395 Cluster Name TestCluster Cluster GUID 0050569b6db2ad086861001a2f1dd1d02473 Node Count 52 Current OneFS Version 8.2.2.0 Destination OneFS Version 9.2.1.5 Destination OneFS Version WARN WARN: There is a newer patch release available for OneFS 9.2.1: 9.2.1.9 Network Pools - Parallel Upgrade Support FAIL CRITICAL: A parallel upgrade runs the risk of making one or more external networks temporarily unavailable. Affected pools: groupnet0.subnet0.pool1, groupnet0.subnet1.pool1, groupnet0.subnet1.pool2 ============================== Node Neighborhoods ============================== 1: [ 1, 7, 10, 16, 19, 24, 27, 31, 35, 40, 43, 45, 47 ] 2: [ 2, 6, 9, 15, 18, 23, 26, 29, 33, 38, 41, 46, 48 ] 3: [ 3, 5, 12, 14, 17, 22, 25, 30, 34, 37, 42, 50, 51 ] 4: [ 4, 8, 11, 13, 20, 21, 28, 32, 36, 39, 44, 49, 52 ]3) The possible resolution in this case would be to :a) A quick fix would be to add all clusters nodes to the impacted network pools groupnet0.subnet0.pool1 & groupnet0.subnet1.pool1, groupnet0.subnet1.pool2 b) Add more nodes to affected network pools ( suggested nodes are based on neighborhood command output ) :- Possible resolution to groupnet0.subnet0.pool1 : at least add node 7 to the network pool as node 7 exists in the same neighborhood as nodes 1 - Possible resolution to groupnet0.subnet1.pool1 : at least add node 34 to the network pool as node 34 exists in the same neighborhood as nodes 37 - Possible resolution to groupnet0.subnet1.pool2 : at least add node 33 to the network pool as node 33 exists in the same neighborhood as nodes 384) After applying the network changes, re-run the IOCA check to confirm that there are no issues: # perl IOCA -o 9.2.1.5,parallel -e -r "checkNetworkParallelUpgrade"
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.