OPERATIONAL DEFECT DATABASE
...
...
Document VersionRelease DateDetails306/10/2020Updated information about PHSS_44778 in Description and updated Resolution.211/26/2018Updated formatting in the Description section.111/21/2018Original Document Release.Use of restricted words in the HPE Serviceguard cluster may result in the cluster failing. The use of restricted keywords will result in cmcld causing a core dump with SIGSEGV. Such use refers to having some string-based attribute value in the cluster database (CDB) which starts with one of the following keywords.acpsaddressapplied_configuration_versioncapacitiesclustercmconditionconfigcriteriadependenciesdetachednodesenvironmentgeneric_resourcesgmsgroupsiplanmodule_versionsnetworksnodespeerspkgspolling_targetsprefixesquorum_serversresourcesscript_log_filesecurityservicesstorage_groupssubnetweightsweight_defaultsCommon examples of string-based attribute values in the CDB include the cluster name, a nodename and a package name. The full list of impacted CDB attributes are as follows:Cluster attributes:"CLUSTER_NAME","NODE_NAME","USER_NAME","USER_HOST","HPVM_VOLUME_GROUP","VOLUME_GROUP","OPS_VOLUME_GROUP","GMS_LOCATON","CAPACITY_NAME","WEIGHT_NAME",Package attributes: (Names in lower cases for modular packages and upper cases for legacy ones)"package_name" or "PACKAGE_NAME""node_name" or "NODE_NAME""script_log_file" or "SCRIPT_LOG_FILE""service_name" or "SERVICE_NAME""resource_name" or "RESOURCE_NAME""dependency_name" or "DEPENDENCY_NAME""weight_name" or "WEIGHT_NAME""generic_resource_name" or "GENERIC_RESOURCE_NAME""ip_subnet_node" or "IP_SUBNET_NODE""service_cmd" or "SERVICE_CMD""generic_resource_name" or "GENERIC_RESOURCE_NAME""vg" or "VG""vxvm_dg" or "VXVM_DG""user_name" or "USER_NAME""user_host" or "USER_HOST"For clusters with one or more of the above attributes having a value starting with a restricted keyword, unexpected behaviors may occur when trying to check and/or update the CDB through commands like cmcheckconf, cmapplyconf and cmdeleteconf.A patch fix was previously released, as described in the patch text of PHSS_44549 described below.Defect: QXCR1001473542When the cluster is up, cmapplyconf of a packageconfiguration file having package_name starting withthe word "config" fails because cmcld is unable torecognize the package while scanning the CDB. A wordstarting with "config" is treated by the code as areserved word. When the package name "config_pkg" isreceived by cmcld, it assumes that this is a reservedkey-value pair and this assumption finally leads tothe failure.Resolution:With this fix, cmcheckconf / cmapplyconfwill report that a package name cannot begin withthe word "config". The patch containing this fix willnot install on a system which contains a package withthe package name starting with the word "config" untilNote that the above fix in PHSS_44549 attempted to address the issue only with keyword "config".A later fix introduced in Serviceguard 11.20 patch PHSS_44778 attempted to address ALL identified keywords.Another known unexpected behavior involves an illegal package name starting with a restricted keyword. Running cmapplyconf to apply the package configuration, cmapplyconf should fail with an error. But when the cluster is stopped, cmapplyconf does not complain and allows the illegal package name to be applied.And if such a package is later removed with cmdeleteconf, the removal of the package will leave stale references in the CDB, which can result in cmcld aborting on next cluster reformation. In one reported case, cmcld aborted with SIGSEGV and a stack trace like the following when cluster reformation took place as a result of a cmhaltnode operation.pe_setup_pkg_result () at pe/pe_eval_placement.c:2380pe_setup_pkg_result_all () at pe/pe_eval_placement.c:2332pe_eval_placement () at pe/pe_eval_placement.c:169pm_place_all () at pkg/pkg_pe_support.c:1673pm_sync_done_event () at pkg/pkg_reconfig.c:739pm_sync_event () at pkg/pkg_reconfig.c:320pm_event_handler () at pkg/pkg.c:820deliver_events_on_list_of_queues () at utils/cl_event.c:402cl_event_loop () at utils/cl_event.c:520Given the possibility of cmcld abort due to stale references in the CDB, a node can go down (a Serviceguard TOC) or even the entire cluster can go down at cluster reformation.It is therefore important to get rid of any attribute value in the CDB that starts with a restricted keyword. The advisory resolution given below is to detail the steps for the cleanup.
Any HPE system running HP-UX 11i v3 Serviceguard version A.11.20.00 (or later).
The original resolution proposed installing PHSS_44778 as the final fix addressing QXCR1001667763 as shown below.9. Symptom: QXCR1001667763On one node, cmcld core-dump'ed with SIGSEGV and laterresulted in a SG-TOC in cluster reformation afterdeleting a package which name starts with one of therestricted keywords, which is used by Serviceguardinternally for CDB.core.ORG: ELF-32 core file - IA64 from 'cmcld'- received SIGSEGV$(gdb) bt#0 0x4234170:0 in pe_setup_pkg_result () atpe/pe_eval_placement.c:2380#1 0x422bc20:0 in pe_setup_pkg_result_all () atpe/pe_eval_placement.c:2332#2 0x422ac00:0 in pe_eval_placement () atpe/pe_eval_placement.c:169#3 0x42d6230:0 in pm_place_all () atpkg/pkg_pe_support.c:1673#4 0x42d9e20:0 in pm_sync_done_event () atpkg/pkg_reconfig.c:739#5 0x42d8740:0 in pm_sync_event () atpkg/pkg_reconfig.c:320#6 0x42497b0:0 in pm_event_handler () atpkg/pkg.c:820#7 0x439b3b0:0 in deliver_events_on_list_of_queues ()at utils/cl_event.c:402#8 0x439b910:0 in cl_event_loop () atutils/cl_event.c:5209. Defect: QXCR1001667763Serviceguard uses some restricted keywords in clusterdatabase (CDB) for internal purpose. If an attributevalue starts with one of these restricted keywords,Serviceguard incorrectly interprets the CDB valueas attribute. Due to this, sometimes online validationor configuration of cluster or package fails withunknown reasons. There will also be node failuresand cluster failures in next cluster databaseconfiguration change due to caching of theserestricted keywords in cmcld memory.Resolution:cmcheckconf and cmapplyconf have been enhanced withadditional checks to disallow package or clusterconfiguration using Serviceguard restricted keywordswhich are used internally by the cluster database.Unfortunately, the fix in PHSS_44778 exhibited a regression causing an installation failure. The following patch warning was posted.20/03/02 - This Non-Critical Warning has been issued by HPE.PHSS_44778 fails to install on Serviceguard nodes where the accesscontrol parameter "user_host" or "USER_HOST" has been set to "cluster_member_node".Users should take one of the following steps:1) Do not install the patch. Install the replacement patch to be released in April 2020 time frame.2) Set "user_host" or "USER_HOST" to "CLUSTER_MEMBER_NODE" or "Cluster_member_node" to install patch PHSS_44778 anyway.PHSS_44821 was released in April 2020 to address the above regression.2. Defect: QXCR1001727597PHSS_44778, in addressing QXCR1001667763, adds checkingto disallow attribute values starting with keywordsreserved for CDB's internal use. But some predefinedvalues, like 'cluster_member_node' for the 'user_host'attribute should be allowed. The patch installationfails because the checking imposed by the patch does notcater for legitimate predefined values that happen tostart with a restricted keyword.Resolution:This fix will allow the values for attributes whichhave predefined values even those they arereserve strings.At the same time, PHSS_44821 now also bans attribute values starting with restricted keywords in uppercase letters because they could also expose QXCR1001667763. Previously, PHSS_44778 only bans attribute values starting with restricted keywords in lowercase letters.Note: Attribute values starting with restricted keywords in a mixture of uppercase and lowercase letters are allowed. They will not expose QXCR1001667763.IMPORTANT: The (revised) final resolution is to install PHSS_44821.All Serviceguard/HP-UX clusters should install PHSS_44821. During the patch installation, the checkinstall script of PHSS_44821 will scan the CDB to check for the presence of any cluster/package attributes starting with a restricted keyword in either all uppercase or all lowercase letters. If exists, the patch installation will fail.In addition, HPE recommend also performing a manual inspection of "cmviewcl -v -fline" output to look for restricted keywords in the configurations. Please ignore the checks only for cluster and package attribute name "USER_HOST/user_host", which might contain a predefined values that starts with restricted keyword. If other cluster or package attributes in the cluster start with a restricted keyword perform the following steps:Halt the cluster:# cmhaltclRemoval of cluster and package configuration with illegal attribute values starting with a restricted keyword.A. If the illegal attribute is a cluster attribute, the whole cluster needs to be deleted.Ensure the existing cluster and package configuration has a backup before deleting the cluster.B. If the illegal attribute is a package attribute, the entire package must be deleted.Ensure that the existing package configuration has a backup and delete the package.C. If the illegal attribute is a cluster member's node name, the member node must be removed from the cluster.Refer to "Managing Serviceguard A.11.20" manual for steps to remove a node from the cluster.Install PHSS_44821 on all nodes after performing all of 2A, 2B, 2C above. The preinstall script should complete the check successfully.Re-apply the impacted cluster and packages after correction in all attributes of cluster and packages configuration files. If the node was deleted as part of Step 2C, refer to the "Managing Serviceguard A.11.20" manual for steps to add a new node to the cluster.After all the illegal attributes in cluster and packages configuration files are renamed, run the cluster:# cmrunclRECEIVE PROACTIVE UPDATES: Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively via e-mail through HPE Subscriber's Choice. Sign up for Subscriber's Choice at the following URL:Proactive Updates Subscription Form.NAVIGATION TIP: For hints on navigating HPE.com to locate the latest drivers, patches, and other support software downloads for ProLiant servers and Options, refer to theNavigation Tips document.SEARCH TIP: For hints on locating similar documents on HPE.com, refer to theSearch Tips Document.To search for additional advisories related to HP-UX 11i v3 Serviceguard, use the following search string:+Advisory +ProLiant -"Software and Drivers" +HP-UX 11i v3 Serviceguard
Operating Systems Affected:HP-UX 11.x
No external links available for this bug
Click on a version to see all relevant bugs
Hewlett Packard Enterprise Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
