OPERATIONAL DEFECT DATABASE
...
...
This document is intended to serve as a checklist to be confirmed after a successful RAN Network Function deployment. VMware considers a Network Function (DU/CU) to be complete and in a supported state only if all the applicable conditions in this article are met.
DU Server BIOS VMware Validated BIOS Dell XR11 (Icelake) 1.6.5Supermicro (Icelake) 3.4 v2.0 Dell R740 (Cascadelake) 2.11.2 Dell R750 (Icelake) 1.6.5 BIOS Configuration for Dell Icelake Servers - BIOS 1.0.2 or higher (with Telco profile option). XR11 BIOS -> System Profile Settings ->Workload ProfileTelco profileSet all BIOS parameters to match telco RAN requirements Kubernetes cluster Telco Cloud Automation 2.1Kubernetes 1.22.17, 1.23.16, 1.24.10Tanzu Kubernetes Grid 2.1.1 Telco Cloud Automation Item Expected Value Notes Cluster Status Active Cluster also should not display any warnings or errors. Nodepool Status Active Nodepool CPU Manager Scheduling policy Static Set CPU reservations on the Worker nodes as Static for CPU-intensive workloads. At least 1 CPU is reserved for Kubernetes Processes (eg. kubelet, kube-proxy), at least 1 CPU is reserved for System Processes (eg. sshd). If vendor has additional pods for non L1/L2 or non-real time applications. 2 CPUs can be reserved for Kubernetes Processes and 2 CPUs for System Processes. If 2 CPUs are reserved for Kubernetes Processes and System Processes, it is recommended to set isolcpus to be 2-(n-1). In CSAR, it can be configured as “2-{{tca.node.vmNumCPUs}}”. If 4 CPUs are reserved for Kubernetes Processes and System Processes, the isolcpus can be configured as “4-{{tca.node.vmNumCPUs}}”. At least 2 GB of Memory should be reserved, 1 GB for system processes and 1 GB for Kubernetes processes. NF Catalog Status Onboarding State: Onboarded Operational State: Enabled NF CSAR Content infra_requirements must be defined correctly. Make sure the customizations are defined correctly under infra_requirements Example: topology_template: substitution_mappings: node_type: tosca.nodes.nfv.VMware.CNF.du-1.8 node_templates: du-1.8: node_type: tosca.nodes.nfv.VMware.CNF.du-1.8 properties: descriptor_id: nfd_testing provider: xyz vnfm_info: - gvnfmdriver infra_requirements: node_components: Isolcpus in the infra-requirements section Should start with 2 e.g., 2-xx Dependent on static CPU manager policy CPU configuration (for 1+1, should start from 2, in case of 2+2, start from 4) Stalld package version stalld-1.14.1-3 or higher Stalld configuration for Logging, Threshold and Boost Duration In /etc/sysconfig/stalld: LOGGING= THRESH should be at most 5 seconds and BD (Boost Duration) should be less than THRESH. Example: THRESH=”-t 2” (2 seconds) BD=”-d 1” (1 second) LOGGING should NOT be set to: LOGGING=--log_only Tuned package version 2.15.0-3.ph3 or higher Tuned profile Profile: realtime /etc/tuned/realtime-variables.conf should have the same set of isolated_cores as configured in the kernel command-line (/proc/cmdline) Virtual Infrastructure Connection Status: Connected Connection status for that virtual infrastructure should be Connected before instantiating the CNF catalog NF Instantiation Status Instantiated Once the Network Function is instantiated check the status: status: Instantiated ESXi Configuration Item Expected Value ESXi Power management policy High performance Recommended Versions for ESXi, ESXi Drivers and Firmware and Photon OS Driver. Component Vendor VersionESXi VMware VMware ESXi 8.0.0 21203435TCA VMware 2.3vCenter VMware vCenter Server 8.0.0b 21216066PhotonOS PhotonOS VMware "3.0" RT Kernel VMware 4.19.272-4 or highervmxnet3 VMware 1.7.0.0-k-NAPIice (PCI passthru) Intel/VMware 1.11.14 (for NVM4.2, with rt kernel 4.19.272-4)iavf Intel/VMware 4.8.2 (for NVM4.2, with rt kernel 4.19.272-4)i40e (PCI passthru) Intel/VMware 2.22.18 Kubernetes Tanzu Kubernetes GroupVMware2.1.1client VMware 1.22.17, 1.23.16, 1.24.10server VMware 1.22.17, 1.23.16, 1.24.10ESXi Driver (Intel) XXV710 Intel/VMware i40en 2.1.5.0E810-C Intel/VMware Icen_1.11.3.0NIC Firmware XXV710 Intel 8.40 0x8000b1fb 20.5.13E810 Intel NVM 4.20 0x80017789 1.3346.0 ACC100 Intel ibbd-pf Intel 1.0.8-1OEM.700.1.0.15843807_20100251ibbd-tools Intel 1.0.8-1OEM.700.1.0.15843807_18401717 for ESXi 7.0 1.0.8-1OEM.800.1.0.20613240_20922857 for ESXi 8.0DPDK Intel 20.11 , 21.11, 22.11stalld 1.3.0-9tuned 2.15.0-3linuxptp 3.1.1-3Silicom STSSilicomNIC FW: 4.01 0x80014757 1.3256.0 tsyncd 2.1.2.11 ESXi driver: icen_1.9.5 iavf: 4.5.3 ice: 1.9.11 Photon / VM configuration for CU: CU VMs (both CU-CP and CU-UP) run regular (non-real-time) workloads on the Photon Linux (generic) kernel (non-real-time). Item Command Expected Value Notes Kernel Flavor uname -a 4.19.xxx-x.ph3 The kernel version suffix ending with .ph3 indicates that the kernel flavor is Linux (generic). Kernel Version (TCA 2.3) uname -a 4.19.272-5.ph3 Tuned Version rpm -qa tuned 2.15.0-3.ph3 or higher The use and installation of tuned with the workload is up to the CU vendor. Tuned Profile tuned-adm active cpu-partitioning or N/A The tuned profile depends on what the CU vendor selected to use with the workload. A non-real-time system such as a CU should NOT have real-time application threads. This can be verified using ps or top.Example: ps -eo cmd,pid,rtprio | grep -v -e "\[" -e "-$" This prints the command-name, PID and Real-Time priority of all processes, and skips kernel threads (which are always in square brackets []) and those that do not have a real-time priority (indicated by a “-”, as opposed to a priority number. This list should come up empty, which indicates that there are no real-time application threads in the system. Photon / VM Configuration for DU: DU VMs run a mix of real-time and non-real-time workloads on the Photon linux-rt kernel (real-time). Item Command Expected Value Notes Kernel Flavor uname -a 4.19.xxx-rtxx-xx.ph3.rtThe kernel version suffix ending with .ph3-rt indicates that the kernel flavor is linux-rt (real-time). Kernel Version (TCA 2.3) uname -a 4.19.272-4.ph3.rtUpdate using KB87717, KB90345 if differentKernel Command-Line Parameters cat /proc/cmdline The kernel command-line MUST have the following values: isolcpus=<list-of-isolated-CPUs> irqaffinity=<other-than-isolated-CPUs> rcu_nocbs=<list-of-isolated-CPUs> nohz=on nohz_full=<list-of-isolated-CPUs> nosoftlockup nowatchdog nmi_watchdog=0 idle=halt intel_idle.max_cstate=0 cpuidle.off=1 mce=off or mce=ignore_ce (Either one is valid) audit=0 cgroup.memory=nokmem The kernel command-line MAY have the following values to achieve lower latency: intel_pstate=disable skew_tick=1 tsc=reliable softlockup_panic=0 selinux=0 enforcing=0 Tuned Version rpm -qa tuned 2.15.0-3.ph3 or higher Tuned Profile Command 1: tuned-adm active Command 2: /proc/cmdine Output for command 1: realtime or a derived profile specific to the Network Function vendor. Output for command 2: The output must include the command-line parameters configured in /boot/tuned.cfg (which is populated by the chosen tuned profile). This indicates that any bootloader parameters configured by the tuned profile are also active (post reboot). The tuned profile could be ‘realtime’ or a custom profile derived from the real time profile shipped by the Network Function vendor. Stalld Version rpm -qa stalld 1.14.1-0 or higher Stalld Configuration cat /etc/sysconfig/stalld The LOGGING= field should be blank, like shown below: LOGGING= THRESH should be at most 5 seconds and BD (Boost Duration) should be less than THRESH. Eg: THRESH=”-t 2” (2 seconds) and BD=”-d 1” (1 second) LOGGING should NOT be set to: LOGGING=--log_only By default, the stalld package configuration is set to logging only (no priority boosting). This should be changed by the Network Function vendor to make it fully active and not just leave it in logging-only mode. Also, the starving threshold should be small for stalld to detect and revive the system from potential starvation. Stalld Service Status systemctl status stalld The output must include: Active: active (running) Ptp systemd service status Systemctl status ptp4l Systemctl status phc2sys The output must include: Active: active (running) PTP drift status pmc -u -d <domainNumber> -b 0 'GET TIME_STATUS_NP' The output should show gmpresent:true Should also list gmIdentity Typical output would look like this: 40a6b7.fffe.0deb61-0 seq 0 RESPONSE MANAGEMENT TIME_STATUS_NP master_offset -6218 ingress_time 1614359358101173060 cumulativeScaledRateOffset +0.000000000 scaledLastGmPhaseChange 0 gmTimeBaseIndicator 0 lastGmPhaseChange 0x0000'0000000000000000.0000 gmPresent true gmIdentity fcaf6a.fffe.02e8f4 Replace <domainNumber> properly. E.g., pmc -u -d 24 -b 0 'GET TIME_STATUS_NP' PTP Clock statuspmc -u -b 0 -d <domainNumber> 'GET PORT_DATA_SET'The output should show ‘portState’ as SLAVE. # pmc -u -b 0 -d 24 'GET PORT_DATA_SET' sending: GET PORT_DATA_SET 000c29.fffe.666be9-1 seq 0 RESPONSE MANAGEMENT PORT_DATA_SET portIdentity 000c29.fffe.666be9-1 portState SLAVE logMinDelayReqInterval -4 peerMeanPathDelay 0 logAnnounceInterval -3 announceReceiptTimeout 3 logSyncInterval -4 delayMechanism 1 logMinPdelayReqInterval 0 versionNumber 2Replace <domainNumber> properly. vHT settingCheck vHT/SMT tag is added in csar file“enableSMT: true” is added in csar file and CNF is instantiated KB article for updating Photon rt-kernel versions from what is built into TCA: KB 87717 Memory optimization configuration for VM: ESXi commands to free up additional memory if using low memory servers (64GB RAM) Need to login to ESXi host as root. Reboot is required after modification. Item Command Expected Value Notes Intel Icen 1.8.5esxcli software vib install -d /<location of the downloaded driver>/Intel-icen_1.8.5.0-1OEM.xxx.zip Icen driver is upgraded to 1.8.5 NOTE: Intel icen 1.8.5 or higher reduces memory consumption.Intel ACC100 Ibbd-pfesxcli software component apply –d /<location of the extracted driver>/VMW-esx-7.0.0-Intel-ibbd-pf-1.0.8-1OEM.xxx.zip ACC100 ibbd-pf driver is upgraded to 1.0.8 Ibbd-toolesxcli software component apply –d /<location of the extracted driver>/VMW-esx-7.0.0-Intel-ibbd-tool-1.0.8-1OEM.xxx.zip ACC100 ibbd-tool driver is upgraded to 1.0.8 MaxVMs esxcli system settings kernel set -s maxVMs -v 6MaxVMs = 6This should free up ~1GBDisable DRSS/RSSesxcfg-module icen -s 'DRSS=0,0,0,0 RSS=0,0,0,0DRSS = 0,0,0,0 RSS = 0,0,0,0This should free up ~450 GB
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
