Symptoms

Note that technology is deterministic - by definition, something failing where it was previously working requires something to have changed. A newly implemented component with no working history is significantly different to troubleshoot.Obvious and identifiable changes: Software version or configuration changeHardware or transport change or reconfigurationEnvironmental events like power outages, transport failures, disk failures Undetected changes Corruption of software resources or filesystemsConflicting operations on network, SAN or hostsReconfiguration of transport or hosts by other teamsData volume or transport traffic increaseNon-catastrophic environmental events which don't raise alerts When preparing a problem for extended review and technical troubleshooting work, ensure the following general directives are observed. These items will ensure that sufficient data is collected mechanically, and that the correct questions are asked to properly qualify the issue.

Cause

Define the problem clearly: WHAT is the operation or attribute which is failing, deviant, or is the subject of questions? What details of the failure sequence are available?WHICH specific components and data paths are involved, e.g. binaries, SAN / network switches? Software versions? Storage types or instances?WHERE in the normal workflow does the problem occur? If the same operation is attempted using a different workflow, do results differ?WHEN did the problem first occur? How long does it last? What dates/times has it occurred since? Have recent changes been identified?HOW does the problem occur: consistently or intermittently? How is the problem alleviated after it occurs?WHO are the hosts involved in the problem, and what are their properties (NetWorker host type, NetWorker version, name, IP, OS, CPU etc.)?WHY are some components affected, when similar components are not? What differences exist between unaffected but like components?

Resolution

GENERIC LOGS Most NetWorker logs can be collected from Windows and Linux NetWorker servers, storage nodes, and/or clients using the NSRGET utility. The utility can be downloaded from: https://central.dell.com/solutions/Networker-Tools 1. Extract the NSRGET utility to any temporary directory2. Instructions on how to use NSRGET can be found in: NetWorker: How to Use the NSRGet NetWorker Data Collection ToolThe following KB details logs used by NetWorker: NetWorker: Log Files and LocationsCollect the following from all involved NetWorker hosts, (including Server and Storage Nodes): NetWorker debug files: directory listing of [nsr]/debug This directory should be empty. If there are files under /nsr/debug which coincide with the date an issue arose, it could be indicative of a problem. NetWorker core files: Linux: /nsr/coresWindows: C:\Program Files\EMC NetWorker\nsr\cores NetWorker app logs: zip of [nsr]/apps; zip of [nsr]/applogs; zip of [nsr]/rmagentps/logs; bundle of [nsr]/logs/lib*, nsrbr*, nsrsnap**NetWorker logs: [nsr]/logs/daemon.raw, [nsr]/logs/networkr.raw NetWorker: How to use nsr_render_logNetWorker: How to automatically render daemon.raw to daemon.log in real time NetWorker client configuration: zip of [nsr]/res/nsrladb, [nsr]/res/servers file (may not exist), [nsr]/res/nsrdb (if NetWorker server) These are collected by the NSRGET utility. Startup script: Solaris & Linux - /etc/init.d/networker, HPUX - /sbin/init.d/networkerBackup policy logs: Linux: /nsr/logs/policy/POLICY_NAME/WORKFLOW_NAMEWindows: C:\Program Files\EMC NetWorker\nsr\logs\policy\POLICY_NAME\WORKFLOW_NAME​​​​​​​ NetWorker version and build: NetWorker: Methods for Identifying NetWorker Software versionKnown patches installed for NetWorkerOperating system, Edition, Patch Level, CPU and bit-width (32/64 bit) Linux: cat /etc/*release ; uname -mWindows: systeminfo | findstr "OS Type"NSRGET or EMCREPORTS will also detail the OS version and system architecture. EMC Reports utility: https://www.dell.com/support/home/en-us/product-support/product/emcreports/drivers EMC Reports collects additional host information including full OS details and configuration, operating system messages or event logs; some of this information is also collected in NSRGET; whereas EMC Reports is more operating specific logs. EMC Reports can be collected in conjuction with NSRGET or on a system which may not have a NetWorker client software installed; for example, a VMware Virtual Machine backed up by vProxy. Core dumps: NetWorker: Troubleshooting Process Core DumpsFor suspected name resolution issues: NetWorker: Name Resolution Troubleshooting Best Practices Collect the following from the NetWorker Server only: NetWorker resource database: Zipped copy of [nsr]/res/nsrdbNetWorker media database: How To Export the NetWorker Media Database Using nsrmmdbasmNSRGET will collect most logs needed; however it will not collect: AUTHC (authentication) logs: Linux: /nsr/logs/authcWindows: C:\Program Files\EMC NetWorker\nsr\authc-server\tomcat\logs NetWorker Management Console (NMC) (gstd) logs: Linux: /opt/lgtonmc/logs/Windows: C:\Program Files\EMC NetWorker\Management\GST\logs NetWorker Web User Interface (NWUI) logs: Linux: /opt/nwui/logsWindows: C:\Program Files\EMC NetWorker\nwui\logs Collect the following from any NetWorker Storage Node (including Server): NetWorker discovery command: inquire -lc outputNetWorker bus interrogation: lusbinfo / lusbinfo -v outputNetWorker SJI command outputs: sjisn [x.y.z], sjirdtag [x.y.z], sjirjc [x.y.z] (where x.y.z is SCSI address OR device file handle)NetWorker CDI command outputs: cdi_inq -f [dev] -v, cdi_get_config -f [dev] -v, cdi_get_status -f [dev] -v, cdi_get_blocksize -f [dev] -v (where dev is device file handle)NetWorker device and mount state summary: nsrmmNetWorker simple autochanger output: nsrjb -C [jukebox_name]NetWorker disk details: recursive directory listings for any affected AFTD/DD/FTD structure, as well as details of the disk device and its connection method to host.Operating system, Edition, Patch Level, CPU and bit-width (32/64 bit) Linux: cat /etc/*release ; uname -mWindows: systeminfo | findstr "OS Type"NSRGET or EMCREPORTS will also detail the OS version and system architecture. NSRGET and/or EMC Reports are recommended for support engagements. Note: The above device commands, as well as OS device commands listed below, may cause SCSI resets if used on active devices. Use only as needed when all tape devices attached to the system, or those specified, are idle across all host instances.Collect the following from any affected NetWorker License Manager Host: Licensing command output: nsrlic -CNetWorker License database: zip of [nsr]/res/lgtolm.res, [nsr]/res/lictype.res, [nsr]/logs/lgtolmd.log, [nsr]/logs/license* Collect the following from any affected NetWorker Management Console Host: Operating system, Edition, Patch Level, CPU and bit-width (32/64 bit) Linux: cat /etc/*release ; uname -mWindows: systeminfo | findstr "OS Type"NSRGET or EMCREPORTS will also detail the OS version and system architecture. Management Console Log (gstd.raw - ideally, rendered): Linux - /opt/lgtonmc/logs Windows - [Install path]\Management\GST\logsNetWorker: How to use nsr_render_log NMC version including build number NetWorker: Methods for Identifying NetWorker Software version Oracle Java JRE version: https://www.java.com/en/download/help/version_manual.html NetWorker Runtime Environment (NRE): Linux: rpm -qa | grep nre/opt/nre/java/latest/bin/java -version Windows: Java version will be listed under: C:\Program Files\NRE\java\jre#.#.#_### If multiple Java folders are listed check System Environment Variable JAVA_HOME or NSR_JAVA_HOME for which path is currently used by NetWorker NRE version will be listed from Control Panel, under Programs and Features Java Webstart file: [nmc]/web/gconsole.jnlpNMC configuration file: [nmc]/etc/gstd.conf NetWorker VMware Protection (NVP) vProxy Appliance: Log bundles: NVP-vProxy: How to collect bundle for vProxyProxyHC (Health Check) Tool: NVP-vProxy: How to use health check tool ProxyHC on vProxy appliance

Support Cases