Symptoms

Avamar Rman plugin backups may fail with avoracle_util:get_oracle_version get_oracle_version_by_CLI() failed.Error: 2017/03/21-19:55:47.02547 [avoracle_assist] avoracle_util::get_oracle_version_by_select Entering.2017/03/21-19:55:47.02746 [avoracle_assist] avoracle_util::get_oracle_version_by_select The Oracle user name is not set. Exiting.2017/03/21-19:55:47.02949 [avoracle_assist] avoracle_util:get_oracle_version get_oracle_version_by_select() failed. Trying get_oracle_version_by_CLI().2017/03/21-19:55:47.03134 [avoracle_assist] avoracle_util::get_oracle_version_by_CLI Entering.2017-03-21 15:55:47 avoracle Info <14190>: sqlplus cmd:<$ORACLE_HOME/bin>/sqlplus var:/opt/AVMRclnt/var/clientlogs2017-03-21 15:55:47 avoracle Info <7783>: Setting spawn user/group (<default>)2017-03-21 15:55:47 avoracle Info <7953>: Username set to "<oracle_username>"2017/03/21-19:55:47.05429 [avoracle_assist] avspawn::spawn command_line_args:sqlplus -V priority background2017/03/21-19:55:47.05521 [avoracle_assist] avspawn::spawn executable_path '<$ORACLE_HOME/bin>/sqlplus'2017/03/21-19:55:47.05612 [avoracle_assist] avspawn::spawn execargs[0] '<$ORACLE_HOME/bin>/sqlplus' (arg0)2017/03/21-19:55:47.05702 [avoracle_assist] avspawn::spawn execargs[1] '-V' (arg1)2017/03/21-19:55:47.05791 [avoracle_assist] avspawn::spawn new command line 'sqlplus -V'2017/03/21-19:55:47.07329 [avoracle_assist] avspawn::avspawn(14, stdout) fd:152017/03/21-19:55:47.07626 [stdout] Startup, pid=9 tid=39 stack=fe5fbe37 td=a671802017/03/21-19:55:47.07779 [stdout] avspawnpipe::body stdout2017/03/21-19:55:47.07787 [avoracle_assist] avspawn::avspawn(16, stderr) fd:172017/03/21-19:55:47.08071 [stderr] Startup, pid=10 tid=40 stack=fe4dbe37 td=a98f402017/03/21-19:55:47.08232 [stderr] avspawnpipe::body stderr2017/03/21-19:55:47.16012 [stdout] avspawnpipe::body EOF stdout2017/03/21-19:55:47.16289 [stdout] avspawnpipe::body stdout exiting2017/03/21-19:55:47.16389 [stdout] Thread finished stdout2017/03/21-19:55:47.16014 [stderr] avspawnpipe::body EOF stderr2017/03/21-19:55:47.16637 [stderr] avspawnpipe::body stderr exiting2017/03/21-19:55:47.16744 [stderr] Thread finished stderr2017/03/21-19:55:47.16862 [avoracle_assist] avspawn::run_and_wait calling wait2017/03/21-19:55:47.16996 [avoracle_assist] avspawn::wait setting error_num from 0 to 02017/03/21-19:55:47.17086 [avoracle_assist] avspawn::run_and_wait back from wait error_num:0 wait_error:655452017/03/21-19:55:47.17175 [avoracle_assist] called avspawnpipe::~avspawnpipe2017/03/21-19:55:47.17264 [avoracle_assist] avspawnpipe::~avspawnpipe done with avspawnpipe::wait stdout2017/03/21-19:55:47.17364 [avoracle_assist] called avspawnpipe::~avspawnpipe2017/03/21-19:55:47.17457 [avoracle_assist] avspawnpipe::~avspawnpipe done with avspawnpipe::wait stderr2017/03/21-19:55:47.17612 [avoracle_assist] avoracle_util::get_oracle_version_by_CLI The sqlplus command returned no output. Exiting.2017/03/21-19:55:47.17806 [avoracle_assist] avoracle_util:get_oracle_version get_oracle_version_by_CLI() failed. Exiting.This article applies if tracing avagent shows the OS is not able to find: libsqlplus.so: 7744: stat("/opt/AVMRclnt/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/share/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/ccs/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/openwin/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/sadm/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/sfw/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/openwin/sfw/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/dt/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/X11/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/j2se/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/j2se/jre/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/openv/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/openv/java/jre/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/etc/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/etc/security/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFFFFFFFF7F2000007744: stat("/opt/SUNWcluster/lib/sparcv9/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/opt/oracle/product/12cR1_tptqa1/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/opt/ORCLcluster/lib//libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/lib/64/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: stat("/usr/lib/64/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT7744: write(2, " l d . s o . 1 : s q l".., 79) = 79How to trace avagent: On Solaris: truss -a -vall -e -f -p <avagent_pid> > /tmp/avagent.trc 2>&1 On Linux: strace -a -vall -e -f -p <avagent_pid> > /tmp/avagent.trc 2>&1

Cause

Avamar Rman plugin backups are failing because sqlplus is not finding libsqlplus.so which is required for the binary to run.

Resolution

Workaround: vi avagent.dLook for line: LD_LIBRARY_PATH LD_LIBRARY_PATH="${BASEDIR}/lib:${LD_LIBRARY_PATH}" Replace this line with: LD_LIBRARY_PATH="${BASEDIR}/lib:${LD_LIBRARY_PATH}":<ORACLE_HOME>/lib where <ORACLE_HOME> is the actual path to ORACLE HOME install Example: /u01/app/oracle/product/12.1.0/dbhome_1 Note: avagent.d is located on: Linux: /usr/local/avamar/etc Solaris: /opt/AVMRclnt/etc avagent.d restart Resolution: Contact Dell EMC Avamar support referring to this KB article to obtain a fix for this problem. 

Support Cases