Simpana 10 – Linux client prepost command execution failure

Came across an interesting condition today, which took me a bit of testing to identify why the job would go into a pending state. This one relates to Simpana 10 on a Linux client where you have a File System iDA with a PrePost command being executed. In my test below the script is doing nothing special, it’s merely to have something to execute to show the behavior. I’ve provided it below purely for reference.

[root@jldb1 bin]# cat pre-scan.sh
#!/bin/sh
# test
#

echo $1 $2 $3 $4 $5 $6 $7 $8 $9 >> /root/pre-scan.log
exit 0

Job goes pending and produced the following errors and output below;

JPR (Job Pending Record)
Error Code: [7:75]
Description: Unable to run [/usr/local/bin/pre-scan.sh] on client.
Source: jwcs, Process: startPrePostCmd

[JobManager.log – commserve]

3024  d88   03/27 18:16:26 21  Scheduler  Set pending cause [Unable to run [/usr/local/bin/pre-scan.sh] on the client.                 ]::Client [jwcs] Application [startPrePostCmd] Message Id [117440587] RCID [0] ReservationId [0].  Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].
3024  118c  03/27 18:16:26 21  Scheduler  Phase [Failed] message received from jwcs.lab.heimic.net] Module [startPrePostCmd] Token [21:3:1] restartPhase [0]
3024  118c  03/27 18:16:26 21  JobSvr Obj Phase [3-Pre Scan] for Backup Job Failed. Backup will continue with phase [Pre Scan].

[startPrePostCmd.log – commserve]

4940  e4c   03/27 20:21:46 ### Init() - Initializing job control [token=21:3:7,cn=jwcs], serverName [jwcs.lab.heimic.net], ControlFlag [1], Job Id [21]
4940  e4c   03/27 20:21:47 ### Cvcl::init() - CVCL: Running in FIPS Mode
4940  e4c   03/27 20:21:48 ### CVJobCtrlLog::registerProcess(): successfully created file [C:Program FilesCommVaultSimpanaBaseJobControl4.940]
4940  e4c   03/27 20:21:48 ### ::main() - jobId 21 - restoreTaskId = 0
4940  e4c   03/27 20:21:48 ### ::main() - jobId 21 - adminTaskId = 0
4940  e4c   03/27 20:21:48 ### ::getBackupCmdAndMachine() - jobId 21 - before construct application id
4940  e4c   03/27 20:21:49 ### ::getBackupCmdAndMachine() - appTypeId = 29
4940  e4c   03/27 20:21:49 ### ::getBackupCmdAndMachine() - jobId 21 - symbolic AppId = 2:20
4940  e4c   03/27 20:21:49 ### ::getBackupCmdAndMachine() - jobId 21 - prePostId = 1
4940  e4c   03/27 20:21:49 ### ::getBackupCmdAndMachine() - jobId 21 - preifind cmd = /usr/local/bin/pre-scan.sh
4940  e4c   03/27 20:21:49 ### ::main() - jobId 21 - commandPath = /usr/local/bin/pre-scan.sh
4940  e4c   03/27 20:21:49 21  ::main() - jobId 21 - before execute cmd
4940  e4c   03/27 20:21:49 21  ::main() - jobId 21 - Use Local System Acct.
4940  e4c   03/27 20:21:49 21  ::main() - jobId 21 - remoteexename = [/usr/local/bin/pre-scan.sh]
4940  e4c   03/27 20:21:49 21  ::main() - jobId 21 - args = [ -bkplevel 1 -attempt 7 -job 21]
4940  e4c   03/27 20:21:49 21  executePrePostCmd() -  Attempting to execute remote command on client [jldb1]..
4940  e4c   03/27 20:21:49 21  executePrePostCmd() - jobId 21 - Received error text from server cvsession [Unknown Error]
4940  e4c   03/27 20:21:49 21  executePrePostCmd() - jobId 21 - Error [0] returned from executeRemoteCommand /usr/local/bin/pre-scan.sh
4940  e4c   03/27 20:21:49 21  EvEvent::setMsgEventArguments() - MsgId[0x0700004b], Arg[1] = [117440623]
4940  e4c   03/27 20:21:49 21  EvEvent::setMsgEventArguments() - MsgId[0x0700004b], Arg[2] = [/usr/local/bin/pre-scan.sh]
4940  e4c   03/27 20:21:49 21  EvEvent::setMsgEventArguments() - MsgId[0x0700004b], Arg[3] = []
4940  e4c   03/27 20:21:49 21  EvEvent::setMsgEventArguments() - [MsgId[0x0700004b][]: [3] Args Pushed, [1] Args expected.
4940  e4c   03/27 20:21:49 21  ::exitHere() - jobId 21 - Exiting due to failure.
4940  e4c   03/27 20:21:49 21  BKP CALLED COMPLETE (PHASE Status::FAIL), 21. Token [21:3:7]
4940  e4c   03/27 20:21:53 21  ::exitHere() - jobId 21 - startPrePostCmd Terminating Event.
4940  238c  03/27 20:21:53 21  CVJobCtrlLog::unregisterProcess(): successfully removed file [C:Program FilesCommVaultSimpanaBaseJobControl4.940]

[cvd.log – client]

30846 427e0940 03/27 20:21:50 ### [CVipcD] Requests from non-CS with hostname [jwcs.lab.heimic.net] and clientname [jwcs] to execute in user entered path are not allowed

I worked out this problem is caused by lack of value in regkey sCSGUID as found in the location below;

/etc/CommVaultRegistry/Galaxy/Instance001/CommServe/.properties

Sample below;

[root@jldb1 ]# cat /etc/CommVaultRegistry/Galaxy/Instance001/CommServe/.properties | more
bCSConnectivityAvailable 1
sCSCLIENTNAME jwcs
sCSGUID
sCSHOSTNAME jwcs.lab.heimic.net
sCSHOSTNAMEinCSDB jwcs.lab.heimic.net

sCSGUID should be populated and its lack of value causes this condition with pre-scan script execution.

Fix:

Easiest method to recreate this regkey value is to do a local uninstall of the simpana services on the client. Revoke the client certificate in Simpana Console via Control Panel – Certificate Administration for the client in question. Followed by a reinstall.

Observation:

Subclients that have no scripts being executed as part of the backup will run fine if this regkey value is missing. You will never see a problem until you add a script. In addition, clients that have a simpana firewall configuration will be broken and subclients without scripts will break too. As the regkey value is used for simpana firewall configuration exchange I believe based on my testing.

Hope you enjoy my post… drop me a comment if you like the content and/or it helps you.