CommVault Simpana Linux client can backup but not restore

Had an interesting issue with a Linux client that was running CommVault Simpana where it could backup fine but any attempts to restore would not work. Problem is the error indicate network/comms related. Although wasn’t the case.

Errors indicated kernel parameters, upon further investigation it was indeed kernel parameters. The errors in the logs are shown below for reference;

ClRestore.log
24346 407db90  11/09 12:20:28 105 [PIPELAYER  ]  Pipeline not Created Yet or is missing. retrying...
24346 407db90  11/09 12:20:46 105 [PIPELAYER  ] ERROR: Error: Received Message type=7 on sd=13
24346 407db90  11/09 12:20:46 105 CPipelayer::InitiatePipeline() - Error initiating pipeline!  plInitiatePipeline returned -1
24346 407db90  11/09 12:20:47 105 CCVAPipelayer::StartPipeline() - Failed to initiate pipeline
24346 407db90  11/09 12:20:47 105 CVArchive::StartPipeline() - Startup of DataPipe failed

And…

Cvd.log
12462 b7efd6d0 11/09 10:28:15 ### [CVD        ] IPCKEYS Path=//opt/hds/Base/Temp/1320798495_5482_55344016, curr=1, dest=2, topid=0
12462 b7efd6d0 11/09 10:28:15 ### [CVD        ] IPCKEYS key[0]=0x540e0055, key[1]=0x0a0e0055, key[2]=0x0b0e0055, key[3]=0x090e0055, ReaderKey=0x3b0e0055
12463 b7f4e6d0 11/09 10:28:15 ### [CVD        ] IPCKEYS Path=//opt/hds/Base/Temp/1320798495_5482_55344016, curr=2, dest=3, topid=0
12462 b7efd6d0 11/09 10:28:15 ### [CVD        ] ERROR: initIpc: shmget() err 22, flag 950, size 2007080
12463 b7f4e6d0 11/09 10:28:15 ### [CVD        ] IPCKEYS key[0]=0x540e0055, key[1]=0x0b0e0055, key[2]=0x0c0e0055, key[3]=0x090e0055, ReaderKey=0x3b0e0055
12462 b7efd6d0 11/09 10:28:15 ### [CVD        ] ERROR: plInitIpc: initIpc() fail err 22
12463 b7f4e6d0 11/09 10:28:15 ### [CVD        ] ERROR: initIpc: shmget() err 22, flag 950, size 2007080
12463 b7f4e6d0 11/09 10:28:15 ### [CVD        ] ERROR: plInitIpc: initIpc() fail err 22

I took another look at the kernel parameters set for some key kernel.* items;

kernel.sem = 500	64000	64	256
kernel.msgmnb = 65536
kernel.msgmni = 16
kernel.msgmax = 65536
kernel.shmmni = 8192
kernel.shmall = 0
kernel.shmmax = 0

Of course those last 2 caused me concern, got these modified and the the client could restore.

Turns out, the host in question although a 32bit Linux host with a PAE kernel had some how had the sysctl.conf that appeared to have values for these two kernel parameters with integers from the 64bit initscripts package. As such the values had been so large, that when applied by sysctl -p at boot, they wrap to 0. Hence the output as seen above.
From 64bit sysctl.conf file;

..
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

From 32bit sysctl.conf file;

..
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4294967295

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 268435456

Anyways, problem solved. Talk about a bit of research and testing to find the culprit on that one. Don’t ask me how the host got the 64bit sysctl.conf, I can only assume they got pushed via a scripted change, as I can’t see a bug filed against initscripts for RHEL 5.7 for this. I can only assume it was introduced by script and/or manually.