If you run Simpana 9.0 and have some issues backing up Windows System State via CentOS/RHEL 6.2 MediaAgents, don’t fear SP5a will fix some quirks.
I had an issue matching the above and seems it will be resolved in SP5a (when released). If you have any issues, I recommend opening a support ticket with CommVault to have it checked out.
Came across an interesting thing today and thought I would publish it, as it could certainly catch people out.
You’ll find the Client Readiness Check will pass on Windows, Linux and Macintosh clients (those I have checked so far), although you backup fails against the hosts with the error similar to below at the 5% mark.
Error Code: [19:599] Description: Loss of control process ifind.exe. Possible causes: 1. The control process has unexpectedly died. Check Dr Watson log or core file. 2. The communication to the control process machine wentx862k8-1 might have gone down due to network errors. 3. If the machine wentx862k8-1 is a cluster, it may have failed over. 4. The machine wentx862k8-1 may have rebooted.
And readiness check against the client will pass as per below;
It appears the readiness check doesn’t actually check all services/processes on the client required to perform the backup. i.e. In this situation, I killed the EvMgrC process, which is certainly required, it’s as important the cvd process for the client to function and be backed up. Without it, the backup will fail to work, however as you can see Readiness Check will pass for the client, as it looks like EvMgrC is not actual checked.
Of course it’s rare for this EvMgrC to be unavailable like this, however it certainly could happen and cause confusion when client readiness check is performed and clearly says everything is okay.
On Windows clients be sure to check that the essential services are running, and on Unix platforms run the following command to ensure the key items have an associated PID.
# cd /opt/simpana/Base # ./simpana list +---------------------------------+---------+----------------------------------+ | Service name | PID | Service command | +---------------------------------+---------+----------------------------------+ | cvlaunchd | 4020 | /opt/simpana/Base/cvlaunchd | +---------------------------------+---------+----------------------------------+ | cvd | 4091 | /opt/simpana/Base/cvd | +---------------------------------+---------+----------------------------------+ | EvMgrC | 4085 | /opt/simpana/Base/EvMgrC | +---------------------------------+---------+----------------------------------+
Hope this helps anyone in the internet that might come across this. This type of failure can be seen with the following error message reported in the FileScan.log on the client too as outlined below, good indication of the issue probably being as described.
3044 be8 11/15 19:28:01 ### EvSocket::doConnect() - Could not connect to wentx862k8-1(wentx862k8-1):EvMgrC: Connect to 127.0.0.1:8402 failed: Connection refused
Although it appears that Windows clients appear to recover the EvMgrC process after a period of time, so it should correct itself, however not seen that same behaviour on Linux and/or Macintosh clients as yet. Will keep investigating if they too recovery it.
Had an interesting issue with a Linux client that was running CommVault Simpana where it could backup fine but any attempts to restore would not work. Problem is the error indicate network/comms related. Although wasn’t the case.
Errors indicated kernel parameters, upon further investigation it was indeed kernel parameters. The errors in the logs are shown below for reference;
ClRestore.log 24346 407db90 11/09 12:20:28 105 [PIPELAYER ] Pipeline not Created Yet or is missing. retrying... 24346 407db90 11/09 12:20:46 105 [PIPELAYER ] ERROR: Error: Received Message type=7 on sd=13 24346 407db90 11/09 12:20:46 105 CPipelayer::InitiatePipeline() - Error initiating pipeline! plInitiatePipeline returned -1 24346 407db90 11/09 12:20:47 105 CCVAPipelayer::StartPipeline() - Failed to initiate pipeline 24346 407db90 11/09 12:20:47 105 CVArchive::StartPipeline() - Startup of DataPipe failed
And…
Cvd.log 12462 b7efd6d0 11/09 10:28:15 ### [CVD ] IPCKEYS Path=//opt/hds/Base/Temp/1320798495_5482_55344016, curr=1, dest=2, topid=0 12462 b7efd6d0 11/09 10:28:15 ### [CVD ] IPCKEYS key[0]=0x540e0055, key[1]=0x0a0e0055, key[2]=0x0b0e0055, key[3]=0x090e0055, ReaderKey=0x3b0e0055 12463 b7f4e6d0 11/09 10:28:15 ### [CVD ] IPCKEYS Path=//opt/hds/Base/Temp/1320798495_5482_55344016, curr=2, dest=3, topid=0 12462 b7efd6d0 11/09 10:28:15 ### [CVD ] ERROR: initIpc: shmget() err 22, flag 950, size 2007080 12463 b7f4e6d0 11/09 10:28:15 ### [CVD ] IPCKEYS key[0]=0x540e0055, key[1]=0x0b0e0055, key[2]=0x0c0e0055, key[3]=0x090e0055, ReaderKey=0x3b0e0055 12462 b7efd6d0 11/09 10:28:15 ### [CVD ] ERROR: plInitIpc: initIpc() fail err 22 12463 b7f4e6d0 11/09 10:28:15 ### [CVD ] ERROR: initIpc: shmget() err 22, flag 950, size 2007080 12463 b7f4e6d0 11/09 10:28:15 ### [CVD ] ERROR: plInitIpc: initIpc() fail err 22
I took another look at the kernel parameters set for some key kernel.* items;
kernel.sem = 500 64000 64 256 kernel.msgmnb = 65536 kernel.msgmni = 16 kernel.msgmax = 65536 kernel.shmmni = 8192 kernel.shmall = 0 kernel.shmmax = 0
Of course those last 2 caused me concern, got these modified and the the client could restore.
Turns out, the host in question although a 32bit Linux host with a PAE kernel had some how had the sysctl.conf that appeared to have values for these two kernel parameters with integers from the 64bit initscripts package. As such the values had been so large, that when applied by sysctl -p at boot, they wrap to 0. Hence the output as seen above.
From 64bit sysctl.conf file;
.. # Controls the maximum shared segment size, in bytes kernel.shmmax = 68719476736 # Controls the maximum number of shared memory segments, in pages kernel.shmall = 4294967296
From 32bit sysctl.conf file;
.. # Controls the maximum shared segment size, in bytes kernel.shmmax = 4294967295 # Controls the maximum number of shared memory segments, in pages kernel.shmall = 268435456
Anyways, problem solved. Talk about a bit of research and testing to find the culprit on that one. Don’t ask me how the host got the 64bit sysctl.conf, I can only assume they got pushed via a scripted change, as I can’t see a bug filed against initscripts for RHEL 5.7 for this. I can only assume it was introduced by script and/or manually.
Seem’s I just keep on finding these documentation errors for Simpana 9.0. I keep reporting them, so I hope they keep getting fixed, in any case I’ll also blog about them so if they don’t get fixed I got something to reference back to, so I can check them later for correction.
The following page here, contains the following faults, as per screen captures;
Completed both CommVault Specialist tracks yesterday. Disaster Recovery and Storage. I am sure I still have plenty to learn, so will continue to develop my knowledge and skills.
Logo for certification below.
CVCSpecialist
Attempted the CommVault Specialist exams available in the last 2 days and unfortunately failed to pass by 1-2 questions between the two of them. Oh well guess I need to do some serious review.
I ended up doing one of them without having done the course and/or seen the material. So based it purely on my experience of the product for the last 2+ months, which I thought was not bad considering I failed to pass by only 2 questions.
Anyways, I will hit them again soon, once I have done some more revision.
Finally, got it to work. Will post a tutorial on what I did to perform it. I have been able to do a restore to a new instance on the same host as the original instance that was backed up using a selective online full. The auxiliary instance I restored to is not known to CommCell Console.
Will post more details in the next few days, and will also attempt the same restore when the instance is already known to CommCell Console.
Not to mention perform on Linux to Linux too. Which would be similar to any other Unix to Unix platform.
All I am trying to do is a simple restore from one instance to a new instance, and just can’t seem to get it.
I have followed various documents, resolved various problems along the way. Yet it still won’t work. It shouldn’t seriously be this difficult. As I am doing it on the same host which happens to be Windows.
I could pull out my hair and seriously throw the machine across the room. I know I am close, I guess I need to stick at it.
Grrr.
PS. When I do, I will document it in a more simple baby steps type way so anyone can follow it.
Completed my CommVault Certified Support certification exam today. Decided to take a stab at it and passed easily enough.
This concludes the 3 core certifications. Will now start to revise the 2 specialist ones and attempt these soonish.
Hope to start doing some RHCE revision, as I am borrowing the course book from a friend who recently went on the course.
CVCS
And today I took an attempt at the CommVault Certified Engineer exam and passed. So I am now CommVault Certified Engineer.
I only need to complete one more which is the CommVault Certified Support exam, which I will do very shortly. At which point I have completed the set of 3 I needed to do.
CVCE