Tag Archives: Linux

CommVault Simpana Linux client can backup but not restore

Leave a reply

Had an interesting issue with a Linux client that was running CommVault Simpana where it could backup fine but any attempts to restore would not work. Problem is the error indicate network/comms related. Although wasn’t the case.

Errors indicated kernel parameters, upon further investigation it was indeed kernel parameters. The errors in the logs are shown below for reference;

ClRestore.log
24346 407db90  11/09 12:20:28 105 [PIPELAYER  ]  Pipeline not Created Yet or is missing. retrying...
24346 407db90  11/09 12:20:46 105 [PIPELAYER  ] ERROR: Error: Received Message type=7 on sd=13
24346 407db90  11/09 12:20:46 105 CPipelayer::InitiatePipeline() - Error initiating pipeline!  plInitiatePipeline returned -1
24346 407db90  11/09 12:20:47 105 CCVAPipelayer::StartPipeline() - Failed to initiate pipeline
24346 407db90  11/09 12:20:47 105 CVArchive::StartPipeline() - Startup of DataPipe failed

And…

Cvd.log
12462 b7efd6d0 11/09 10:28:15 ### [CVD        ] IPCKEYS Path=//opt/hds/Base/Temp/1320798495_5482_55344016, curr=1, dest=2, topid=0
12462 b7efd6d0 11/09 10:28:15 ### [CVD        ] IPCKEYS key[0]=0x540e0055, key[1]=0x0a0e0055, key[2]=0x0b0e0055, key[3]=0x090e0055, ReaderKey=0x3b0e0055
12463 b7f4e6d0 11/09 10:28:15 ### [CVD        ] IPCKEYS Path=//opt/hds/Base/Temp/1320798495_5482_55344016, curr=2, dest=3, topid=0
12462 b7efd6d0 11/09 10:28:15 ### [CVD        ] ERROR: initIpc: shmget() err 22, flag 950, size 2007080
12463 b7f4e6d0 11/09 10:28:15 ### [CVD        ] IPCKEYS key[0]=0x540e0055, key[1]=0x0b0e0055, key[2]=0x0c0e0055, key[3]=0x090e0055, ReaderKey=0x3b0e0055
12462 b7efd6d0 11/09 10:28:15 ### [CVD        ] ERROR: plInitIpc: initIpc() fail err 22
12463 b7f4e6d0 11/09 10:28:15 ### [CVD        ] ERROR: initIpc: shmget() err 22, flag 950, size 2007080
12463 b7f4e6d0 11/09 10:28:15 ### [CVD        ] ERROR: plInitIpc: initIpc() fail err 22

I took another look at the kernel parameters set for some key kernel.* items;

kernel.sem = 500	64000	64	256
kernel.msgmnb = 65536
kernel.msgmni = 16
kernel.msgmax = 65536
kernel.shmmni = 8192
kernel.shmall = 0
kernel.shmmax = 0

Of course those last 2 caused me concern, got these modified and the the client could restore.

Turns out, the host in question although a 32bit Linux host with a PAE kernel had some how had the sysctl.conf that appeared to have values for these two kernel parameters with integers from the 64bit initscripts package. As such the values had been so large, that when applied by sysctl -p at boot, they wrap to 0. Hence the output as seen above.
From 64bit sysctl.conf file;

..
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

From 32bit sysctl.conf file;

..
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4294967295

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 268435456

Anyways, problem solved. Talk about a bit of research and testing to find the culprit on that one. Don’t ask me how the host got the 64bit sysctl.conf, I can only assume they got pushed via a scripted change, as I can’t see a bug filed against initscripts for RHEL 5.7 for this. I can only assume it was introduced by script and/or manually.

LPIC-1 Certification complete

Leave a reply

Today I passed my LPIC-1 Exam 102, so I now have completed my LPIC-1 certification. I passed LPIC-1 Exam 101 last month.

Next goal was to do LPIC-2 which is 2 more exams, that was planned for next year, however I might start doing the reading and prep, maybe I can slip in 1 more before years end. Although I won’t stress doing so, as I am waiting on BF3 to show up by post and start playing as a reward.

It’s a good to celebrate.

Move over screen, we have a new player in town

Leave a reply

As long as I have used Linux, I have been a huge fan of GNU/Screen. Although it appears we now have a new player in town.

Let me introduce Tmux, highly recommend if you haven’t checked it out, do so.

I too just happened to stumble across a post by another blogger, at the link here which introduced this as an alternative. Had I not seen the post, I wouldn’t know it even existed.

Linux kernel compiling

Leave a reply

I must say it’s been literally years since I compiled my own custom kernel, although I do recall doing it very often just to stay pretty bleeding edge. Besides it was good practise.

In the last week I took it upon myself to go over this fine art and was quite surprised to see how things have changed. I remember when the kernel had changes to allow the .config file to be available in proc file system, however it seems atleast on my Debian 6.0 install this doesn’t exist and has been left out of the kernel options. Although the config file used one the kernels shipped/installed is available in /boot file system. Don’t understand why they don’t make use of this, as it was a big thing when finally available in the kernel, guess they have reasons.

I also found out that ‘make dep’ is no longer required during the compiling process, in addition, mkinitrd is no more. Been replaced by mkinitramfs which requires the kernel config file in /boot to make the initrd anyways. And last but not least, grub2 no longer requires you to modify a menu.lst to boot the new kernel.

In any case, with a bit of research I managed to compile and install kernel 3.0.4 on a Debian 6.0 host without much fuss.

michaelf@bravo:~$ uname -a
Linux bravo 3.0.4 #1 SMP Wed Aug 31 20:59:49 EST 2011 i686 GNU/Linux

Confluence issues

Leave a reply

Just as get a few steps ahead, I hit something else. Currently if I shutdown my confluence instance, and start up using the supplied catalina.sh (run|stop) I get the following error;

SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1e648bd]) and a value of type [org.dom4j.DocumentFactory] (value [org.dom4j.DocumentFactory@1b1acc3]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.log4j:ERROR LogMananger.repositorySelector was null likely due to error in class reloading, using NOPLoggerRepository.

Of course kb says you can ignore it, of course that’s no problem except that when I do get this error after first restart the web application no longer works and I get http 404 type errors. Thus the installation is broken. This error happened right after I got the following one;

org.springframework.dao.DataIntegrityViolationException: Hibernate operation: could not insert: [com.atlassian.confluence.core.BodyContent#98322]; SQL []; Duplicate entry '98322' for key 'PRIMARY'; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '98322' for key 'PRIMARY'

at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.translate(SQLStateSQLExceptionTranslator.java:110)

caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '98322' for key 'PRIMARY'

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

Which is not fixed by the work around of setting a value in my.cnf as per the kb.

Hopefully we can get the issues resolved, would really like to use confluence with the mysql backend.

 

Mildly annoyed

Leave a reply

I was attempting to setup Confluence to use a mysql database on both Windows and MacOS X. Neither of them would work, same error produced on both. It seems to be fairly common error too.

Tried to fix it, but didn’t manage to do so. Spent a good 1-2 hours on it I would imagine too. In the end I opened a ticket with Atlassian to assist. Still waiting to hear back regarding that ticket, and it’s now been nearly 24 hours. So hopefully I should hear something tomorrow.

I am going to build a Linux virtual machine, and host it under that I think. I’ve setup Jira previously for a customer where I use to work, but never played with Confluence until now. Must admit, I quite like it too. So I think if I do get it working with mysql which I am sure we will soon enough, might have to purchase a $10 license.

So begins 2011

Leave a reply

So begins 2011. Wonder what the year will bring. All I know is that my immediate goals will be to complete my MCITP: EA. Only need to do two more exams, and if that means work won’t help with it, then I will put some funds aside when they become available to take the 2 exams I need to obtain it.

In the meantime I will wait for work to confirm if they’d like me to do my RHCE exam, so might need to start some revision for that soon as I complete the EA exams.

See what happens.

In other news, I think the blog will be moved to the cloud. Will more then likely create an A record and point it to my posterous account for the future web content. Just don’t have the time or the design skill to maintain my own site (and everything associated with it). So you will see changes happen in that department, thinking of transfering the domain to another registrar. directnic.com whom I got the domain from has been good, but I’ve been happy with the services offered by planetdomain.com, so might transfer to them. As I get free dns hosting with them without any extra associated costs, unlike directnic.com.

I only need 6 or more MX records and then some A/CNAME records after all.