Category: Backup

ERROR 2013 (HY000): Lost connection to MySQL server during query

Recently had a problem with a MySQL iDA backup on a Linux platform. I was seeing a failure where we would get the error below (and per subject) included in the CommVault backup error.

ERROR 2013 (HY000): Lost connection to MySQL server during query

Upon further investigation it was determined that the error was being triggered during a MySQL query as performed by the MySQL iDA during backup attempt. When we attempted to do the same query manually against a MySQL connection we’d also get the same error. It was determined from the MySQL iDA logs that the database causing us the problem was called “prod”.

See the example below;

# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 4
Server version: 5.1.32-community-log MySQL Community Server (GPL)

Type 'help;' or 'h' for help. Type 'c' to clear the buffer.

mysql> use prod
Database changed
mysql> show table status;

ERROR 2013 (HY000): Lost connection to MySQL server during query

As you can see the error was reproducible. Further investigation determined that the database was corrupt in some way, as we could see a large number of errors in the MySQL logs. Fix the corruption on the database and/or exclude it from the backups until resolved by the MySQL DBA.

Simpana v10 Service Pack 4 is available

Simpana v10 Service Pack 4 is available now.

You can read about the new features here, and release notes here.

The release of this service pack marks the possibly of customers to do upgrades, you should of course go check out all the requirements for this at the Maintenance Advantage Portal.

Simpana 10 – Oracle iDA – Backup current controlfile via RMAN command line

Quick demo on how you can backup from command line using RMAN and still have this backup go via Simpana 10 and the Oracle iDA. This backup is a control file only backup, follow on from the GUI based example in my last post.

Simpana 10 documentation link covering this can be found here.

Contents of my backup.txt from screen cast on this post.

run {
allocate channel ch1 type 'sbt_tape'
PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,
ENV=(CvClientName=lcosx86v56-1,CvInstanceName=Instance001)";
backup current controlfile;
release channel ch1;
}

 

Simpana 10 Oracle iDA installation onto Linux client

Demo deployment of Simpana 10 Oracle iDA installation onto Linux client.

You’ll note I first check to see which user is running the pmon process for our database, in this case it’s “oracle”.

Next I change to the user oracle and determine the users primary group. As this group will be used for the installation. In this example it’s “oinstall”.

I’ll be posting another part to this, as the installation is merely the first step. Once installed you have to do some configuration from the Simpana console. i.e. create the oracle instance etc.

Commvault documentation covering the install above can be found here.

Simpana 10 Local uninstall of Simpana services on Linux

Demo of a Simpana 10 Local uninstall of Simpana services on Linux.

Note that I select the option only to remove the software/services on the client and not update the commserve about this removal. As I want to keep the backup history of my client in the commserve. This will enable me to reinstall the software with the same details on this client to continue my backups.

MediaAgent Installation on Linux – Simpana 10

Needed to install the Simpana 10 MediaAgent onto a Linux client in my test environment and I thought it might be a good idea to record and share.

The install is an interactive one using the media (DVD3).

As you can see in the video my freshly installed client still had it’s hostname associated with the localhost entry in the hosts file. The installer detects this and recommends you fix it. So I did.

In addition, I also fixed up the services file entries for 8400 and 8402. Installer will see these and not allow you to pick those ports during the install, so since I know this condition was going to happen next I edit these files too. That condition is going to be resolved in a future service pack, so you won’t need to do that.

Enjoy.

MySQL backup via Simpana fails with Error Code 87:15

So you’ve just deployed the Simpana MySQL iDA and attempt your first backup, however get an error per below;

Error Code: [87:15]

Description: MySQL Database: [~client~] Data Backup Failed with MySQL Error: [~mysqldump: Got error: 1227: Access denied; you need the RELOAD privilege for this operation when doing refresh ~].

This failure is talking about the MySQL user that you configured in the GUI for the MySQL iDA. Basically the account lacks “RELOAD” privilege, so you should go fix it up before trying again.

Simpana 10 – SAP for Oracle backup failure

Came across an Simpana 10 SAP for Oracle iDA condition where the job would fail per below;

Error Code: [18:45]
Description: Snap Initialization failed with an error: [Backup Index Initialization failed. Please check the logs.]
Source: clientname, Process: backint_oracle

If you look into the $SAPDATA_HOMEsapbackup directory for the detail file associated with the failed backup you should also see something like below;

Rel 10.0.0(BUILD116) - CS=<commserve> CC=<clientname> retCode=1 bUpgradeFlag=208 upgradeTo9oTime=0
Got Mangled MediaAgent=<ma.libobk.so*ma*8400*8402> from Archive Manager
creating BCD dir=</opt/simpana/iDataAgent/jobResults/2/115/417/> failed with retCode=<-1>

BR0280I BRBACKUP time stamp: 2013-04-30 14.52.41
BR0279E Return code from '/sapmnt/OBK/exe/uc/rs6000_64/backint -u OBK -f backup -i /oracle/OBK/sapbackup/.belcejum.lst -t file -p /oracle/OBK/112_64/dbs/initOBK.utl -c': 2

This failure is indicative of a permissions error. i.e. Simpana installation path and/or jobResults  (if been placed outside of the Simpana installation path) have permissions that restrict the SAP user from writing. Since the SAP for Oracle backups are ran via a Unix SAP user account.

Simpana 9 Oracle or File System SnapProtect Failure via NetApp

Had an interesting condition in the last few days and really thought I would share, as it seemed like something really simple that could be checked by anyone.

Likewise, this could affect Oracle and/or File System SnapProtect jobs via NetApp.

Error: 62:2145

Description: The client machine initiator address is not visible on the file server. Please check the SAN or iSCSI connectivity between the file server and the client.

Upon reviewing this one, I identified the following conditions;

[ClOraAgent.log]
11502 1   04/16 18:20:07 19593 ManageONTAP::OpenConnection: Trying HTTP on port:[80]
11502 1   04/16 18:20:08 19593 ManageONTAP::OpenConnection: FileServer:[LBnNNNnn-n] Version:[NetApp Release 8.0.2P6 7-Mode: Fri Jan 27 14:48:08 PST 2012]
11502 1   04/16 18:20:09 19593 CVSnapEngineNetApp::cvso_snapDevices Check for snap clone dependences on volume:[volume_boot] on file server:[LBnNNNnn-n] returned success:[true] dependency on:[false]

First connection to the FileServer completes, however…

[CVMA.log]
17401 11  04/16 18:20:23 19593 ManageONTAP::OpenConnection: Trying HTTP on port:[80]
17401 11  04/16 18:20:23 19593 ManageONTAP::OpenConnection: Failed to get HTTP system version for server [LBnNNNnn-n]. Error:[-1][Can't connect to host (err=-8).]. This could be because 1. The server is powered down. 2. The server name is not resolvable from this host 3. Error with HTTP configuration. Please rectify the above errors (if any) and try again.
17401 11  04/16 18:20:23 19593 ManageONTAP::OpenConnection: Trying SSL on port:[443]
17401 11  04/16 18:20:23 19593 ManageONTAP::OpenConnection: Failed to get HTTPS system version for server[LBnNNNnn-n]. Error:[-1][Can't connect to host (err=-8).]. This could be because 1. The server is powered down. 2. The server name is not resolvable from this host 3. Error with HTTPS configuration. Please rectify the above errors (if any) and try again.

As you can see the next attempt fails… The final attempt will be to determine if the HBA’s from the client can be seen against the FileServer. Should we also get failures again connecting to the FileServer it will fall through with the error below.

[CVMA.log]
17401 11  04/16 18:20:49 19593 CvSnapNetApp::detectHBA() - No initiators logged in. Getting all initiators.
17401 11  04/16 18:20:49 19593 CvSnapNetApp::detectHBA() - Returning initiator ID list length:[1] iSCSI:[false] LoggedIn:[false]
17401 11  04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.
17401 11  04/16 18:20:49 19593 ERROR:60516:Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.
17401 11  04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps Printing status of clones at the end of map
17401 11  04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps pSnapInfo->SnapDevice: [] status [11]
17401 11  04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps iSnaps=0
17401 11  04/16 18:20:49 19593 CVMASnapHandler::mapSnaps() - From Snap Engine VolSnap:Status-[60500] Err-[60516:Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.]: [60500:].
17401 11  04/16 18:20:49 19593 CVMASnapHandler::mapSnaps() - Failed to map snaps. Err [60500:Error occurred in Snap Engine Error[]].
17401 11  04/16 18:20:49 19593 CVMASnapHandler::mountVolumes() - Request for mountVolumes Failed. Status [60500].
17401 11  04/16 18:20:49 #####  WORKER   [211977/0/0          ]  Failed to Mount Volume . Err [60516:Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.].

Things we have learnt, if communication between Client and Fileserver is impacted, it can cause the failure above. So be sure to validate that communication between the client and Fileserver is not affected.

Side note, if you wish to lock down the Simpana to use http vs https for the ManageONTAP communication you can use the registery key below;

s<arrayname>_NETAPPONTAPUSEADMINTRANSPORT

Defined under the iDataAgent branch of the simpana registry. Where the value of the key is http or https.

i.e. Example

vi /etc/CommVaultRegistry/Galaxy/Instance001/iDataAgent/.properties
Add line;

sLBnNNNnn-n_NETAPPONTAPUSEADMINTRANSPORT http

Now our communication will be http to FileServer LBnNNNnn-n, per regkey.

Commvault Simpana 9 Linux push deployment failure

Wanted to post about a very common Linux push deployment failure, infact I do believe this is applicable to other Unix platforms, so be sure to check this common thing against other Unix platforms.

An attempt to push Simpana 9 Linux File System iDA (and/or other iDA’s) will result in a failure like below, I’ll name the logs the information is obtained from so end readers can follow along.

[DistributeSoftware.log]

3000 ea4 04/09 19:29:18 27 CCvPatchesUpdateDBClient::InstallUnixSeed STDERR[]
3000 ea4 04/09 19:29:18 27 CCvPatchesUpdateDBClient::InstallUnixSeed STDOUT[*** UNIX SEEDING SETUP_ERROR 2042 detected and returned. ***
]

3000 9f8 04/09 19:29:28 27 ****** Now Check status of all clients ******
3000 9f8 04/09 19:29:28 27 Check status started for client [192.168.11.24][-1]
3000 9f8 04/09 19:29:58 27 Clients Failed:
3000 9f8 04/09 19:29:58 27 Client [-1] [192.168.11.24] [QINSTALL_BASE_PACKAGE_FAILED]
3000 9f8 04/09 19:29:58 27 *******************************************

3000 9f8 04/09 19:29:58 27 Job Failed.
3000 9f8 04/09 19:29:58 27 COMPLETE CALLED (JOB Status::FAIL), Job ID = 27

The most common cause for this failure is the client has a hosts file that is configured like so;

[/etc/hosts] – Bad version

127.0.0.1 localhost ldebx86-1.lab.libobk.so ldebx86-1

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Essentially your clients primary hostname and FQDN has been associated to a localhost entry. Modify the hosts file so that these hostname entries are defined against your primary interface that the machine has, as shown below;

[/etc/hosts] – Good version

127.0.0.1 localhost
192.168.11.24 ldebx86-1.lab.libobk.so ldebx86-1

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

This would have to be one of the most common issues for a push deployment.

If you attempted an interactive installation on this client when the hosts file was in the bad state, you would get the failure below during interactive install which will clearly indicate the problem.

Simpana 9.0.0 (BUILD84) Linux glibc2.11
——————————————————————————-
Physical Machine Host Name

We found one network interface available on your machine. We will associate it
with the physical machine being installed, and it will also be used by the
CommServe to connect to the physical machine. Note that you will be able to
additionally customize Datapipe Interface Pairs used for the backup data
traffic later in the Simpana Java GUI.

Please check the interface name below, and make corrections if necessary:

Physical Machine Host Name: ldebx86-1.lab.libobk.so

*** We found that /etc/hosts file associates name you provided
*** (ldebx86-1.lab.libobk.so) with loopback address.
*** Please check the /etc/hosts file and make corrections if necessary to
*** remove this association, or provide a new Host Name.

Press <ENTER> to try again …

If this post assists you, please leave comments.

Ubuntu Linux and Commvault Simpana 9 startup

As a follow up to the post yesterday about Debian Linux and Commvault Simpana 9 startup as found here.

Ubuntu Linux is also going to have the same issue. Since Ubuntu is based on Debian, it too has a runlevel 2 by default.

I installed Ubuntu 11.10 and out of the box it was on runlevel 2.

The same workaround I provided on the Debian post would be applicable here too.

EDIT:

Ubuntu 11.10 is not affected, as the installer detects the runlevel 2 correctly and sets the correct symlinks to the init scripts so that startup of Ubuntu will see the services start up fine.

Debian Linux and Commvault Simpana 9 startup

If you’ve recently installed Commvault Simpana 9 software/services onto a Debian Linux client, be sure check your runlevel and ensure that the software/services will start on the next reboot automatically.

By default simpana 9 software/services will be linked into runlevel 3 and runlevel 5, however on Debian the default runlevel is 2. (When your not running any X11 that is).

To confirm what runlevel the system runs by default you can check the inittab file and confirm an entry. Below is an example;

root@ldebx86-1:~# cat /etc/inittab | grep initdefault
id:2:initdefault:

And the same output from a CentOS 5.6 server (which is running X windows);

[root@lcosx86v56-1 ~]# cat /etc/inittab | grep initdefault
# 0 – halt (Do NOT set initdefault to this)
# 6 – reboot (Do NOT set initdefault to this)
id:5:initdefault:

From the output we can see that the top most one will default to runlevel 2 and the bottom one runlevel 5.

If you cat the inittab file you will see some comments indicating the runlevels;

# /etc/init.d executes the S and K scripts upon change
# of runlevel.
#
# Runlevel 0 is halt.
# Runlevel 1 is single-user.
# Runlevels 2-5 are multi-user.
# Runlevel 6 is reboot.

And from CentOS 5.6 server;

# Default runlevel. The runlevels used by RHS are:
# 0 – halt (Do NOT set initdefault to this)
# 1 – Single user mode
# 2 – Multiuser, without NFS (The same as 3, if you do not have networking)
# 3 – Full multiuser mode
# 4 – unused
# 5 – X11
# 6 – reboot (Do NOT set initdefault to this)
#

Post my Commvault Simpana 9 Linux File System iDA deployment, we go check for the Galaxy initscript being linked from the runlevel directories, and as you can see below, no script for runlevel 2, thus these services will not come up automaticaly upon reboot on this server.

root@ldebx86-1:/etc/rc3.d# ls -l *Galaxy*
lrwxrwxrwx 1 root root 16 Apr 8 19:07 S23Galaxy -> ../init.d/Galaxy
root@ldebx86-1:/etc/rc3.d# cd ..
root@ldebx86-1:/etc# cd rc2.d
root@ldebx86-1:/etc/rc2.d# ls -l *Galaxy*
ls: cannot access *Galaxy*: No such file or directory
root@ldebx86-1:/etc/rc2.d#

This can be resolved by of course creating the same symlink in the runlevel 2 directory and it will work next time as expected.

cd /etc/rc2.d
ln -s ../init.d/Galaxy S23Galaxy

Now we check again to confirm it created;

root@ldebx86-1:/etc/rc2.d# ls -l *Galaxy*
lrwxrwxrwx 1 root root 16 Apr 8 19:15 S23Galaxy -> ../init.d/Galaxy

Job done.

IBM Express-C

I was introduced by another work colleague to IBM Express-C. The reason for this introduction is that I needed a free version of DB2 which could be installed on Windows and/or Linux for a test environment. i.e. I wanted to simulate an environment where I could install the Commvault Simpana DB2 iDA to see how it’s configured, in addition perform backups and restores.

So if your in a situation like myself, I can highly recommend it.

IBM Express-C can be downloaded from here. In addition you can obtain a copy of a free e-book for it called “Getting started with Express-C” from here. The later item probably only of use if you want to develop on the database platform. Forums also exist as linked here.

Commvault Simpana 9 DB2 iDA documentation is found here.

Output of the commands below against a DB2 client are handy for troubleshooting purposes (command in bold);

  • db2level – obtain db2 revision/version information
  • db2ilist
  • db2 get instance – list current instance
  • db2 list db directory – list all DB’s in current instance (as defined in OS environment variable DB2INSTANCE)
  • db2 get dbm cfg – list all database manager configuration file values
  • db2 get db cfg for <DB> – list all DB parameters for <DB> and shows values.