Just a quick demo of installing MySQL on Windows Server 2008 R2, followed by the installation of Simpana 10 MySQL iDA.
We also enable Binary Logging on MySQL so we can perform Log backups, configure the MySQL instance in Simpana and run some backup tests.
You can watch the following video for the demo/example.
If you find this video of benefit, please leave feedback. Thanks.
I know it’s been a little while since I did one of these, but I suddenly found myself in a position where I could do one so I quickly put this together.
Unfortunately its in 3 parts, so if you watch the 3 clips below end to end it will make sense.
Part 1 of 3
Part 2 of 3
Part 3 of 3
You’ll notice I even made some mistakes in part 3, but I worked with it and used it to demonstrate how you can troubleshoot this type of issue.
If you find this valuable and/or have any questions, just drop me a comment. Always love to hear feedback.
I am going to assume that this is a test deployment and as such will expect that you have installed your CentOS 5.10 x64 Linux the way you want it, and I will follow on from that point on what I needed to perform to get the distribution release of PostgreSQL to work with Simpana 10 PostgreSQL iDA to perform a backup. Of course some assumed knowledge present.
CommVault Documentation references:
Recently had a problem with a MySQL iDA backup on a Linux platform. I was seeing a failure where we would get the error below (and per subject) included in the CommVault backup error.
ERROR 2013 (HY000): Lost connection to MySQL server during query
Upon further investigation it was determined that the error was being triggered during a MySQL query as performed by the MySQL iDA during backup attempt. When we attempted to do the same query manually against a MySQL connection we’d also get the same error. It was determined from the MySQL iDA logs that the database causing us the problem was called “prod”.
See the example below;
# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 4 Server version: 5.1.32-community-log MySQL Community Server (GPL) Type 'help;' or 'h' for help. Type 'c' to clear the buffer. mysql> use prod Database changed mysql> show table status; ERROR 2013 (HY000): Lost connection to MySQL server during query
As you can see the error was reproducible. Further investigation determined that the database was corrupt in some way, as we could see a large number of errors in the MySQL logs. Fix the corruption on the database and/or exclude it from the backups until resolved by the MySQL DBA.
Quick demo on how you can backup from command line using RMAN and still have this backup go via Simpana 10 and the Oracle iDA. This backup is a control file only backup, follow on from the GUI based example in my last post.
Simpana 10 documentation link covering this can be found here.
Contents of my backup.txt from screen cast on this post.
run {
allocate channel ch1 type 'sbt_tape'
PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,
ENV=(CvClientName=lcosx86v56-1,CvInstanceName=Instance001)";
backup current controlfile;
release channel ch1;
}
Follow on from my last post, this is now the configuration of the instance in the GUI/console.
Next up we create a subclient to just backup the control file of the orcl instance and of course run the job immediately to demo that it works.
Demo deployment of Simpana 10 Oracle iDA installation onto Linux client.
You’ll note I first check to see which user is running the pmon process for our database, in this case it’s “oracle”.
Next I change to the user oracle and determine the users primary group. As this group will be used for the installation. In this example it’s “oinstall”.
I’ll be posting another part to this, as the installation is merely the first step. Once installed you have to do some configuration from the Simpana console. i.e. create the oracle instance etc.
Commvault documentation covering the install above can be found here.
So you’ve just deployed the Simpana MySQL iDA and attempt your first backup, however get an error per below;
Error Code: [87:15] Description: MySQL Database: [~client~] Data Backup Failed with MySQL Error: [~mysqldump: Got error: 1227: Access denied; you need the RELOAD privilege for this operation when doing refresh ~].
This failure is talking about the MySQL user that you configured in the GUI for the MySQL iDA. Basically the account lacks “RELOAD” privilege, so you should go fix it up before trying again.
Came across an Simpana 10 SAP for Oracle iDA condition where the job would fail per below;
Error Code: [18:45] Description: Snap Initialization failed with an error: [Backup Index Initialization failed. Please check the logs.] Source: clientname, Process: backint_oracle
If you look into the $SAPDATA_HOMEsapbackup directory for the detail file associated with the failed backup you should also see something like below;
Rel 10.0.0(BUILD116) - CS=<commserve> CC=<clientname> retCode=1 bUpgradeFlag=208 upgradeTo9oTime=0 Got Mangled MediaAgent=<ma.libobk.so*ma*8400*8402> from Archive Manager creating BCD dir=</opt/simpana/iDataAgent/jobResults/2/115/417/> failed with retCode=<-1> BR0280I BRBACKUP time stamp: 2013-04-30 14.52.41 BR0279E Return code from '/sapmnt/OBK/exe/uc/rs6000_64/backint -u OBK -f backup -i /oracle/OBK/sapbackup/.belcejum.lst -t file -p /oracle/OBK/112_64/dbs/initOBK.utl -c': 2
This failure is indicative of a permissions error. i.e. Simpana installation path and/or jobResults (if been placed outside of the Simpana installation path) have permissions that restrict the SAP user from writing. Since the SAP for Oracle backups are ran via a Unix SAP user account.
Had an interesting condition in the last few days and really thought I would share, as it seemed like something really simple that could be checked by anyone.
Likewise, this could affect Oracle and/or File System SnapProtect jobs via NetApp.
Error: 62:2145
Description: The client machine initiator address is not visible on the file server. Please check the SAN or iSCSI connectivity between the file server and the client.
Upon reviewing this one, I identified the following conditions;
[ClOraAgent.log] 11502 1 04/16 18:20:07 19593 ManageONTAP::OpenConnection: Trying HTTP on port:[80] 11502 1 04/16 18:20:08 19593 ManageONTAP::OpenConnection: FileServer:[LBnNNNnn-n] Version:[NetApp Release 8.0.2P6 7-Mode: Fri Jan 27 14:48:08 PST 2012] 11502 1 04/16 18:20:09 19593 CVSnapEngineNetApp::cvso_snapDevices Check for snap clone dependences on volume:[volume_boot] on file server:[LBnNNNnn-n] returned success:[true] dependency on:[false]
First connection to the FileServer completes, however…
[CVMA.log] 17401 11 04/16 18:20:23 19593 ManageONTAP::OpenConnection: Trying HTTP on port:[80] 17401 11 04/16 18:20:23 19593 ManageONTAP::OpenConnection: Failed to get HTTP system version for server [LBnNNNnn-n]. Error:[-1][Can't connect to host (err=-8).]. This could be because 1. The server is powered down. 2. The server name is not resolvable from this host 3. Error with HTTP configuration. Please rectify the above errors (if any) and try again. 17401 11 04/16 18:20:23 19593 ManageONTAP::OpenConnection: Trying SSL on port:[443] 17401 11 04/16 18:20:23 19593 ManageONTAP::OpenConnection: Failed to get HTTPS system version for server[LBnNNNnn-n]. Error:[-1][Can't connect to host (err=-8).]. This could be because 1. The server is powered down. 2. The server name is not resolvable from this host 3. Error with HTTPS configuration. Please rectify the above errors (if any) and try again.
As you can see the next attempt fails… The final attempt will be to determine if the HBA’s from the client can be seen against the FileServer. Should we also get failures again connecting to the FileServer it will fall through with the error below.
[CVMA.log] 17401 11 04/16 18:20:49 19593 CvSnapNetApp::detectHBA() - No initiators logged in. Getting all initiators. 17401 11 04/16 18:20:49 19593 CvSnapNetApp::detectHBA() - Returning initiator ID list length:[1] iSCSI:[false] LoggedIn:[false] 17401 11 04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client. 17401 11 04/16 18:20:49 19593 ERROR:60516:Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client. 17401 11 04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps Printing status of clones at the end of map 17401 11 04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps pSnapInfo->SnapDevice: [] status [11] 17401 11 04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps iSnaps=0 17401 11 04/16 18:20:49 19593 CVMASnapHandler::mapSnaps() - From Snap Engine VolSnap:Status-[60500] Err-[60516:Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.]: [60500:]. 17401 11 04/16 18:20:49 19593 CVMASnapHandler::mapSnaps() - Failed to map snaps. Err [60500:Error occurred in Snap Engine Error[]]. 17401 11 04/16 18:20:49 19593 CVMASnapHandler::mountVolumes() - Request for mountVolumes Failed. Status [60500]. 17401 11 04/16 18:20:49 ##### WORKER [211977/0/0 ] Failed to Mount Volume . Err [60516:Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.].
Things we have learnt, if communication between Client and Fileserver is impacted, it can cause the failure above. So be sure to validate that communication between the client and Fileserver is not affected.
Side note, if you wish to lock down the Simpana to use http vs https for the ManageONTAP communication you can use the registery key below;
s<arrayname>_NETAPPONTAPUSEADMINTRANSPORT
Defined under the iDataAgent branch of the simpana registry. Where the value of the key is http or https.
i.e. Example
vi /etc/CommVaultRegistry/Galaxy/Instance001/iDataAgent/.properties Add line; sLBnNNNnn-n_NETAPPONTAPUSEADMINTRANSPORT http
Now our communication will be http to FileServer LBnNNNnn-n, per regkey.