My Blog

Category: Database

  • ERROR 2013 (HY000): Lost connection to MySQL server during query

    Recently had a problem with a MySQL iDA backup on a Linux platform. I was seeing a failure where we would get the error below (and per subject) included in the CommVault backup error.

    ERROR 2013 (HY000): Lost connection to MySQL server during query

    Upon further investigation it was determined that the error was being triggered during a MySQL query as performed by the MySQL iDA during backup attempt. When we attempted to do the same query manually against a MySQL connection we’d also get the same error. It was determined from the MySQL iDA logs that the database causing us the problem was called “prod”.

    See the example below;

    # mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 4
Server version: 5.1.32-community-log MySQL Community Server (GPL)

Type 'help;' or 'h' for help. Type 'c' to clear the buffer.

mysql> use prod
Database changed
mysql> show table status;

ERROR 2013 (HY000): Lost connection to MySQL server during query

    As you can see the error was reproducible. Further investigation determined that the database was corrupt in some way, as we could see a large number of errors in the MySQL logs. Fix the corruption on the database and/or exclude it from the backups until resolved by the MySQL DBA.

  • Simpana 10 – Oracle iDA – Backup current controlfile via RMAN command line

    Quick demo on how you can backup from command line using RMAN and still have this backup go via Simpana 10 and the Oracle iDA. This backup is a control file only backup, follow on from the GUI based example in my last post.

    Simpana 10 documentation link covering this can be found here.

    Contents of my backup.txt from screen cast on this post.

    run {
allocate channel ch1 type 'sbt_tape'
PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,
ENV=(CvClientName=lcosx86v56-1,CvInstanceName=Instance001)";
backup current controlfile;
release channel ch1;
}

     

  • Simpana 10 – Oracle iDA – Configuration – Instance creation and GUI based backup control file test

    Follow on from my last post, this is now the configuration of the instance in the GUI/console.

    Next up we create a subclient to just backup the control file of the orcl instance and of course run the job immediately to demo that it works.

  • Simpana 10 Oracle iDA installation onto Linux client

    Demo deployment of Simpana 10 Oracle iDA installation onto Linux client.

    You’ll note I first check to see which user is running the pmon process for our database, in this case it’s “oracle”.

    Next I change to the user oracle and determine the users primary group. As this group will be used for the installation. In this example it’s “oinstall”.

    I’ll be posting another part to this, as the installation is merely the first step. Once installed you have to do some configuration from the Simpana console. i.e. create the oracle instance etc.

    Commvault documentation covering the install above can be found here.

  • MySQL backup via Simpana fails with Error Code 87:15

    So you’ve just deployed the Simpana MySQL iDA and attempt your first backup, however get an error per below;

    Error Code: [87:15]

Description: MySQL Database: [~client~] Data Backup Failed with MySQL Error: [~mysqldump: Got error: 1227: Access denied; you need the RELOAD privilege for this operation when doing refresh ~].

    This failure is talking about the MySQL user that you configured in the GUI for the MySQL iDA. Basically the account lacks “RELOAD” privilege, so you should go fix it up before trying again.

  • Simpana 9 Oracle or File System SnapProtect Failure via NetApp

    Had an interesting condition in the last few days and really thought I would share, as it seemed like something really simple that could be checked by anyone.

    Likewise, this could affect Oracle and/or File System SnapProtect jobs via NetApp.

    Error: 62:2145

    Description: The client machine initiator address is not visible on the file server. Please check the SAN or iSCSI connectivity between the file server and the client.

    Upon reviewing this one, I identified the following conditions;

    [ClOraAgent.log]
11502 1   04/16 18:20:07 19593 ManageONTAP::OpenConnection: Trying HTTP on port:[80]
11502 1   04/16 18:20:08 19593 ManageONTAP::OpenConnection: FileServer:[LBnNNNnn-n] Version:[NetApp Release 8.0.2P6 7-Mode: Fri Jan 27 14:48:08 PST 2012]
11502 1   04/16 18:20:09 19593 CVSnapEngineNetApp::cvso_snapDevices Check for snap clone dependences on volume:[volume_boot] on file server:[LBnNNNnn-n] returned success:[true] dependency on:[false]

    First connection to the FileServer completes, however…

    [CVMA.log]
17401 11  04/16 18:20:23 19593 ManageONTAP::OpenConnection: Trying HTTP on port:[80]
17401 11  04/16 18:20:23 19593 ManageONTAP::OpenConnection: Failed to get HTTP system version for server [LBnNNNnn-n]. Error:[-1][Can't connect to host (err=-8).]. This could be because 1. The server is powered down. 2. The server name is not resolvable from this host 3. Error with HTTP configuration. Please rectify the above errors (if any) and try again.
17401 11  04/16 18:20:23 19593 ManageONTAP::OpenConnection: Trying SSL on port:[443]
17401 11  04/16 18:20:23 19593 ManageONTAP::OpenConnection: Failed to get HTTPS system version for server[LBnNNNnn-n]. Error:[-1][Can't connect to host (err=-8).]. This could be because 1. The server is powered down. 2. The server name is not resolvable from this host 3. Error with HTTPS configuration. Please rectify the above errors (if any) and try again.

    As you can see the next attempt fails… The final attempt will be to determine if the HBA’s from the client can be seen against the FileServer. Should we also get failures again connecting to the FileServer it will fall through with the error below.

    [CVMA.log]
17401 11  04/16 18:20:49 19593 CvSnapNetApp::detectHBA() - No initiators logged in. Getting all initiators.
17401 11  04/16 18:20:49 19593 CvSnapNetApp::detectHBA() - Returning initiator ID list length:[1] iSCSI:[false] LoggedIn:[false]
17401 11  04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.
17401 11  04/16 18:20:49 19593 ERROR:60516:Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.
17401 11  04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps Printing status of clones at the end of map
17401 11  04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps pSnapInfo->SnapDevice: [] status [11]
17401 11  04/16 18:20:49 19593 CVSnapEngineNetApp::cvso_mapSnaps iSnaps=0
17401 11  04/16 18:20:49 19593 CVMASnapHandler::mapSnaps() - From Snap Engine VolSnap:Status-[60500] Err-[60516:Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.]: [60500:].
17401 11  04/16 18:20:49 19593 CVMASnapHandler::mapSnaps() - Failed to map snaps. Err [60500:Error occurred in Snap Engine Error[]].
17401 11  04/16 18:20:49 19593 CVMASnapHandler::mountVolumes() - Request for mountVolumes Failed. Status [60500].
17401 11  04/16 18:20:49 #####  WORKER   [211977/0/0          ]  Failed to Mount Volume . Err [60516:Initiators on host:[libobk-client] Are not logged into file server:[LBnNNNnn-n]. Please check the SAN or iSCSI connectivity between the file server and the client.].

    Things we have learnt, if communication between Client and Fileserver is impacted, it can cause the failure above. So be sure to validate that communication between the client and Fileserver is not affected.

    Side note, if you wish to lock down the Simpana to use http vs https for the ManageONTAP communication you can use the registery key below;

    s<arrayname>_NETAPPONTAPUSEADMINTRANSPORT

    Defined under the iDataAgent branch of the simpana registry. Where the value of the key is http or https.

    i.e. Example

    vi /etc/CommVaultRegistry/Galaxy/Instance001/iDataAgent/.properties
Add line;

sLBnNNNnn-n_NETAPPONTAPUSEADMINTRANSPORT http

    Now our communication will be http to FileServer LBnNNNnn-n, per regkey.

  • IBM Express-C

    I was introduced by another work colleague to IBM Express-C. The reason for this introduction is that I needed a free version of DB2 which could be installed on Windows and/or Linux for a test environment. i.e. I wanted to simulate an environment where I could install the Commvault Simpana DB2 iDA to see how it’s configured, in addition perform backups and restores.

    So if your in a situation like myself, I can highly recommend it.

    IBM Express-C can be downloaded from here. In addition you can obtain a copy of a free e-book for it called “Getting started with Express-C” from here. The later item probably only of use if you want to develop on the database platform. Forums also exist as linked here.

    Commvault Simpana 9 DB2 iDA documentation is found here.

    Output of the commands below against a DB2 client are handy for troubleshooting purposes (command in bold);

    • db2level – obtain db2 revision/version information
    • db2ilist
    • db2 get instance – list current instance
    • db2 list db directory – list all DB’s in current instance (as defined in OS environment variable DB2INSTANCE)
    • db2 get dbm cfg – list all database manager configuration file values
    • db2 get db cfg for <DB> – list all DB parameters for <DB> and shows values.

  • Recover MySQL root password

    Last week I found myself in one of those situations where a machine I setup nearly 2 years ago required some additional MySQL databases created and for the life of me I couldn’t find where I had recorded the MySQL root password.

    I figured setting a new one should be possible, however I’ve never had to do it. After about literally one search on Google and a few short clicks later I had found what I needed. I ended up at the following site here.

    Process looked easy enough so I scheduled an outage window, since the MySQL server in question was being used for a Confluence Wiki.

    Sure enough the process worked fine, I now had set a new MySQL root password and brought the services back online.

    I of course immediately added the password into my password safe, so I don’t have to do it ever again.