Alternate Copy and Backup Method for Large Scrutinizer Databases

 

Some users with large amounts of data being stored by Scrutinizer may find that using MySQL Administrator for backups is extremely slow and CPU intensive.    MySQL Administrator takes a very conservative (safe) approach to doing the backup.  This is reasonable for a general purpose backup tool with no application knowledge of what is being backed up.  For Scrutinizer the additional overhead can degrade performance and is not necessary for a successful backup.

What follows is a description of the recommended method to implement backups for large networks..

 

Backing up the data

The history and configuration data for Scrutinizer is stored in \Scrutinizer\mysql\data.  Users are not recommended to backup \Scrutinizer\mysql\data directly.  The recommended solution is to copy the data directory to a backup location and optionally run the backup on that directory.


Copying the data:

Use the  windows command xcopy to hotcopy the MySQL data files to the backup directory.

C:\>xcopy /D /E /I /Y /Z "\SCRUTINIZER\mysql\data" “\SCRUTINIZER_backup

The above command will only copy files that have changed.  The initial copy will need to duplicate the entire DB. Subsequent runs will only copy files that have been modified since the previous run.

 

This technique relies on the fact that after a known period of time the historical tables in Scrutinizer are static (read only).  If the tables aren't being updated, it is safe to just copy the data tables.  Even when tables are being updated MySQL MyISAM tables are very robust and seldom corrupt.  The following table shows how long each table is written to before it becomes read only:


Table Type

duration

1m

1 hour

5m

5 hours

30m

30 hours

2h

5 days

12h

30 days

1d

60 days

1w

420 days (60 weeks)



Up through the 30m tables, they become read only in 30 hours or less.  Above 30m tables, a little knowledge about when tables are written to can help to ensure that as much current data is saved as possible.

·        2h are written to starting at the top of even hours

·        12h are written to starting after 12 and midnight

·        1d are written to starting after midnight

·        1w are written after midnight on Saturday

 

By picking a time when little is going on (e.g. 1pm Sunday or 1am Monday) it is possible to safely backup of your data without needing MySQL Administrator.


Caveats

Incrementally copying files with xcopy will not remove tables from the backup directory that have been dropped by Scrutinizer.  To get a list of files to delete from the backup directory simply reverse the xcopy command and add the /L to only print results.

xcopy /D /E /I /Y /Z /L "\SCRUTINIZER_backup" "\SCRUTINIZER\mysql\data"