One of the major feature updates to Qube 7.0 is the switch of the supervisor's database server from MySQL to PostgreSQL. Accordingly, when upgrading from a pre-7.0 version (such as 6.10-0a) to 7.0 or above, there are extra manual steps that must be taken outside of the Qube Installer (or the RPM/MSI/PKG installer, if you chose to bypass the QubeInstaller), particularly if you require that the old data from MySQL be transferred to PostgreSQL. This step-by-step document will guide you through the straightforward process.
Upgrading the supervisor and transferring the data is essentially a five-step process, and the order is critical:
- Stop all activities on the current farm
- Export data from MySQL to csv files on disk
- Upgrade the supervisor
- Import data into PostgreSQL from the csv files
- Enable activities on the new supervisor
Step One: Stop All Activities on the Current Farm
You will want to first stop all current farm activities before doing the upgrade of the Supervisor. Run the following commands on a command prompt with a qube administrator account to prevent the supervisor from dispatching jobs, and stop accepting new job submissions from users:
You will also want stop workers from picking up new work. If you want to allow workers to finish up their current frames, do:
If you'd rather stop running jobs immediately, do:
Then wait for all activities to stop.
Step Two: Export Data from MySQL to .csv (Comma-separated values) files
In this step, you'll be using a script that we provide to dump MySQL data to .csv files into a folder on disk.
Make sure that the MySQL server is running, and that you can connect to it using the mysql client, and that your qube table version is at 37.
If you haven't changed the database administrator user and password, you should be able to do the following on a command prompt to confirm that the MySQL server is running:
Make sure that the above command works and returns:
If you get something less than 37 returned by the above command, it means that your current Qube supervisor version is older than 6.10-0, and that you need to update your MySQL database tables first, before you can upgrade the database schema. To do so:
Download the "
upgrade_supervisor" program suitable for your supervisor platform from http://repo.pipelinefx.com/downloads/pub/db_migration_tools/
On a command prompt, run the
upgrade_supervisor programthat you just downloaded.
- On Windows, you will need to unzip the upgrade_supervisor_WIN32-6.1-x64.zip file first, and run the upgrade_supervisor.bat file found in the unzipped folder.
- Check that there weren't any critical errors reported by
- Check that the version is now indeed updated to 37, by running the
mysql -u root -e 'SELECT * FROM qube.tableversion'command again
Choose a destination folder on your supervisor for the MySQL csv files. Make sure that your user and the mysql server process both have write permission to this folder and all its parent folders, and that the volume is sufficiently large. Also note that a faster disk, such as an SSD, will help speed up the export/import process.
- Download the export_data_from_mysql.py and fix_mysql_column_orders.sql scripts from http://repo.pipelinefx.com/downloads/pub/db_migration_tools/ and copy them into the destination folder.
On a command prompt, go to the destination folder, and run
export_data_from_mysql.py. Running it without any argument will create a subfolder in the current directory named "
qube_mysqldump" and dump all files into it.
You may override the dump subfolder and DB username, password, and mysql install location. Run "
export_data_from_mysql.py -h" to see the list of options.
Sit back. This process can take a long time to complete, depending on how many jobs you have on the system.
Once the process completes, make sure there were no errors reported on the terminal. Also have a look a the dump directory to confirm that there is a subfolder "
qube" and a bunch of subfolders like "
- Take a note of the dump directory location, and proceed to the next step, "Upgrade the Supervisor".
Proceed with the upgrade of the supervisor software. Using the QubeInstaller is recommended, but you can also run the individual installer packages (RPMs, DEBs, MSIs, or PKGs), should you choose.
Step Three: Upgrade the Supervisor
See Upgrading Qube! for details, but come back here after upgrading the supervisor software.
Step Four: Import Data into PostgreSQL from the .csv files
Importing the previously exported data
Once you upgrade the supervisor, you are ready to import data into the new PostgreSQL server.
Make sure that PostgreSQL server is running, and accepting connections:
Note that this should return:
- On a command prompt, go to the folder where you ran the export script earlier. This should be the parent folder of the "qube_mysqldump" folder, by default.
import_data_into_pgsql.pyscript to import data from the csv files that were generated earlier.
- Sit back. This process will also take some time to complete, although it should be significantly faster than the export.
- Make sure there weren't any errors reported on the terminal.
Step Five: Enable Activities on the New Supervisor
Run the following commands to enable the new supervisor to accept new jobs and start dispatching jobs to workers
You'll also need to unlock the workers you want to start using again. If you'd like to unlock all workers, then do:
Congratulations, you are done. Enjoy the new ride!