Upgrading to VoltDB V1.3

written by Andrew Gent on April 12, 2011 with no comments

VoltDB V1.3 contains a number of new features, as well as changes and enhancements to existing features. In some cases, these enhancements involve changes to the behavior or the interface that existing users need to be aware of. The purpose of this article is to identify compatibility issues when upgrading from version 1.2 to 1.3 of VoltDB and explain how to plan accordingly.

Summary of New Features

Version 1.3 introduces several important new features and improvements to the VoltDB product, including:

  • Admin Mode — Admin mode allows operators to “pause” a database, stopping client activity, while sensitive administrative tasks are being completed, such as startup, shutdown, save, and restore. It is also possible to start a database in admin mode to ensure a restore or scripted data load can complete before any client activity begins.
  • Improved Export— The export function has been updated to improve robustness. Specifically, the export connector now writes overflow data to disk-based storage when the client receiver cannot keep up with the queue. This helps limit the memory usage of the export connector under load. It also allows the export queue to persist across server sessions.At the same time, the ExportToFile client has been improved to provide the following functions:
    • Rolling files, where new files are created periodically to improve manageability of the exported data.
    • Automatic detection of the cluster configuration and connecting to all nodes.
    • Resilience against dropped connections. If a cluster node fails, the export client now re-detects and reconnects to the new cluster configuration. This allows the export to persist across node failures (in a K-safe environment) or even across shutdown and restart of the cluster without user intervention.

    Finally, export is now restricted to export-only tables only. Previous versions allowed you to mix both export-only and normal data tables as sources for export, resulting in ambiguous code conditions (such as a delete operation on an exported table). Export tables and normal data tables are now distinct.

  • Improved Rejoin — In previous versions attempting to rejoin a node might fail if the cluster was under heavy load (particularly in the case of long-running transactions). The rejoin function has been improved to reduce the situations where rejoin can fail. It is still possible that the remaining database nodes are too busy to assist in the rejoin process. For these cases, using admin mode to “pause” client activity while rejoining is recommended.
  • Improved Deployment Management — The contents of the project definition and deployment configuration files have been reorganized to better match the separation of design and deployment. Specifically, control of the export and snapshot functions have moved from the project file to the deployment file, as have the specification of users and passwords. The deployment file itself has been restructured to better support these new functions.
  • Redesigned Enterprise Manager(Enterprise Edition only) — The Enterprise Manager has been redesigned from the ground up, providing a significantly improved user experience as well as new feaures:
    • Ability to manage multiple databases from a single instance of the Enterprise Manager
    • Improved performance graphs and status indicators
    • Programmable REST API
    • Separate database and server management

Preparing to Upgrade

Many of the new features, such as the improvements to rejoin and export, take effect without any changes to existing VoltDB applications. And for simple applications, no changes are required.

However, some of the changes in V1.3 may require existing applications to be modified to run. In particular, the reorganization of the project and deployment files require some minor edits to existing applications that utilize automatic snapshots, export, or network partition detection. The following sections explain what changes are necessary and how to prepare for the upgrade based on the VoltDB features that your application uses. The topics covered include:

  • General Upgrade Instructions
  • Automatic Snapshots
  • Export
  • Network Partition Detection
  • JSON Interface
  • Startup and Shutdown Procedures
  • Enterprise Manager (Enterprise Edition only)

General Upgrade Instructions

As with any upgrade, your VoltDB application catalog must be recompiled before it can run on the new version of VoltDB. If your application utilizes automatic snapshots, export, or network partition detection, you must edit your project and/or deployment files before recompiling. See the following sections for details.

In addition, VoltDB has a new feature, the VoltDB root path, that you should be aware of. The VoltDB root is a default location for any activity that involves writing to disk, including snapshots (both automatic snapshots and network partition snapshots) and export overflow. By default, VoltDB creates a directory under the current default path (./voltdbroot) for this purpose. If you operate in an environment with disk constraints, you may want to redirect disk-based activity to a specific device or location, using the <paths> tag in the deployment file. You can change the default root, specify specific paths for individual functions, or both. For example, the following deployment file sets the default path to /opt/dbroot and the path for export overflow to /tmp/exportoverflow:

      <voltdbroot path="/opt/dbroot" />
      <exportoverflow path="/tmp/exportoverflow/" />

Upgrading Automatic Snapshots

If your VoltDB database enables automatic snapshots, you must edit the project and deployment files before building your application with VoltDB V1.3 In previous versions, snapshots were enabled in the project definition file. In V1.3, automatic snapshots are controlled in the deployment file.

Before using an existing VoltDB project with the new version:

  1. Remove the <snapshots> tag from the project definition file.
  2. Add the <snapshot> tag and the attributes frequency, prefix, and retain to the deployment file.
  3. Optionally, specify a path for snapshots in the deployment file using <paths><snapshots> … </paths>.

Upgrading Export

If your existing VoltDB database application uses export, there are at least two changes you need to make before rebuilding with VoltDB V1.3:

  • Separate normal database tables from export-only tables.
  • Move control of export from the project definition file to the deployment file.

In V1.2, you could specify a mix of both export-only and normal tables as the source for export. In V1.3, export works on export-only tables only. You cannot export data directly from “live” data tables.

This means that if you used tables for both actual data storage and export, you will need to modify your schema to add an export-only version of the data table. You must also modify your stored procedures to perform an INSERT into the export-only copy whenever you do an INSERT or an UPDATE into the data table.

Within the project definition file, the declaration of export is simplified, since it now needs only to identify which tables are for export (and export only). For example:

         <table name="FLIGHT_EXPORT"/>
         <table name="CUSTOMER_EXPORT"/>

Then within the deployment file, you add the <export> tag to enable or disable the actual export function at runtime:

   <export enabled="true"/>

Finally, the ExportToFileClient receiver as been significantly improved and enhanced. The receiver now provides:

  • “Rolling” output files (creating new files on a user-settable cycle, which defaults to every 60 minutes)
  • Improved connectivity to the cluster
  • Resilience against node failures

Rather than manually connecting to each cluster node, the export client now queries one or more of the database nodes to determine the full cluster configuration and then creates connections to all nodes in the cluster. If a node fails, the export client repeats this discovery phase to re-establish connection with all active nodes.

As part of this redesign, both the command line for the export client and the format of the export files themselves have changed slightly. For example, the command line now lets you specify whether to connect to the client or admin port for export. In the files themselves, the file names have changed to include a starting transaction ID and the date format has been corrected.

So, to summarize, if an existing application uses export, you must:

  1. Make an export-only copy of any normal schema tables that previously were used for both storage and export.
  2. Update any stored procedures that inserted or updated those mixed use tables to add an INSERT into the export-only copy.
  3. Edit the project definition file to:
    • Remove the <connector> tag.
    • Remove the export-only attribute from the table tags (since all export tables are export-only now).
    • Replace any normal data tables in the export <table> tags with the corresponding export-only copy.
  4. Edit the deployment file to add the <export> tag to enable export.
  5. Review the new syntax for the export receiver command line and the format of the export output files to make sure they are consistent with any command scripts you are currently using.

Upgrading Network Partition Detection

If your existing VoltDB database uses network partition detection, there is no change to the function, but the tag to enable partition detection has been moved within the deployment file from a child of the <cluster> tag to its own peer element. Therefore, you must edit the deployment file to:

  1. Remove the <network-partition> tag from the <cluster> tag and make it a child of <deployment>.
  2. Remove the path attribute from the <snapshot> tag within <network-partition> and either add the appropriate <networkpartitionsnapshot> tag as a child of <paths> or accept the default path as described earlier.

Upgrading JSON Interface

If your existing application uses the JSON API to access the VoltDB database, you need to update your deployment configuration file for V1.3. The HTTPD port and the JSON interface are now disabled by default. You must explicitly enable them in the deployment file to have access to the JSON interface. Edit your deployment file and add the <httpd> and <jsonapi> tags, like so:

      <jsonapi enabled="true"/>

Upgrading Startup and Shutdown Procedures

All existing startup and shutdown procedures (such as @Shutdown and @Quiesce) work perfectly well with VoltDB V1.3 and don’t require any changes. However, the availability of admin mode may make you want to update your procedures.

The primary goal of admin mode is to pause client activity on the database cluster so you can perform administrative tasks while the database is in a known state. In particular, we encourage the use of admin mode to quiesce the system when performing a final snapshot before shutting down the cluster. Similarly, you can perform an orderly startup using the following steps:

  1. Start the database in admin mode (by adding <admin-mode adminstartup=”true”/> to the deployment file).
  2. Perform a snapshot restore.
  3. Call the @Resume system procedure to re-enable client traffic.

Upgrading the Enterprise Manager (Enterprise Edition only)

VoltDB Enterprise Edition contains a completely redesigned Enterprise Manager with many new features (not the least of which is the ability to manage multiple databases from a single interface). Because of the number of changes, the new Enterprise Manager is a replacement for — not an update to — the previous version. This means you must start the new Enterprise Manager as a fresh install.

To move an existing database from Enterprise Manager V1.2 to V1.3, you should take the following steps:

  1. Create a copy of your application source files.
  2. In the copy, make the changes to the schema and project definition file described in the preceding sections for compatibility with V1.3. (Note you do not need to modify the deployment file, since the Enterprise Manager builds the deployment file for you.)
  3. Recompile your application catalog.
  4. Start the new Enterprise Manager and create a new database definition, using the new catalog and appropriate settings from the deployment file.
  5. Create a snapshot of the existing V1.2 database and shutdown the cluster.
  6. Add the cluster nodes to the new database in the V1.3 Enterprise Manager and start the database, making sure to start in admin mode.
  7. Outside of the Enterprise Manager, manually copy the snapshot from Step #5 to one of the cluster nodes and restore the V1.2 data to the new V1.3 database using a call to the @SnapshotRestore system procedure (using the admin port).
  8. Once the restore is complete, Use the Enterprise Manager to take a V1.3 snapshot of the data and “unpause” the database and enable client activity.

Andrew Gent