Documentation

Swapping is an operating system feature that optimizes memory usage when running multiple processes. However, memory is a critical component of the VoltDB server process. Any contention for memory, including swapping, will have a very negative impact on performance and functionality.

We recommend using dedicated servers and disabling swapping when running the VoltDB database server process. Use the swapoff command to disable swapping on Linux systems. If swapping cannot be disabled for any reason, you can reduce the likelihood of VoltDB being swapped out by setting the kernel parameter vm.swappiness to zero.

There is an issue where, under certain conditions, the use of TCP segmentation offload (TSO) and generic receive offload (GRO) can cause nodes to randomly drop out of a cluster. The symptoms of this problem are that nodes timeout — that is, the rest of the cluster thinks they have failed — although the node is still running and no other network issues (such as a network partition) are the cause.

Disabling TSO and GRO is recommended for any VoltDB clusters that experience such instability. The commands to disable offloading are the following, where N is replaced by the number of the ethernet card:

ethtool -K ethN tso off
ethtool -K ethN gro off

Note that these commands disable offloading temporarily. You must issue these commands every time the node reboots.

Previously all VoltDB system procedures operated as database transactions. That is, system procedures executed as multi-partition transactions with two consequences:

Since statistical procedures do not change the state of the database, they are now executed separately from database transactions. This applies to the system procedures @Statistics, @SystemInformation, and @SystemCatalog. The consequences of this change are that the statistical system procedures have significantly less impact on database performance and, although the resulting statistics are not transactionally exact, they are an accurate reflection of overall database performance at the time the procedure executes. More importantly, it is now possible to get information about database operations even when an errant stored procedure or server is blocking other database operations.

VoltDB now supports outer and explicit joins. Previously, all joins were implicitly inner joins (using a comma-separated list of table names). You can now use explicit join syntax to specify either inner or outer joins on specific columns using the ON or USING clause. The new syntax for the SELECT statement is as follows:

 Select-statement  [{set-operator}  Select-statement ]  ... 
 Select-statement: 
  SELECT  [ TOP integer-value ]
 { *  |   [ ALL  |   DISTINCT ] { column-name  |   selection-expression } [AS alias] [,...] }
  FROM    table-reference    [ join-clause ]... 
 [WHERE  [NOT] boolean-expression   [ {AND | OR}  [NOT] boolean-expression]...]
 [clause...]
 table-reference: 
 { table-name  |   view-name } [AS alias]
 join-clause: 
  ,table-reference 
 [ INNER | {LEFT | RIGHT} [OUTER] ]  JOIN  {table-reference} [join-condition]
 join-condition: 
  ON conditional-expression 
  USING (column-reference [,...]) 
 clause: 
  ORDER BY  {  column-name  |   alias  } [ ASC  |   DESC ] [,...]
  GROUP BY  {  column-name  |   alias  } [,...]
  LIMIT  { integer-value [OFFSET row-count]  |   ALL }
 set-operator: 
  UNION  [ALL]
  INTERSECT  [ALL]
  EXCEPT 

For this initial release, outer joins support the joining of two tables only. Explicit and implicit inner joins support joining more than two tables.

Web Studio allows you to invoke stored procedures on a running database, entering procedure arguments as text. Previously, Web Studio had some difficulty parsing strings containing multiple embedded quotation marks. Argument parsing has been enhanced to support arbitrarily complex strings.

This release includes a number of improvements to the planning of SQL statements. In particular, improvements have been made to how indexes are applied under various conditions.

For systems with multiple network interfaces, the VoltDB Enterprise Manager automatically listens to the specified port on all interfaces. You can now tell Enterprise Manager to use only one interface by specifying the IP address on the command line with the -a flag. For example, the following command line forces Enterprise Manager to listen to port 8989 on the network interface 10.12.14.16:

$ ./enterprise_manager.sh -p 8989 -a 10.12.14.16

To enable security using VoltDB Enterprise Manager, you must check the "Enable Security" checkbox on the Create/Edit Database dialog and you must upload a deployment file containing user account definitions. In previous versions, any subsequent changes to the database configuration would silently remove the user definitions. You had to upload the user definitions every time you modified the configuration to keep the user definitions active. This bug has been fixed. Enterprise Manager now maintains the last uploaded user definitions when other configuration changes are made. To change the user definitions after they have been uploaded you must explicitly upload a new deployment file containing only the user definitions that you want to use.

Previously, the @SnapshotStatus system procedure only returned results for the server processing the request, not all nodes in the cluster. Starting with this release, a call to @SnapshotStatus returns status information for all nodes in the cluster.

By default, when enabling security the deployment file contains definitions of users and passwords. These definitions are included as plain text. If your environment or operating procedures require the deployment file to be secured against observation, you can preprocess the file to obscure the passwords using the voltdb mask command. These processed deployment files can then be used to start the VoltDB database. The syntax for the voltdb mask command is as follows:

$ voltdb mask deployment-file [new-deployment-file]

If you specify one file name, the file is processed in place. If you specify two file names, the first is used as the input file and the second as the output file. The processed deployment file contains an additional attribute on the <user> tag, plaintext="false" indicating to the VoltDB server that the passwords have been obscured.

When VoltDB compiles the database schema, it displays a summary of the stored procedure queries as part of the compiler output. This output includes annotations identifying the type of procedure (read vs. write and single vs. multi-partitioned), potentially slow queries due to a sequential table scan, and possible dangers introduced by non-deterministic output. Several of these annotations have been changed to improve readability. The read/write annotation has been changed to [READ] and [WRITE] and [Seq] has been changed to [TABLE SCAN].

When replacing the catalog on an existing database, the VoltDB Enterprise Manager displays a list of differences between the two catalogs to help you verify the changes are correct. Previously, the report only identified added or removed tables and procedures. The report now also identifies tables and procedures that have been modified, changes to security settings, and the requirements for the update. That is, whether the database must be stopped and restarted or the update can be done while the database is running.

This release fixes a critical bug where updating the schema "on the fly" could cause the database to stop and make the command logs unusable for recovery. The issue only affects application catalogs that change a unique index by either adding or removing columns from the unique index (or primary key). Because of the seriousness of this bug, all users are strongly urged to upgrade to the latest version at their earliest possible convenience.

Until you upgrade, avoid updating the database catalog while the database is running if the update changes unique indexes. Note this issue only affects live updates. You can still update the schema using a maintenance window by performing the following sequence of commands: save, shutdown, create and restore.

It is now possible to make many more changes to the schema "on the fly" — while the database is running — than in previous versions. You can now add, delete, or modify columns and indexes as well as tables. The only remaining limitations are that you cannot add a new constraint to an existing column or index and you cannot modify the definition of an existing view. (However, you can add and remove views.)

With this new functionality, the ability to perform line schema changes becomes a commercial feature. In other words, starting with V3.2 the @UpdateApplicationCatalog system procedure and the voltadmin update command are available in the Enterprise Edition only.

In addition to new capabilities in live schema updates, the updates themselves have been improved in terms of performance and resilience to node failures. Updates to large catalogs, especially those with many stored procedures, are significantly faster than in previous versions. In addition, the update process has been hardened to avoid problems if a node fails during an update.

Warning concerning previous versions: If you perform catalog updates using previous versions of VoltDB, it is recommended that you take a snapshot prior to and immediately after the update as a precaution in case of database failure. Command logs and other durability features may be in an inconsistent state following a catalog update and before a new snapshot is taken. This issue is corrected in the current release.

It is possible for a snapshot restore (using the voltadmin restore command or @SnapshotRestore system procedure) to partially succeed. For example, if the schema has changed since the snapshot was created or some snapshot files are missing, only part of the data may be restored.

Previously, this condition was reported as success with details about what succeeded and what failed in the VoltTable response. To avoid confusion with a complete and successful restore, this function has been changed to report an error, OPERATIONAL_FAILURE, if one or more tables or partitions are not restored.

The default heartbeat timeout has been increased from 10 seconds to 90 seconds. In production, it has been found that network and process contention issues can make VoltDB mistakenly exceed the 10 second timeout, resulting in still functioning nodes being timed out. As a result, the default has been increased.

A consequence of this change is that it will now take longer for actual node failures to be recognized by the rest of the cluster, potentially stalling transactions until the failure is detected. You can change the timeout period using the <heartbeat> element in the deployment file or in the configuration dialog of the VoltDB Enterprise Manager. See the appendix on server configuration options in the VoltDB Management Guide for details.

The following issues have been fixed:

V3.1 includes production-ready support for Database Replication (DR). The changes to transaction coordination introduced in V3.0 required a significant rewrite of the DR functionality. As a result, the initial V3.0 release included a beta release of DR only. This release returns DR to full operational status. Users who stayed with earlier VoltDB releases to use DR in production are now encouraged to upgrade to V3.1.

V3.1 introduces a new function, CAST(), that lets you convert the datatype of an expression. CAST() can be very useful when performing operations on column values or function results that are not in the appropriate datatype. For example, you can use CAST() to convert the results of the FIELD() function, which always returns a string, to a numeric datatype. See Using VoltDB for more information.

V3.1 also includes additional functions for traversing JSON data. The ARRAY_LENGTH() and ARRAY_ELEMENT() functions return the number of elements in a JSON array and a specific element from that array, respectively. These functions can be combined with other functions to retrieve specific items within the JSON structure. For example, to following SQL fragment retrieves the first element of an array in the JSON field named "options" from the column JSONDATA:

SELECT ARRAY_ELEMENT(FIELD(JSONDATA,"options"),0) AS first_option

See Using VoltDB for more information on using these functions.

The Log4J messages generated by VoltDB have been reorganized. Messages previously logged under the JOIN component are now logged under REJOIN and a new logger, SNAPSHOT, has been added for logging snapshot activity.

VoltDB 3.0 includes a new transaction coordination architecture that reduces latency and improves transaction throughput. Some of the benefits of VoltDB 3.0 include lower overall latency, higher throughput, and reduced dependency on NTP.

V3.0 integrates much of the database configuration information into the schema, eliminating the need for a separate project definition file. Features that can now be defined in data definition language (DDL) include:

In addition, several new command line tools make the development process simpler and more consistent. New shell commands for compiling, starting, and managing VoltDB databases include voltdb, voltadmin and sqlcmd.

In addition to improvements in transaction coordination and the development process, a number of new functional capabilities have been added. New features include:

Users of earlier versions of VoltDB should be aware of the following changes in behavior that could impact their applications:

To ensure complete and accurate restoration of a database, recovery using command logs can only be performed to a cluster with the same number of unique partitions as the cluster that created the logs. If you restart and recover to the same cluster with the same deployment options, there is no problem. But if you change the deployment options for number of nodes, sites per host, or K-safety, recovery may not be possible.

For example, if a four node cluster is running with four sites per host and a K-safety value of one, the cluster has two copies of eight unique partitions (4 X 4 / 2). If one server fails, you cannot recover the command logs from the original cluster to a new cluster made up of the remaining three nodes, because the new cluster only has six unique partitions (3 X 4 / 2). You must either replace the failed server to reinstate the original hardware configuration or otherwise change the deployment options to match the number of unique partitions. (For example, increasing the site per host to eight and K-safety to two.)

VoltDB reserves the subfolder "segments" under the command log directory for storing the actual command log files. Do not add, remove, or modify any files in this directory. In particular, do not set the command log snapshot directory to a subfolder "segments" of the command log directory, or else the server will hang on startup.

Using the VoltDB Enterprise Manager, if a replica database was started with command logging, then stopped (intentionally or by accident), the Enterprise Manager cannot restart the database as a normal database using the recover action to reinstate the database's previous state. The Enterprise Manager can restore from a snapshot.

If you want to use the Enterprise Manager to stop a replica and restart it as a normal database, the recommended procedure is:

Note that this limitation is specific to the Enterprise Manager. Failed replica databases can be recovered manually using the command line.

If your SELECT statement includes two aggregates of the same column, VoltDB will return only one column value as part of the result set. For example, if your SQL statement is SELECT COUNT(ColumnA), COUNT(ColumnA) FROM MyTable, the resulting VoltTable will have only one column value per row. The workaround is to either not request the same value twice or aggregate on different columns (for example, SELECT COUNT(Column10), COUNT(Column2)).

In SELECT statements, the selection expression can include columns, aggregate functions (such as COUNT), or arithmetic expressions involving columns (such as Col1 + Col2). However, they cannot include arithmetic involving aggregate functions (such as SELECT SUM(Price) + 2). Using aggregate functions in an arithmetic expression results in an error when you compile the application catalog.

The workaround is to use the aggregate in the SELECT statement and then perform the additional arithmetic on the result set either in the stored procedure or in the client application after the transaction completes.

Use of SELECT DISTINCT is supported for a single column (such as SELECT DISTINCT Price FROM Inventory). However, using DISTINCT with multiple columns or arithmetic expressions is not currently supported. For example, the following SELECT DISTINCT statements should not be used:

SELECT DISTINCT Price, Discount FROM Inventory
SELECT DISTINCT (Price - Discount) FROM Inventory

VoltDB currently intercepts assertions as part of its handling of stored procedures. Attempts to use assertions in stored procedures for debugging or to find programmatic errors will not work as expected.

When creating a view of a partitioned table, using the CREATE VIEW statement, the column on which the table is partitioned must be included in the GROUP BY clause. If not, queries of the view using SELECT ... FROM can give incorrect results. Note that VoltDB does not detect or warn you of this condition. Be sure to include the partitioning column in the GROUP BY clause when creating a view on a partitioned table.

There is a problem with how the math library used to build the C++ client library handles large decimal values on 32-bit operating systems. As a result, the C++ library cannot serialize and pass Decimal datatypes reliably on these systems.

Note that the C++ client interface can send and receive Decimal values properly on 64-bit platforms.

To ensure proper recovery on startup, either from command logs or the last database snapshot, make sure all snapshot files — or at least complete subsets of the snapshot files — are available on the nodes of the cluster. If you delete or move snapshot files (for example, copying all snapshot files to a single node) be sure to keep all of the files for each node together. Do not selectively delete or move individual files or else the recovery may fail.

You can modify the database schema of an existing database by recompiling the application catalog with the changes, saving the current database contents, restarting with the modified catalog and then restoring the contents. However, if the modified catalog adds a column with the default value specified as NULL, the database will fail when you attempt to restore a snapshot created using the old catalog. The workaround, for the time being, is to not specify a default value for the new column, specify a default other than NULL, or save the database contents as CSV files and use the csvloader utility to load each table.

You can add and remove tables on the fly using voltadmin update or the @UpdateApplicationCatalog system procedure. To modify individual tables, such as adding or removing columns, you must save a snapshot, create a new database with the modified catalog, and then restore the snapshot. However, this latter method does not currently support changing a column from partitioned to replicated, or the reverse.

The workaround for changing a partitioned table to replicated or a replicated table to partitioned is the following:

Normally, manual snapshots (those created with the Take a Snapshot button) are copied to the management server. However, if automated snapshots are also being created and copied to the management server, it is possible for an automated snapshot to override the manual snapshot.

If this happens, the workaround is to turn off automated snapshots (and their copying) temporarily. To do this, uncheck the box for copying snapshots, set the frequency to zero, and click OK. Then re-open the Edit Snapshots dialog and take the manual snapshot. Once the snapshot is complete and copied to the management server (that is, the manual snapshot appears in the list on the dialog box), you can re-enable copying and automated snapshots.

When the Enterprise Manager starts, it unpacks files that the web server uses into a subfolder of the /tmp directory. It does not delete these files when it stops. Under normal operation, this is not a problem. However, if you upgrade to a new version of the Enterprise Edition, files for the new version become intermixed with the older files and can result in the Enterprise Manager starting databases using the wrong version of VoltDB. To avoid this situation, make sure these temporary files are deleted before starting a new version of VoltDB Enterprise Manager.

The /tmp directory is emptied every time the server reboots. So the simplest workaround is to reboot your management server after you upgrade VoltDB. Alternately, you can delete these temporary files manually by deleting the winstone subfolders in the /tmp directory:

$ rm -vr /tmp/winstone*

When upgrading VoltDB Enterprise Edition, please note that the configuration files for the Enterprise Manager are not upwardly compatible. New product features may make existing database and/or deployment definitions unusable. It is always a good idea to delete existing configuration information before upgrading. You can delete the configuration files by deleting the ~/.voltdb directory. For example:

$ rm -vr ~/.voltdb

In the past, it was possible to run two (or more) databases on a single physical server by defining two logical servers with the same IP address and making the ports for each database unique. However, as a result of internal optimizations introduced in VoltDB 2.7, this technique no longer works when using the Enterprise Manager.

We expect to correct this limitation in a future release. Note that it is still possible to start multiple databases on a single server manually using the VoltDB shell commands.

The export-to-Hadoop client uses Hadoop and the Sqoop import application from Cloudera to automate the export of data from VoltDB to Hadoop. It is the user's responsibility to ensure that Hadoop (and Sqoop) are installed, configured, and running properly before using the export-to-Hadoop client. This includes making sure that the environment variables HADOOP_HOME and SQOOP_HOME are defined and that all of the required Hadoop and Sqoop JAR files are in the class path.

This is particularly important because the VoltDB export client does not attempt to access Sqoop or Hadoop until after the initial set of data has already been polled and acked from the VoltDB database. So any failure in connecting to Sqoop and/or Hadoop will result in that initial export data being lost. Future releases of VoltDB may attempt to detect and report this failure condition earlier. But for the current release, it is highly recommended that you perform a test run first to ensure all components are configured correctly.

For partitioned tables, the value of the column used to partition the table determines what partition the row belongs to. If you use UPDATE to change this value and the new value belongs in a different partition, the UPDATE request will fail and the stored procedure will be rolled back.

Updating the partition column value may or may not cause the record to be repartitioned (depending on the old and new values). However, since you cannot determine if the update will succeed or fail, you should not use UPDATE to change the value of partitioning columns.

The workaround, if you must change the value of the partitioning column, is to use both a DELETE and an INSERT statement to explicitly remove and then re-insert the desired rows.

If you refer to a table or column name that does not exist, the VoltDB compiler issues the error message "user lacks privilege or object not found". This can happen, for example, if you misspell a table or column name.

Another situation where this occurs is if you mistakenly use double quotation marks to enclose a string literal (such as WHERE ColumnA="True"). ANSI SQL requires single quotes for string literals and reserves double quotes for object names. In the preceding example, VoltDB interprets "True" as an object name, cannot resolve it, and issues the "user lacks privilege" error.

The workaround is, if you receive this error, to look for misspelled table or columns names or string literals delimited by double quotes in the offending SQL statement.

VoltDB opens a file descriptor for every client connection to the database. In normal operation, this use of file descriptors is transparent to the user. However, if there are an inordinate number of concurrent client connections, or clients open and close many connections in rapid succession, it is possible for VoltDB to exceed the process limit on file descriptors. When this happens, new connections may be rejected or other disk-based activities (such as snapshotting) may be disrupted.

In environments where there are likely to be an extremely large number of connections, you should consider increasing the operating system's per-process limit on file descriptors.

This is not a problem when looking at VoltDB logs separately. However, you should be aware of this distinction when integrating logging of VoltDB with logging of other system components that use the local time zone (rather than GMT). You may want to convert one or the other log streams so the time zones match.

The file voltdb/log4j.xml lists all of the VoltDB-specific logging categories. It also serves as a useful logging schema. The sample applications and the VoltDB shell commands use this file to configure logging and it is recommended for new application development.

The master and the replica databases must match in the following regards for replication to begin:

If not, the DR agent refuses to start.

The master database initially queues replication data in memory. If the pending data exceeds the allocated queue size, data is overflowed to disk in the directory voltdbroot/dr_overflow. The values for the queue size and overflow directory are not currently configurable.

When using a multi-homed server (that is, a server with two or more network interface cards and network addresses) as part of a master database, you must explicitly identify which interface the server uses for both internal and external communication when you start the VoltDB server process. For example:

$ voltdb create deployment deployment.xml \
  catalog catalog.jar leader serverA \
  license license.xml \
  externalinterface 10.11.169.10 \
  internalinterface 10.12.171.14

The specified external interface is used for the client and admin ports and database replication. The internal interface is used for all inter-node cluster communication. If you do not explicitly specify which interface to use, replication will fail when the DR agent attempts to identify and connect to the master cluster nodes.