Documentation

Swapping is an operating system feature that optimizes memory usage when running multiple processes. However, memory is a critical component of the VoltDB server process. Any contention for memory, including swapping, will have a very negative impact on performance and functionality.

We recommend using dedicated servers and disabling swapping when running the VoltDB database server process. Use the swapoff command to disable swapping on Linux systems. If swapping cannot be disabled for any reason, you can reduce the likelihood of VoltDB being swapped out by setting the kernel parameter vm.swappiness to zero.

There is an issue where, under certain conditions, the use of TCP segmentation offload (TSO) and generic receive offload (GRO) can cause nodes to randomly drop out of a cluster. The symptoms of this problem are that nodes timeout — that is, the rest of the cluster thinks they have failed — although the node is still running and no other network issues (such as a network partition) are the cause.

Disabling TSO and GRO is recommended for any VoltDB clusters that experience such instability. The commands to disable offloading are the following, where N is replaced by the number of the ethernet card:

ethtool -K ethN tso off
ethtool -K ethN gro off

Note that these commands disable offloading temporarily. You must issue these commands every time the node reboots.

Previously all VoltDB system procedures operated as database transactions. That is, system procedures executed as multi-partition transactions with two consequences:

Since statistical procedures do not change the state of the database, they are now executed separately from database transactions. This applies to the system procedures @Statistics, @SystemInformation, and @SystemCatalog. The consequences of this change are that the statistical system procedures have significantly less impact on database performance and, although the resulting statistics are not transactionally exact, they are an accurate reflection of overall database performance at the time the procedure executes. More importantly, it is now possible to get information about database operations even when an errant stored procedure or server is blocking other database operations.

VoltDB now supports outer and explicit joins. Previously, all joins were implicitly inner joins (using a comma-separated list of table names). You can now use explicit join syntax to specify either inner or outer joins on specific columns using the ON or USING clause. The new syntax for the SELECT statement is as follows:

 Select-statement  [{set-operator}  Select-statement ]  ... 
 Select-statement: 
  SELECT  [ TOP integer-value ]
 { *  |   [ ALL  |   DISTINCT ] { column-name  |   selection-expression } [AS alias] [,...] }
  FROM    table-reference    [ join-clause ]... 
 [WHERE  [NOT] boolean-expression   [ {AND | OR}  [NOT] boolean-expression]...]
 [clause...]
 table-reference: 
 { table-name  |   view-name } [AS alias]
 join-clause: 
  ,table-reference 
 [ INNER | {LEFT | RIGHT} [OUTER] ]  JOIN  {table-reference} [join-condition]
 join-condition: 
  ON conditional-expression 
  USING (column-reference [,...]) 
 clause: 
  ORDER BY  {  column-name  |   alias  } [ ASC  |   DESC ] [,...]
  GROUP BY  {  column-name  |   alias  } [,...]
  LIMIT  { integer-value [OFFSET row-count]  |   ALL }
 set-operator: 
  UNION  [ALL]
  INTERSECT  [ALL]
  EXCEPT 

For this initial release, outer joins support the joining of two tables only. Explicit and implicit inner joins support joining more than two tables.

Web Studio allows you to invoke stored procedures on a running database, entering procedure arguments as text. Previously, Web Studio had some difficulty parsing strings containing multiple embedded quotation marks. Argument parsing has been enhanced to support arbitrarily complex strings.

This release includes a number of improvements to the planning of SQL statements. In particular, improvements have been made to how indexes are applied under various conditions.

Previously, the @SnapshotStatus system procedure only returned results for the server processing the request, not all nodes in the cluster. Starting with this release, a call to @SnapshotStatus returns status information for all nodes in the cluster.

By default, when enabling security the deployment file contains definitions of users and passwords. These definitions are included as plain text. If your environment or operating procedures require the deployment file to be secured against observation, you can preprocess the file to obscure the passwords using the voltdb mask command. These processed deployment files can then be used to start the VoltDB database. The syntax for the voltdb mask command is as follows:

$ voltdb mask deployment-file [new-deployment-file]

If you specify one file name, the file is processed in place. If you specify two file names, the first is used as the input file and the second as the output file. The processed deployment file contains an additional attribute on the <user> tag, plaintext="false" indicating to the VoltDB server that the passwords have been obscured.

When VoltDB compiles the database schema, it displays a summary of the stored procedure queries as part of the compiler output. This output includes annotations identifying the type of procedure (read vs. write and single vs. multi-partitioned), potentially slow queries due to a sequential table scan, and possible dangers introduced by non-deterministic output. Several of these annotations have been changed to improve readability. The read/write annotation has been changed to [READ] and [WRITE] and [Seq] has been changed to [TABLE SCAN].

This release fixes a critical bug where updating the schema "on the fly" could cause the database to stop and make the command logs unusable for recovery. The issue only affects application catalogs that change a unique index by either adding or removing columns from the unique index (or primary key). Because of the seriousness of this bug, all users are strongly urged to upgrade to the latest version at their earliest possible convenience.

Until you upgrade, avoid updating the database catalog while the database is running if the update changes unique indexes. Note this issue only affects live updates. You can still update the schema using a maintenance window by performing the following sequence of commands: save, shutdown, create and restore.

It is now possible to make many more changes to the schema "on the fly" — while the database is running — than in previous versions. You can now add, delete, or modify columns and indexes as well as tables. The only remaining limitations are that you cannot add a new constraint to an existing column or index and you cannot modify the definition of an existing view. (However, you can add and remove views.)

With this new functionality, the ability to perform line schema changes becomes a commercial feature. In other words, starting with V3.2 the @UpdateApplicationCatalog system procedure and the voltadmin update command are available in the Enterprise Edition only.

In addition to new capabilities in live schema updates, the updates themselves have been improved in terms of performance and resilience to node failures. Updates to large catalogs, especially those with many stored procedures, are significantly faster than in previous versions. In addition, the update process has been hardened to avoid problems if a node fails during an update.

Warning concerning previous versions: If you perform catalog updates using previous versions of VoltDB, it is recommended that you take a snapshot prior to and immediately after the update as a precaution in case of database failure. Command logs and other durability features may be in an inconsistent state following a catalog update and before a new snapshot is taken. This issue is corrected in the current release.

It is possible for a snapshot restore (using the voltadmin restore command or @SnapshotRestore system procedure) to partially succeed. For example, if the schema has changed since the snapshot was created or some snapshot files are missing, only part of the data may be restored.

Previously, this condition was reported as success with details about what succeeded and what failed in the VoltTable response. To avoid confusion with a complete and successful restore, this function has been changed to report an error, OPERATIONAL_FAILURE, if one or more tables or partitions are not restored.

The default heartbeat timeout has been increased from 10 seconds to 90 seconds. In production, it has been found that network and process contention issues can make VoltDB mistakenly exceed the 10 second timeout, resulting in still functioning nodes being timed out. As a result, the default has been increased.

A consequence of this change is that it will now take longer for actual node failures to be recognized by the rest of the cluster, potentially stalling transactions until the failure is detected. You can change the timeout period using the <heartbeat> element in the deployment file or in the configuration dialog of the VoltDB Enterprise Manager. See the appendix on server configuration options in the VoltDB Management Guide for details.

The following issues have been fixed:

V3.1 includes production-ready support for Database Replication (DR). The changes to transaction coordination introduced in V3.0 required a significant rewrite of the DR functionality. As a result, the initial V3.0 release included a beta release of DR only. This release returns DR to full operational status. Users who stayed with earlier VoltDB releases to use DR in production are now encouraged to upgrade to V3.1.

V3.1 introduces a new function, CAST(), that lets you convert the datatype of an expression. CAST() can be very useful when performing operations on column values or function results that are not in the appropriate datatype. For example, you can use CAST() to convert the results of the FIELD() function, which always returns a string, to a numeric datatype. See Using VoltDB for more information.

V3.1 also includes additional functions for traversing JSON data. The ARRAY_LENGTH() and ARRAY_ELEMENT() functions return the number of elements in a JSON array and a specific element from that array, respectively. These functions can be combined with other functions to retrieve specific items within the JSON structure. For example, to following SQL fragment retrieves the first element of an array in the JSON field named "options" from the column JSONDATA:

SELECT ARRAY_ELEMENT(FIELD(JSONDATA,"options"),0) AS first_option

See Using VoltDB for more information on using these functions.

The Log4J messages generated by VoltDB have been reorganized. Messages previously logged under the JOIN component are now logged under REJOIN and a new logger, SNAPSHOT, has been added for logging snapshot activity.

VoltDB 3.0 includes a new transaction coordination architecture that reduces latency and improves transaction throughput. Some of the benefits of VoltDB 3.0 include lower overall latency, higher throughput, and reduced dependency on NTP.

V3.0 integrates much of the database configuration information into the schema, eliminating the need for a separate project definition file. Features that can now be defined in data definition language (DDL) include:

In addition, several new command line tools make the development process simpler and more consistent. New shell commands for compiling, starting, and managing VoltDB databases include voltdb, voltadmin and sqlcmd.

In addition to improvements in transaction coordination and the development process, a number of new functional capabilities have been added. New features include:

Users of earlier versions of VoltDB should be aware of the following changes in behavior that could impact their applications:

If your SELECT statement includes two aggregates of the same column, VoltDB will return only one column value as part of the result set. For example, if your SQL statement is SELECT COUNT(ColumnA), COUNT(ColumnA) FROM MyTable, the resulting VoltTable will have only one column value per row. The workaround is to either not request the same value twice or aggregate on different columns (for example, SELECT COUNT(Column10), COUNT(Column2)).

In SELECT statements, the selection expression can include columns, aggregate functions (such as COUNT), or arithmetic expressions involving columns (such as Col1 + Col2). However, they cannot include arithmetic involving aggregate functions (such as SELECT SUM(Price) + 2). Using aggregate functions in an arithmetic expression results in an error when you compile the application catalog.

The workaround is to use the aggregate in the SELECT statement and then perform the additional arithmetic on the result set either in the stored procedure or in the client application after the transaction completes.

Use of SELECT DISTINCT is supported for a single column (such as SELECT DISTINCT Price FROM Inventory). However, using DISTINCT with multiple columns or arithmetic expressions is not currently supported. For example, the following SELECT DISTINCT statements should not be used:

SELECT DISTINCT Price, Discount FROM Inventory
SELECT DISTINCT (Price - Discount) FROM Inventory

VoltDB currently intercepts assertions as part of its handling of stored procedures. Attempts to use assertions in stored procedures for debugging or to find programmatic errors will not work as expected.

When creating a view of a partitioned table, using the CREATE VIEW statement, the column on which the table is partitioned must be included in the GROUP BY clause. If not, queries of the view using SELECT ... FROM can give incorrect results. Note that VoltDB does not detect or warn you of this condition. Be sure to include the partitioning column in the GROUP BY clause when creating a view on a partitioned table.

There is a problem with how the math library used to build the C++ client library handles large decimal values on 32-bit operating systems. As a result, the C++ library cannot serialize and pass Decimal datatypes reliably on these systems.

Note that the C++ client interface can send and receive Decimal values properly on 64-bit platforms.

To ensure proper recovery on startup, either from command logs or the last database snapshot, make sure all snapshot files — or at least complete subsets of the snapshot files — are available on the nodes of the cluster. If you delete or move snapshot files (for example, copying all snapshot files to a single node) be sure to keep all of the files for each node together. Do not selectively delete or move individual files or else the recovery may fail.

You can modify the database schema of an existing database by recompiling the application catalog with the changes, saving the current database contents, restarting with the modified catalog and then restoring the contents. However, if the modified catalog adds a column with the default value specified as NULL, the database will fail when you attempt to restore a snapshot created using the old catalog. The workaround, for the time being, is to not specify a default value for the new column, specify a default other than NULL, or save the database contents as CSV files and use the csvloader utility to load each table.

You can add and remove tables on the fly using voltadmin update or the @UpdateApplicationCatalog system procedure. To modify individual tables, such as adding or removing columns, you must save a snapshot, create a new database with the modified catalog, and then restore the snapshot. However, this latter method does not currently support changing a column from partitioned to replicated, or the reverse.

The workaround for changing a partitioned table to replicated or a replicated table to partitioned is the following:

For partitioned tables, the value of the column used to partition the table determines what partition the row belongs to. If you use UPDATE to change this value and the new value belongs in a different partition, the UPDATE request will fail and the stored procedure will be rolled back.

Updating the partition column value may or may not cause the record to be repartitioned (depending on the old and new values). However, since you cannot determine if the update will succeed or fail, you should not use UPDATE to change the value of partitioning columns.

The workaround, if you must change the value of the partitioning column, is to use both a DELETE and an INSERT statement to explicitly remove and then re-insert the desired rows.

If you refer to a table or column name that does not exist, the VoltDB compiler issues the error message "user lacks privilege or object not found". This can happen, for example, if you misspell a table or column name.

Another situation where this occurs is if you mistakenly use double quotation marks to enclose a string literal (such as WHERE ColumnA="True"). ANSI SQL requires single quotes for string literals and reserves double quotes for object names. In the preceding example, VoltDB interprets "True" as an object name, cannot resolve it, and issues the "user lacks privilege" error.

The workaround is, if you receive this error, to look for misspelled table or columns names or string literals delimited by double quotes in the offending SQL statement.

VoltDB opens a file descriptor for every client connection to the database. In normal operation, this use of file descriptors is transparent to the user. However, if there are an inordinate number of concurrent client connections, or clients open and close many connections in rapid succession, it is possible for VoltDB to exceed the process limit on file descriptors. When this happens, new connections may be rejected or other disk-based activities (such as snapshotting) may be disrupted.

In environments where there are likely to be an extremely large number of connections, you should consider increasing the operating system's per-process limit on file descriptors.

This is not a problem when looking at VoltDB logs separately. However, you should be aware of this distinction when integrating logging of VoltDB with logging of other system components that use the local time zone (rather than GMT). You may want to convert one or the other log streams so the time zones match.

The file voltdb/log4j.xml lists all of the VoltDB-specific logging categories. It also serves as a useful logging schema. The sample applications and the VoltDB shell commands use this file to configure logging and it is recommended for new application development.