## Text Tables as a Standard Feature of HSQLDB

Text Table support for HSQLDB was originally developed by Bob Preston independently from the Project. Subsequently Bob joined the Project and incorporated this feature into version 1.7.0, with a number of enhancements, especially the use of conventional SQL commands for specifying the files used for Text Tables.

In a nutshell, Text Tables are CSV or other delimited files treated as SQL tables. Any ordinary CSV or other delimited file can be used. The full range of SQL queries can be performed on these files, including SELECT, INSERT, UPDATE and DELETE. Indexes and unique constraints can be set up, and foreign key constraints can be used to enforce referential integrity between Text Tables themselves or with conventional tables.

HSQLDB with Text Table support is the only comprehensive solution that employs the power of SQL and the universal reach of JDBC to handle data stored in text files and will have wide ranging use way beyond the currently established Java realm of HSQLDB.

### Goals of the Implementation

1. We aimed to finalise the DDL for Text Tables so that future releases of HSQLDB use the same DDL scripts.
2. We aimed to support Text Tables as GLOBAL TEMPORARY or GLOBAL BASE tables in the SQL domain.

#### The Implementation

1. #### Definition of Tables

Text Tables are defined similarly to conventional tables with the added TEXT keyword:

    CREATE [TEMP] TEXT TABLE <tablename> (<column definition>… [<constraint definition>…])

In addition, a SET command specifies the file and the separator character that the Text table uses:

    SET TABLE <tablename> SOURCE <quoted_filename_and_options> [DESC]

Text Tables cannot be created in memory-only databases (databases that have no script file).

2. #### Scope and Reassignment

- A Text table without a file assigned to it is READ ONLY and EMPTY.
- A Temporary Text table has the scope and the lifetime of the Session.
- Reassigning a Text Table definition to a new file has implications in the following areas:

1. The user is required to be an administrator.
2. Existing transactions are committed at this point.
3. Constraints, including foreign keys referencing this table, are kept intact. It is the responsibility of the administrator to ensure their integrity.

3. #### Null Values in Columns of Text Tables.

- Empty columns not of type CHARACTER and its variants are treated as NULL.
- Empty columns of type CHARACTER and its variants are treated in two different ways according to the setting of the database property textdb.<tablename>.empty_is_null. The default value of empty_is_null is true:

- false: Columns of type CHARACTER and its variants are always NOT NULL. Empty columns are treated as zero-length strings.
- true: Columns of type CHARACTER and its variations cannot hold an empty string. Empty columns are treated as NULL.

4. #### Configuration

The default field separator is a comma (,). A different field separator can be specified in either the <databasename>.properties file or within the SET TABLE SOURCE statement. For example, to change the field separator for the table mytable to a vertical bar, either place the following in the <databasename>.properties file:

    textdb.mytable.fs=|

Or, within the SET TABLE SOURCE statement, for example:

    SET TABLE mytable SOURCE "myfile;fs=|"

Since HSQLDB treats CHAR’s, VARCHAR’s, and LONGVARCHAR’s the same, the ability to assign different separators to the latter two is provided. When a different separator is assigned to a VARCHAR or LONGVARCHAR field, it will terminate any CSV field of that type. For example, if the first field is CHAR, and the second field LONGVARCHAR, and the separator fs has been defined as the pipe (|) and vs as the period (.) then the data in the CSV file for a row will look like:

First field data|Second field data.Third field data

The following example shows how to change the default separator to the pipe (|), VARCHAR separator to the period (.) and the LONGVARCHAR separator to the tilde (~). Either place the following in the .properties file:

    textdb.mytable.fs=|
    textdb.mytable.vs=.
    textdb.mytable.lvs=~

Or, within the SET TABLE SOURCE statement, for example:

    SET TABLE mytable SOURCE "myfile;fs=|;vs=.;lvs=~"

HSQLDB also recognises the following special indicators for separators:

\semi - semicolon
\quote - quote
\space - space character
\apos - apostrophe
\n - newline - Used as an end anchor (like $ in regular expressions)
\r - carriage return
\t - tab
\\ - backslash
\u#### - a Unicode character specified in hexadecimal

Furthermore, HSQLDB provides csv file support with two additional boolean options: ignore_first and quoted. The ignore_first option (default false) tells HSQLDB to ignore the first line in a file. This option is used when the first line of the file contains column headings. The quoted option (default true) tells the program that some fields may be quoted. These options may also be specified in the .properties file:

    textdb.mytable.ignore_first=true
    textdb.mytable.quoted=false

Or, within the SET TABLE SOURCE statement:

SET TABLE mytable SOURCE "myfile;ignore_first=true;quoted=false"

When the default quoted option is in force, fields that are written to a line of the csv file will be quoted only if they contain the separator character.

Finally, HSQLDB provides the ability to read a text file from the bottom up, by placing the keyword "DESC" at the end of the SET TABLE SOURCE statement:

    SET TABLE mytable SOURCE "myfile" DESC

This feature provides functionality similar to the Unix tail command, by re-reading the file each time a select is executed. Using this feature sets the table to read-only mode. Afterwards, it will no longer be possible to change the read-only status with SET TABLE <tablename> READONLY TRUE. [Does this feature allow a different program to modify the csv file from time to time while it is being used by HSQLDB?]

- #### Text File Issues

- By default, file locations are restricted to below the directory that contains the database. This can be overridden by the database property:

    textdb.<tablename>.allow_full_path=true.

- The file location for a text table created with

    SELECT <select list> INTO TEXT <tablename> FROM …

is the directory that contains the database and the file name is based on the table name. The table name is converted into the file name by replacing all the non-alphanumeric characters with the underscore character, conversion into lowercase, and adding the ".csv" suffix.

- Each text table has a system-defined primary key that refers to the file offset of the row and cannot be changed. This prevents use of IDENTITY columns with text tables, as an IDENTITY column is implicitly defined as a PRIMARY KEY. You can, however, use the UNIQUE constraint with text tables. Hence, to acheive the functional equivalent of an IDENTITY column, you can set up your own counter, using an ordinary table with only one row and one integer column. You can then increment this value each time and read the result value to use in a text table INTEGER column with the UNIQUE constraint. Secondary indexes and UNIQUE constraints can be used on any column of a text table.
- When a table source file is used with the`textdb.mytable.ignore_first=true` option, the first, ignored line is deleted after updating the table and replaced with a blank line.

Authors: Bob Preston and Fred Toussi - 29 May 2002 - updated by FT 11 Oct 2002

Copyright 2002 Bob Preston and Fred Toussi. Permission is granted to distribute this document without any alteration under the terms of the HSQLDB license. Additional permission is granted to the HSQLDB Development Group to distribute this document with or without alterations under the terms of the HSQLDB license.

http://hsqldb.sourceforge.net