Dealing with SQL Width in the Progress OpenEdge RDBMS: AUTHORIZED DATA TRUNCATION As a continuation of our efforts to deliver extensive SQL enhancements to Progress® OpenEdge® 11, customers will now find two new features to help better manage issues that may arise due to SQL width. Data that extends beyond its SQL width definition can cause query failures and limit the ability to obtain complete result sets. To assist database administrators in addressing the issue of SQL width, we are presenting as a series, two complementary technical whitepapers, each detailing a specific solution: Dealing with SQL Width in the Progress OpenEdge RDBMS: Authorized Data Truncation (ADT), and Dealing with SQL Width in the Progress OpenEdge RDBMS: Autonomous Schema Update (ASU). OVERVIEW Progress OpenEdge 11.5.1 contains a new SQL feature called Authorized Data Truncation.* This feature helps overcome the SQL width problem that is encountered when database column values are larger than the column’s size as defined in SQL. Authorized Data Truncation can prevent queries from failing when the large column values are read. By default, if the width of the column data being operated on in a query exceeds the defined width of the column, SQL returns an error, and as a result the query fails. If Authorized Data Truncation is enabled, SQL will instead truncate the large value down to the defined size. This truncation is acceptable as a temporary workaround to help applications succeed. Truncation changes data values significantly. Users will be able to see that the values were truncated, as Data Truncation optionally logs the truncation. The database column may acquire a data value larger than its defined size in the following ways: • When a large value is inserted or updated using ABL • Reducing the column size after using the SQL Alter Table command inserting right-sized data using ABL or SQL * - 11.7 Progress® OpenEdge® has introduced a new option for Authorized Data Truncation feature. The new option is “OUTPUT.” Also, the “ON” option for Authorized Data Truncation feature has been renamed to “ALL.” Progress.com 2 TRUNCATION OF DATA OpenEdge SQL throws an error when operating on data that is larger than the SQL Width size. It is now possible to authorize SQL to truncate large data while operating on it. SQL can be authorized to truncate large data for the varchar column type only. There are two ways to authorize SQL to truncate large data: 1. Server Startup Parameter 2. Connection URL Server Startup Parameter The following syntax is used to authorize SQL to truncate large data during server startup: proserve –db <dbname> -S <port-number> -SQLTruncateTooLarge <all/output/off> 1. The SQLTruncateTooLarge value is remembered for the lifetime of the server for all connections. 2. SQL will truncate data exceeding the column size at all stages if SQLTruncateTooLarge is set to “all.” This means that SQL reads truncated data right from the database and does user specified operations on truncated data that is read from the database. 3. SQL will truncate data exceeding the column size at only result stage if SQLTruncateTooLarge is set to “output.” This means that SQL reads full data from the database and does user specified operations on it. At the stage where output is presented to the user, results are truncated. 4. SQL will return an error in case of data exceeding the column size if SQLTruncateTooLarge is set to “off.” Progress.com 3 5. SQL returns an error in case of data exceeding the column size if the value of SQLTruncateTooLarge is not set. 6. The value of SQLTruncateTooLarge is logged in the database lg file (irrespective of whether “SQLTruncateTooLarge” is specified or not). Connection URL Data truncation authorization option can be embedded in connection URL as follows: jdbc:datadirect:openedge://<ip>:<port>;databaseName=<dbname>; truncateTooLarge=<all/output/off>; 1. The value of truncateTooLarge is connection specific and will be remembered by SQL for the connection session. This option appears only for the client which used the truncateTooLarge parameter in connection URL. 2. If the value of truncateTooLarge is set to ALL, SQL truncates data exceeding the column size from the time where it reads data from the database. 3. If the value of truncateTooLarge is set to OUTPUT, SQL truncates data exceeding the column size only at the time where result is presented to the user. Complete data is read from the database and all the user operations are done on full data. 4. If the value of truncateTooLarge is set to OFF, SQL returns an error if the data exceeds the column size. 5. If the value of truncateTooLarge is not set, SQL returns an error in case the data exceeds the column size. Progress.com 4 Rules for the Options 1. SQL truncates data for large data if the value for either truncateTooLarge or SQLTruncateTooLarge is set to either ALL or OUTPUT. 2. If neither of the two values is provided, SQL returns an error and does not truncate data that exceeds column size. This provides the default behavior which Progress OpenEdge SQL has always provided. 3. If both values are specified, connection attribute will override startup option. Connection attribute has precedence over server startup parameter. Example 1 – truncation and authorization parameters For example, suppose the following table is created in the database: Create table TAB (name varchar (5)); The column name has an entry “TomCruise” which exceeds the size of the column i.e., five. The following table describes the output of the query “select * from tab” for different combinations of server startup parameters and connection URL options: Progress.com 5 SERVER STARTUP PARAMETER CONNECTION URL PARAMETER OUTPUT OF USE CASE EXPLANATION (SQLTruncate- TooLarge ) (truncateTooLarge) QUERY Both are specified. So connection URL gets precedence. 1 OFF OFF ERROR As this is set as OFF, SQL returns an error. Both are specified. So connection URL will get precedence. 2 OFF ALL TomCr As this is set as ALL, data will be truncated. So “TomCr” will be the output truncating the data after 5th position. Both are specified. So connection URL will get precedence. 3 ALL OFF ERROR As this is set as OFF, SQL returns an error. Both are specified. So connection URL will get 4 ALL ALL TomCr precedence. As this is set as ALL, data will be truncated. So “TomCr” will be the output truncating the data after 5th position. Only connection URL is specified and set as ALL. 5 NOT SPECIFIED ALL TomCr So “TomCr” will be the output truncating the data after 5th position. Only connection URL is specified and set as OFF. 6 NOT SPECIFIED OFF ERROR So SQL returns an error. Only server startup parameter is specified and set as ALL. So 7 ALL NOT SPECIFIED TomCr “TomCr” will be the output truncating the data after 5th position. Only server startup parameter is specified and set as OFF. So SQL 8 OFF NOT SPECIFIED ERROR returns an error. Both are absent. So SQL returns an error. 9 NOT SPECIFIED NOT SPECIFIED ERROR Both are specified. So connection URL will get precedence. As this is set to ALL, 10 OUTPUT ALL TomCr data will be truncated. So “TomCr” will be the output truncating the data after 5th position. Both are specified. So connection URL will get precedence. As this is set to 11 OUTPUT OUTPUT TomCr OUTPUT, data will be truncated. So “TomCr” will be the output truncating the data after 5th position. Both are specified. So connection URL will get precedence. 12 OFF OFF ERROR As this is set as OFF, SQL returns an error. Only server startup parameter is specified and set as OUTPUT. 13 NOT SPECIFIED NOT SPECIFIED TomCr So “TomCr” will be the output truncating the data after 5th position. Both are specified. So connection URL will get precedence. 14 OUTPUT OUTPUT TomCr As this is set to OUTPUT, data will be truncated. So “TomCr” will be the output truncating the data after 5th position. Both are specified. So connection URL will get precedence. 15 OUTPUT OUTPUT TomCr As this is set to OUTPUT, data will be truncated. So “TomCr” will be the output truncating the data after 5th position. Both are specified. So connection URL will get precedence. 16 OUTPUT OUTPUT TomCr As this is set to OUTPUT, data will be truncated. So “TomCr” will be the output truncating the data after 5th position. Progress.com 6 Example 2 – truncation and predicates Using the table and data from Example 1, consider the simple query: • Select * from Tab where name >= ‘T’ • This would return a row with the value of column name as ‘TomCr’ • Consider the simple query: Select * from Tab where name = ‘TomCruise’ This would return 0 rows when Authorized Data Truncation is set to “ALL” because the truncated value of name does not equal TomCruise. SQL in this case will read data as “TomCr” from database as Authorized Data Truncation is set to “ALL.” On the other hand, if Authorized Data Truncation is set to “OUTPUT,” the output of the query will be “TomCruise.” In this case, SQL will read complete data and hence WHERE condition becomes true. ADT value Output of “SELECT * FROM TAB WHERE NAME = ‘TomCruise’” ALL Empty Result Set OUTPUT TomCruise OFF Data exceeding max length or precision error NOT SPECIFIED Data exceeding max length or precision error Table describing output of a SELECT statement for different ADT values Progress.com 7 Example 3 – relationships between tables with truncated keys This example illustrates one type of unanticipated outcome that truncation can lead to when Authorized Data Truncation is set to “ALL.” Suppose that two tables are related based on a part number, which is defined to SQL as 8 characters in size.