Upgrading from  6.1x to Pervasive.SQL 2000i

Revision 7

October 2001

Pervasive Software Inc.

Pervasive Software Inc. 12365 Riata Trace Parkway II Austin, Texas 78727 Public Relations Contact: Marian Kelley Telephone: 800-287-4383/ 512-231-6000 Fax: 512-231-6010 Internet: http://www.pervasive.com

 2001-2002 Pervasive Software Inc., Pervasive, Pervasive.SQL, Btrieve, and the Pervasive logo are registered trademarks of Pervasive Software Inc. All other product names are trademarks of their respective companies. All rights reserved worldwide.

TABLE OF CONTENTS

Overview ...... 1

What’s New ...... 1

Installation ...... 1

Workstation...... 1 Workgroup ...... 2 Server ...... 3 Requesters or Server First?...... 3 Server Engine and User Count Key...... 3 Windows NT...... 3 NetWare...... 3 Unix ...... 4 Requesters ...... 4 Configuring Smart Components to Work ...... 4 Using the Network Installation ...... 5 Documentation...... 5 Troubleshooting installations...... 6 /Workgroup editions...... 6 Server editions ...... 6

Upgrade complete...... 7

Service Packs ...... 8

Troubleshooting...... 8 Available online resources ...... 8

Summary...... 9

Rebuilding Files (Optional) ...... 10

Optimizing the rebuild process...... 10 Configuration parameters and rebuilding files ...... 12 Sort Buffer Size setting ...... 12 Index page size...... 12 How much memory do you need? ...... 13 Cache Allocation setting ...... 14 Calculating depth of B-tree index (optional)...... 14

2 OVERVIEW

This paper describes an approach for upgrading applications from Btrieve 6.10 or Btrieve 6.15 to Pervasive.SQL 2000i. The number of required steps in the upgrade process is small, but depending on your environment, it can be a time-consuming process. This paper is designed to anticipate problems you may experience and help minimize the time required to complete your upgrade.

The content is organized in three general sections.

Installation of Pervasive.SQL by configuration (Workstation, Workgroup, or Server) and any known issues that you may encounter along the way

Rebuilding existing Btrieve files if needed - though the rebuild process is optional, it may be required to take advantage of some of the new features in Pervasive.SQL 2000i

Testing your application

WHAT’S NEW

Pervasive.SQL 2000i is a major step in the evolution of the Pervasive.SQL engine.

Completely redesigned Relational interface Workgroup edition database engine Distributed Tuning Interface Improved database security Redesigned utilities, including the new Pervasive Control Center and the Pervasive System Analyzer Support for connectivity with JDBC (Level 2, Type IV), OLE DB, and a replacement plug-in for the Borland Database Engine Row-Level Locking Auto-Reconnect functionality to avoid minor network disruptions Lowered Total-cost-of-Ownership Support for Windows XP Support for NetWare 6

The goal of this paper is not to review all of these features in detail. Please refer to the Pervasive.SQL 2000i Web site at http://www.pervasive.com/portals/psql.tml for details on each of the above features. It is, however, important to draw your attention to the first two points of the list.

INSTALLATION

Workstation

The Workstation version of Pervasive.SQL 2000i is only available as a 32-bit Windows database engine. This does not mean that your existing DOS-based and 16-bit Windows applications are no longer supported, but it does mean that your workstation must run /98/ME/XP/NT/2000.

Simply install the Pervasive.SQL 2000i as a Windows application and follow the instructions to adjust the configuration to run your application.

1 Windows 32-bit applications require no further adjustments to take advantage of Pervasive.SQL. Pervasive.SQL will transparently take over control of your database access.

Windows 16-bit applications also require no further adjustments to take advantage of Pervasive.SQL.

DOS applications will require a slight change to any batch files that are used to launch the application. Simply remove the command in the batch file that launches Btrieve. Pervasive.SQL takes care of providing the required interface for your DOS-based application.

Note: There was no workstation engine available with Btrieve 6.10

Workgroup

Available only with Pervasive.SQL 2000i, the Workgroup edition is the latest edition to the Pervasive.SQL family. It is designed from the ground up to meet the needs of workgroup and peer-to-peer configurations. This is the first release of Pervasive.SQL to address this configuration in such a manner. In Btrieve 6.15, the database engine used a mechanism called Multi-Engine File Sharing (MEFS) to provide connectivity in these environments. Though solid, the MEFS technology imposed certain limits on the workgroup, sacrificing performance for security in larger workgroup configurations.

Pervasive.SQL 2000i Workgroup solves the performance issue and raises the level of security and reliability in the workgroup. Designed essentially as a scaled-down version of the client/server database engine, Pervasive.SQL 2000i assigns an “owner” to the database through which all other workgroup members access the data store. The owner provides a central access point to the database, either on the local hard drive or on a remote storage device not running a Pervasive.SQL 2000i Workgroup engine. Through this central access, Pervasive.SQL 2000i Workgroup is able to manage file locks, record locks, transactions, and the like quickly and reliably. You can define the owner statically or let the system dynamically establish the owner engine each time the database is opened. In configurations where the database resides on a computer that is also running the database engine, Pervasive recommends that you include Pervasive.SQL 2000i Workgroup in the Startup folder of the workgroup computer hosting the database. In this manner there will always be an engine running where the database is located and it will assume ownership. In configurations where the database resides on a computer that cannot run the database engine, the first Pervasive.SQL 2000i Workgroup database engine accessing the database will become a Gateway to the remote database. Should you prefer or require that the Gateway always be the responsibility of a particular machine, you should use the “Gateway locator” feature to statically define the Gateway.

The base edition of the Workgroup engine comes with support for 3 concurrent users to access the database. If you require additional users, single user add-on licenses can be applied to increase the concurrent user count. Simply install the Workgroup engine on the new station requiring access to the database and apply the add-on user license to the Workstation engine acting as the owner for the database. It is important to note that in the workgroup configuration, the concurrent user count applies to each member of the workgroup independent of the other workgroup members. This means that all three users can access on any number of separate machines with the initial three-user license. For example, all three users could simultaneously be accessing databases on all three machines. Please refer to the Getting Started Guide for more information concerning how to apply user count licenses.

As in the case of the Workstation engine, the installation of the Workgroup engine is a simple and straightforward process. Simply install the Workgroup edition on each computer that will access the database, and your existing Windows 32-bit and 16-bit applications will immediately be able to use Pervasive.SQL 2000i. In the case of DOS applications, you must modify the batch file that previously invoked the DOS Btrieve database engine to remove the “Btrieve” command.

2 Server

Requesters or Server First?

Pervasive's upgrade and support policy dictates that you should always use requesters that are the same version as or newer than your Server engine. Based on this policy, Pervasive tests newer requesters against older server engines for full compatibility; however, it does not test old requesters against newer engines. Thus, Pervasive does not support running old requesters against a newer version of the server engine.

With the release of Pervasive.SQL 2000i (Service Pack 3), this support policy has been modified slightly. Pervasive will support older requesters to a newer Server engine, within a major release. This policy is effective with the launch of Pervasive.SQL 2000i (Service Pack 3, and does not apply to earlier versions or Service Packs. That is to say, Pervasive.SQL 2000i Service Pack 3 requesters will be compatible with Pervasive.SQL 2000i Service Pack 4 server engines. Pervasive.SQL 2000 SP2 requesters however, are not supported against Pervasive.SQL 200i (Service Pack 3).

Whether you install the server or the requester(s) first depends on the mechanics of your site. If, during your system down time, you can install the requesters to all the machines that need them, then you should install the server first and immediately upgrade all the client machines. If you need to upgrade the clients at a different time than the server, and your users need access to the system between these times, you should use a different approach.

One method you can use is to install the Pervasive.SQL server on a test machine. Then, when you install the new requesters, you can provide the Pervasive System Analyzer (PSA) that is run as part of the install (in part to verify communications), the location of the test server to verify the successful client installation. Once the client installation is complete, your applications will use the new requesters against your production server until it is time to upgrade the production server engine.

Server Engine and User Count Key

To install the server engine, simply run the installation program on the CD-ROM and follow the instructions in the manual, Getting Started (Server edition). The major difference you will note from Btrieve 6.10 and Btrieve 6.15 is the provision of a software user count key in Pervasive.SQL. In Btrieve 6.1x, the user count for the database engine was encoded in the server database engine. User count keys, first introduced in Pervasive.SQL 7, permit a flexible user count licensing mechanism allowing cumulative user counts. For example, you can combine a 20 and 10 user count licenses to create a 30 user count for a particular server. When prompted, you must insert the user count key diskette provided with the product.

Windows NT

You must install the Windows NT release of Pervasive.SQL directly from the CD-ROM drive on the target server. If you are installing Pervasive.SQL on Windows NT Server 4.0, please make sure that Service Pack 3 (or later) for Windows NT has been installed first. Pervasive strongly recommends that your NT 4.0 Server be configured with Service Pack 6 for NT 4.0 or better. Please note that there are also known issues regarding running Pervasive.SQL 2000i on NT 4.0 Service Pack 4 level and we do not recommend this configuration.

Novell NetWare

You must install the NetWare release of Pervasive.SQL from a Windows workstation connected to the target NetWare server with administrative access privileges.

3 If you are using Btrieve for NetWare, the newer version of BTRIEVE.NLM no longer accepts command line parameters. In addition, the settings in your old BSTART.NCF file are not migrated. You will receive a new BSTART.NCF that loads the Smart Components TM, MicroKernel, Btrieve, Relational interface and the communications modules. Command-line parameters are not allowed in this file.

As a rule, all older Btrieve Server utilities are not supported with Pervasive.SQL. This means that NLMs such as Bsetup, BTRMon, etc. cannot be used against Pervasive.SQL. Where applicable, replacement NLMs have been provided (ex. Butil.NLM). The new utilities must be used with Pervasive.SQL. Functionality previously found in these utilities can now be found through the client-side Windows-based utilities installed with the client software.

Settings for the Server engine are now saved in the BTI.CFG file, which resides in SYS:SYSTEM if you used the default installation path. A backup copy of your old BSTART.NCF is saved in the SYS:SYSTEM\PVSW7.BCK directory.

Linux

You must install the release of Pervasive.SQL 2000i directly from the CD-ROM drive on the target server. The Pervasive.SQL 2000i Server for Linux supports all deployments based on the 2.2 (or higher) kernel and Glibc 2.1 or higher. This includes most, if not all, of today’s current versions.

If you are planning to access the Pervasive.SQL Btrieve interface across a network from a Windows-based client, the Samba package must be installed on the Unix server. Samba is an open-source software suite that provides seamless file and print services to SMB/CIFS clients. Please refer to the Samba Website, http://www.samba.org, for installation and configuration instructions.

Requesters

This section provides some tips for ensuring successful client installations.

Configuring Smart Components to Work

If you plan to make modifications to your application to take advantage of the new features in Pervasive.SQL, be sure to modify your application to load the new “smart” client interface libraries, or “glue” DLLs. Complete instructions for doing so are provided in the Pervasive.SQL Programmer’s Guide that is part of the Pervasive.SQL SDK.

If you want to upgrade your client requesters without changing anything in the application, there is one important step that you must perform to be sure that the new Smart Components TM architecture in Pervasive.SQL works properly. This step ensures that your applications are loading the new, “smart” versions of the Pervasive.SQL client interface DLLs, rather than the old versions of the DLLs that may still remain on your hard drive. Immediately before installing the new client requesters, you should search for every occurrence of the following files on the workstation hard drive:

Wbtrcall.dll Wbtrv32.dll Wxqlcall.dll Wssql32.dll Wdbnm32.dll Wbnames.dll

4

Rename each occurrence of these DLLs that you find so that, if necessary, you will be able to restore them later. For example, you might change the last character to an “x” so that each file extension reads “.dlx”. Some applications require their original DLL versions, so you need to keep the older DLLs available in case you encounter an application with this limitation. If you do encounter an application that will not run with Pervasive.SQL clients, you need to contact your application vendor.

Pervasive.SQL 2000i Service Pack 3 also provides a new utility to assist you in removing or archiving these older components from your system – the Pervasive System Analyzer (PSA). The PSA is included as part of the installation process where it will scan the most common locations on the hard drive from existing components. If found, the PSA will prompt you regarding the action to take with the found components. You may choose to either Archive, Delete, or Leave the found components. The PSA is also available as a utility from the Pervasive Control Center (PCC) where you can specify that a more detailed search of the hard drive be done to include all subdirectories on the disk.

Once you have renamed all of the existing Btrieve and/or Scalable SQL interface DLLs or performed a system clean up with the PSA, you are ready to start the Pervasive.SQL client installation.

Using the Network Installation

By using the Typical or Custom option during Server engine installation, you can copy a requester installation package to the server machine. This allows you to install the new requesters on all your client computers by connecting to the server and running the appropriate installation program. There is no need to share the CD-ROM over the network or to physically carry the CD-ROM to all client machines in order to install the client requesters. Copying the client installation packages to the server is called “staging” the client installations on the server. Detailed instructions on how to stage the client installations on the server can be found in the book, Getting Started with Pervasive.SQL (Server edition). Please note that this manual, along with the entire Pervasive.SQL documentation set for your platform are included on the CD-ROM as part of the help files.

Another option is to copy the "clients" folder from the CD-ROM to the server. You can then connect to the server from your client machines and run the appropriate installation program from one of the subfolders in the "clients" folder. Client installation programs are available for Win16 and Win32 workstations. To install the client on a DOS workstation, you must copy the DOS requesters directly to the workstation.

Documentation

Pervasive.SQL 2000i (Service Pack 3) includes new support for Microsoft HTML Help. The installation program determines whether your system can support the new HTML Help format. If so, your documentation is provided in HTML Help format (files with .CHM extension). If not, Winhelp documentation is installed (files with .HLP extension). On a server installation, both forms are installed so that clients without support for HTML Help can access the documentation on the server.

HTML Help uses parts of the Internet Explorer (IE) engine to display help files. In the early versions of IE 5, Microsoft included a defect that causes display problems with modular HTML Help systems. These problems can be seen in PSQL2000.CHM, the master help file for Pervasive.SQL 2000i that dynamically loads the other help files. Since this defect only affects the file merging mechanism, you see no problems when loading individual help files. Some specific symptoms you may encounter are:

5 The Contents/Index/Search/Favorites pane at the left side of the help window does not display correctly or is slow to repaint when you select a different tab such as Index or Search. The body pane on the right side of the help window does not display correctly or at all. The splash screen displayed when first loading the help window does not clear until you click on it.

If you encounter any of these problems, the solution is to upgrade to IE 5.01 or higher.

You can also download Adobe Acrobat (PDF) versions from Pervasive’s Web site.

Troubleshooting installations

Workstation/Workgroup editions

If, after installing Pervasive.SQL Workstation or Workgroup, your Windows 16-bit or 32-bit applications do not automatically function (or seem to still be using the Btrieve 6.15 database engine), you may have a conflict with the DLLs on your system. Search for every occurrence of the following files on the workstation hard drive:

Wbtrcall.dll Wbtrv32.dll Wxqlcall.dll Wssql32.dll Wdbnm32.dll Wbnames.dll

Rename each occurrence of these DLLs that is not in the installation path you specified during installation (default is drive:\PVSW) so, if necessary, you will be able to restore them later. For example, you might change the last character to an “x” so each file extension reads “.dlx”. Some applications require their original DLL versions, so you need to keep the older DLLs available in case you encounter an application with this limitation. If you do encounter an application that will not run with Pervasive.SQL clients, you need to contact your application vendor.

As mentioned earlier in this paper, Pervasive.SQL 2000i Service Pack 3 also provides a new utility to assist you in removing or archiving these older components from your system – the Pervasive System Analyzer (PSA). The PSA is included as part of the installation process where it will scan the most common locations on the workstations hard drive from existing components. If found, the PSA will prompt you regarding the action to take with the found components. You may choose to either Archive, Delete, or Leave the found components. The PSA is also available as a utility from the Pervasive Control Center (PCC) where you can specify that a more detailed search of the hard drive be done to include all subdirectories on the disk.

Once you have renamed all of the existing duplicate/redundant DLLs or performed a system clean up with the PSA, you should be able to execute your applications without further problems.

Server editions

On NetWare 4.11 (fully patched) or higher, you must have Administrator security equivalence to the root of the target server where you are installing Pervasive.SQL 2000i. 6 An attempt to install an "Unlimited" user count on top of an existing user count may fail. If you encounter this problem, you must use the User Count Manager utility to remove the existing numbered user count before installing the "Unlimited" user count. On NetWare, if the install fails because BTRIEVE.NLM or BSPXCOM.NLM could not be overwritten, you must make sure that the file attributes are set to read-write mode, by using a Novell-provided utility, and you must ensure it is not loaded while you are trying to install. After installing Pervasive.SQL for NetWare, users may experience problems loading certain Pervasive.SQL modules at the server console. You may also experience problems trying to load other NLMs such as BSPXCOM.NLM. The load command may fail with Undefined Public Symbol errors such as NSGetConfigInfo or NsisValidRequestClass. These NLMs are failing to load because the NetWare server is not configured for TCP/IP. By default, Pervasive.SQL is configured with both SPX and TCP/IP as supported communication protocols. This configuration is set using the Setup Utility and recorded in BTI.CFG on the server in the SYS:\SYSTEM directory. BTI.CFG can be edited with a text editor if the Setup utilities will not connect.

If your NetWare server is configured for SPX only, use the Pervasive.SQL Setup Utility to connect to the server and change the Scalable SQL Communications Manager and Btrieve Communications Manager Supported Protocols to SPX only.

If you want to use TCP/IP to connect to Pervasive.SQL, the Pervasive.SQL configuration is correct by default. The NetWare server and the workstations must be configured for TCP/IP. Depending on the Supported Protocols settings in BTI.CFG for Btrieve Communications Manager, NWBSRVCM auto-loads BSPXCOM for IPX/SPX support and/or BTCPCOM for TCP/IP support for Btrieve.

Since Pervasive.SQL supports the TCP-IP protocol on NetWare, there maybe some difficulties in unloading the MicroKernel (NWMKDE.NLM) because of circular references involving BTRIEVE.NLM and other NetWare NLMs. Often it may be required to restart the NWMKDE because of changes to configuration settings. Since NetWare does not allow Btrieve.NLM to be shut down due to symbol references, the only safe way to do this is to restart the entire system. With Pervasive.SQL 2000i it is now possible to force a shutdown of the NWMKDE by issuing a "BTRV UNLINK" command before issuing the BSTOP command. This will cause Btrieve.NLM to detach itself from NWMKDE, allowing the NWMKDE to be unloaded. BTRIEVE.NLM will still fail to unload because of the OS symbol references. BSTART can be re-issued now to restart the NWMKDE. Note that if any OS NLMs which have references to Btrieve.NLM have files open, they will receive errors if they continue to issue Btrieve operations after Btrieve.NLM has detached itself from NWMKDE. If you are using NSS volumes on your NetWare server, please be sure that the NSS volumes are mounted before the Pervasive.SQL 2000i engine is loaded at the server. Failure to have these volumes mounted before the Pervasive.SQL 2000i engine will result in the engine being unable to access the files.

UPGRADE COMPLETE

If you do not wish to use all of the new Pervasive.SQL features and functionality, your upgrade is complete now. You are not required to rebuild your data files. Pervasive.SQL is completely backward compatible in respect to data file formats supporting file formats 5.x and 6.x.

This said, it is important to note that there are certain Pervasive.SQL features and functionality that will be unavailable if you do not rebuild your data files to the version 7 format.

7 The Pervasive.SQL features that are available with version 6.x file format are:

Performance improvement Transaction Logging and Durability if your data file contains a unique key International Sort Rule (ISR) support Row Level Locking

The Pervasive.SQL features that are available only with the version 7 file format are:

New 64 GB file size limit Transaction Logging and Durability if your data file does not contain a unique key For helpful tips on rebuilding files, refer to Appendix A – Rebuilding Files

SERVICE PACKS

As of this writing (October 2001):

Pervasive.SQL 2000i Server for NetWare, Server for NT/2000, Server for Linux, Workgroup, and Workstation are currently shipping with Service Pack 3. If you already possess any previous version of Pervasive.SQL 2000, Service pack 3 is available as a Web download from the Pervasive Software Web site.

Troubleshooting If, after applying a Pervasive.SQL Service Pack on NetWare, you get the message "Licensing information is out of sync," contact Pervasive Technical Support @ 1-800-287-4383 or via email at [email protected].

Available online resources Pervasive Developer Page http://www.pervasive.com/developerzone

Btrieve News Group news://comp.databases.btrieve

DevTalk Web Forums http://devtalk.pervasive.com/

Knowledge Base of Pervasive Software support issues http://support.pervasive.com/eSupport/

Product Updates & Service Packs http://www.pervasive.com/support/updates/

Technical Library http://www.pervasive.com/support/TechPapers.asp

Trial Downloads http://www.pervasive.com/downloads/index.asp

White papers on Pervasive Software http://www.pervasive.com/support/WhitePapers.asp

8 SUMMARY

Upgrading to Pervasive.SQL is an easy process. However, as with any software, if you have a large or complex environment, upgrading can become more complicated. The information presented in this document should help you successfully install Pervasive.SQL in any size environment.

9 Appendix A

Rebuilding Files

REBUILDING FILES (OPTIONAL)

If you wish to make use of all the new version features, such as larger file sizes or system key generation, you must rebuild your data files so that they use the version 7 file format. See Pervasive.SQL User's Guide, "Converting MicroKernel Data Files" for detailed information on how to use the Rebuild utilities to convert your data files.

The information below should help you make proper use of the Rebuild tools:

You should use the GUI Rebuild Utility if you have a large number of files to rebuild. This version of the utility allows you to select many files at once, making the selection process easier. You must convert DDFs as well as data files. For converting multiple files, the best approach (when using a Windows NT server) is to use the GUI Rebuild utility to build a list of all the files you want to convert, and then convert them all at once. Large files, such as 3 or 4 GB, may take several hours to convert. On NetWare, the best approach for multiple files is to use the BREBUILD.NLM server utility and build a command file as explained in the User's Guide, and include in the command file all the files that you wish to convert. If you have more than one server engine available, you may wish to use each of your available server engines to rebuild a portion of the total number of files that must be converted. During your scheduled down time for the upgrade, you may find it helpful to copy large files out to the computers where the other server engines are, and then copy the files back when you are finished converting. That way you can share the burden of converting among a number of CPUs. You can use the "Continue on Error" option in combination with a *.* file list to convert all the data files in a directory, even if other types of files exist in the directory. The other types of files generate an error when the Rebuild utility attempts to open them. If you have the "Continue on Error" option set, the utility ignores each of these errors and skips to the next file. If you have files in the same directory that are already in the version 7 format, they are also rebuilt, and the physical order of the records in these files may be changed depending on the rebuild options you select. You cannot convert from a more recent format back to an earlier format. For example, you cannot convert a version 7 data file to version 6.x format. There is currently no facility to perform a recursive or multi-directory conversion without building a list of all the individual directories or individual files that you wish to convert. The User's Guide indicates that you can build a list of data files, choose Run from the menu, and then select from your list of files the specific files you wish to convert in this run. These instructions should state the following: after you select the Run option, you cannot exclude any of the listed files from the conversion process that you are about to begin.

Optimizing the rebuild process The Rebuild utility creates copies of files by making Btrieve calls to the MicroKernel. This means that your MicroKernel configuration settings, as well as the amount of physical RAM present on your computer, can have a great effect on the amount of time required to rebuild data files, especially large files.

In general, building indexes requires much more time than building data pages. If you have a data file with many indexes, it will require more time to rebuild than would the same file with fewer indexes.

10 The Rebuild utility is capable of rebuilding a file using two different methods. In the default method, the Rebuild utility performs the following steps using Btrieve operations:

1. Creates a new, empty data file with the same record structure and indexes as defined in the source file. 2. Drops all the indexes from the new file. 3. Copies all the data into the new file, without indexes. 4. Adds the indexes, using the steps below. 5. For a particular key in the source file, the MicroKernel reads as many key values as it can into the memory buffer using the Extended Step operation with the appropriate filter. 6. The MicroKernel sorts the values in the memory buffer and writes out the sorted values to a temporary file. 7. The MicroKernel repeats steps 5 and 6 until it has processed the key value from every record. 8. The temporary file now contains several key value sets, each of which has been individually sorted. (The number of key value sets in the temporary file is approximately equal to the Blocks Required value, discussed later in this paper.) The MicroKernel now merges these sets into index pages, filling up each page to capacity. Each index page is added to the Btrieve file at the end, extending the file length. During this merge phase, the temporary file continues to grow but much more slowly than in the sorting phase (steps 5-7). 9. Steps 5-8 are repeated for each remaining key.

The second method—the "fallback" method—is used by the MicroKernel when there is not enough physical memory to use the default method. In this situation, the MicroKernel performs the following steps:

1. Creates a new, empty data file with the same record structure and indexes as defined in the source file. 2. Drops all the indexes from the new file. 3. Copies all the data into the new file, without indexes. 4. Adds the indexes, one by one, using the steps below. 5. For a particular key in the source file, the MicroKernel reads one record at a time using the Step Next operation. 6. The MicroKernel extracts the key value from the record and inserts it into the appropriate place in the index. This necessitates splitting key pages when they get full. 7. The MicroKernel repeats steps 5 and 6 until it has processed the key value from every record. 8. Steps 5-7 are repeated for each remaining key. The fallback method is typically much slower than the default method. If you have large data files with many indexes, the difference between the two methods can amount to many hours or even days. The only way to ensure that the default (fast) method is used is to be sure that you have enough physical memory on your server machine, and be sure that, in NetWare environments, the available memory is not highly fragmented. The sections below explain how to determine whether you have enough memory on your server.

11 Configuration parameters and rebuilding files The amount of free memory on your computer, along with certain configuration parameters, has a direct effect on how much time the index rebuild process takes. This section helps you determine whether your computer will use the default or fallback method of rebuilding indexes, and what actions you can take to help the process go faster.

In the formulas below, Values In Italics indicate values that are referred to more than once, so you may need to look at the previous paragraphs to remind yourself how to get a particular value.

Sort Buffer Size setting In the Setup Utility, choose the MKDE component, the Memory Resources category and the Sort Buffer Size attribute. This value determines how much memory is available to the MicroKernel for building indexes.

If your Sort Buffer Size attribute is set to a non-zero value, and it is smaller than the calculated Minimum Memory Bytes Required value (see below), then the value of Sort Buffer Size is used to determine the amount of memory the rebuild process should allocate for building indexes.

If your Sort Buffer Size attribute is set to zero, then the MicroKernel calculates the value of Minimum Memory Bytes Required and uses it as the amount of memory it should allocate.

Finally, the MicroKernel compares the amount of memory that it should allocate with 60% of the amount that is actually available. It then attempts to allocate the smaller of the two. The MicroKernel uses the 60% rule to avoid taking all the memory in the computer.

If the memory allocation fails, the MicroKernel attempts to allocate half of the original amount, and so on. If, in the end, the memory allocation fails completely, the MicroKernel builds the index using the fallback method.

If the memory allocation succeeds, the size of the block that has been allocated, the Allocated Block must be at least as large as defined in the formula for Minimum Memory Bytes Required in the next section. In addition, the number of blocks of memory required must meet certain criteria. See the formulas in the following section for the details.

Index page size The page size in your file also affects the speed of index creation. Continuing the previous example using a 20-byte key, if a page size of 512 bytes is used, Btrieve places between 8 and 18 key values in each index page (index pages are automatically kept at least half full). Indexing about 8 million records creates a B-tree about seven levels deep, and most of the key pages will be at the seventh level.

If a page size of 4096 bytes is used, then Btrieve places between 72 and 145 key values in each index page. This B-tree is only about four levels deep, requiring many fewer pages to be examined when inserting each new key value.

Smaller key pages will cause the time required to build indexes using the fallback method to increase dramatically. Key page size has much less affect on the performance of creating an index if the sort/merge method is used.

The time required to build the indexes increases exponentially with increasing depth of the B-tree.

If you are rebuilding a Btrieve file that uses a page size smaller than 4096 bytes, consider changing the page size during your rebuild process. For a discussion of what factors to consider when choosing or changing your page size, see the Pervasive.SQL Programmer's Guide that comes with the Pervasive.SQL SDK. 12 You can calculate the approximate maximum number of key values per index page using the following formula:

Max Keys = ( Page Size – 12 ) / (Key Length + Key Overhead)

How much memory do you need? You can use the formulas below to estimate the optimal and minimum amount of contiguous free memory required to rebuild your indexes on a particular file using the fast method. The optimal memory amount is enough memory to store all merge blocks in RAM. The minimum amount of memory is enough to store one merge block in RAM.

Key Length = total size of all segments of largest key in the file.

Key Overhead = 8 unless the key type is Linked Duplicate, in which case the value is 12.

Record Count = number of records in the file.

Optimal Memory Bytes = ((( Key Length + Key Overhead ) * Record Count ) + 65536 ) / 0.6

Minimum Memory Bytes = Optimal Memory Bytes / 30

For example, if your file has 8 million records, and the longest key is 20 bytes (not Linked Duplicate), the preferred amount of memory is ((( 20 + 8 ) * 8,000,000 ) + 65536 ) / 0.6 = 373,442,560 bytes of memory, or 373.5 MB. The minimum amount of memory is 1/30th of that amount, or 12,448,086 bytes of memory, or 12.45 MB.

The divisor 30 is used because the engine keeps track of no more than 30 merge blocks at once, but only one merge block is required to be in memory at any time. The divisor 0.6 is used because the engine allocates no more than 60% of available physical memory for this process.

If you do not have this minimum amount of physical memory available, the engine will use the fallback sorting method to rebuild your data file.

In this example, the optimal number is 373,442,560, or 373.5 MB of contiguous free memory. If you have this much contiguous free memory available, the sort/merge process takes place entirely in RAM. Because of the 60% allocation limit, this number is actually the amount of contiguous physical memory you need to have free when the rebuild process starts, not the amount that the rebuild process would actually use. Multiply this value by 0.6 if you want to figure the maximum amount that would actually be used by the rebuild process.

13 Finally, the block that is actually allocated must meet two additional criteria:

Blocks Required must be less than or equal to 30, where:

Blocks Required = Round Up ( Optimal Memory Bytes / Allocated Block )

And, Allocated Block must be greater than or equal to:

( ( 2 * Max Keys + 1 ) * ( Key Length + Key Overhead ) ) * Blocks Required

Continuing with the same example, assuming a 512-byte page size, and a block of 12.45 MB successfully allocated, let's calculate the test values:

Blocks Required = 373,500,000 / 12,450,000 = 30

This part of the test passes. On to the next part:

Max Keys = (512-12) / 28 = 18

( ( ( 2 * 18 ) + 1 ) * ( 20 + 8 ) ) * 9 = 9324

Is Allocated Block (12.5 million bytes) larger than 9324 bytes? Yes, so part two of the test passes. The index keys will be written out to a temporary file in 12.45 MB pieces, sorted in memory, and then written out as a B-tree index.

If any failure occurs during this process, such as a failure to open or write the temporary file, Rebuild starts over and uses the fallback method to build the index.

Cache Allocation setting In the Setup Utility, choose the MKDE component, the Memory Resources category and the Cache Allocation attribute. This value determines how much memory is available to the MicroKernel for accessing data files. Note: Cache Allocation is not used for building indexes.

Increasing your Cache Allocation to a high value does not help the indexes build faster and may even slow the process by taking up crucial memory that is now unavailable to the rebuild process going on outside the cache. When rebuilding large files, your best bet is to decrease the cache value to a low value, leaving as much memory as possible available to the index rebuild process.

Calculating depth of B-tree index (optional) You can calculate the depth of a B-tree index, but the formula is a bit more complex. You first must calculate the following parameters:

Page Overhead is the number of bytes of overhead per file page. This value equals 12. Key Density is the ratio of the space used by key values in a file page to the total space in the page. Key Density = 1 if the key has recently been created with Create Index operation. Key Density = .65 if Index Balancing is turned on. Key Density = .55 if Index Balancing is turned off.

14 Unique Keys is the ratio of key values that are unique to the total number of key values for a given key. Unique Keys = 1 if the key type is Linked Duplicate. Unique Keys = unknown otherwise. This value depends on the contents of your database. Keys per Page is the number of key values contained in an index page. Keys per Page = Absolute Value ( ( ( Page Size – Page Overhead ) / ( Key Length + Key Overhead ) ) * Key Density ) Once you have determined all of the above values, you can use the following formula to determine the number of levels in your B-tree:

If (Number of records in file * Unique Keys < Keys per Page) then the number of levels in the b-tree index is 1.

If (Number of records in file * Unique Keys < (Keys per Page + Keys per Page^2)) the number of levels in the b-tree index is 2.

If (Number of records in file * Unique Keys < (Keys per Page + Keys per Page^2 + Keys per Page^3)) the number of levels in the b-tree index is 3.

You may extrapolate this formula as needed to calculate further B-tree levels.

Trademark Information MicroKernel Database Architecture, MicroKernel Database Engine, Navigational Client/Server and Scalable SQL are trademarks and Btrieve is a registered trademark of Pervasive Software Inc.

Novell and NetWare are registered trademarks of Novell, Inc.

Windows, Windows NT and Visual Basic are trademarks and Microsoft is a registered trademark of Microsoft Corporation.

Other brands and product names are the property of their respective owners.

For more information about Pervasive.SQL 2000i, pricing or support, please visit our web site at http://www.pervasive.com or locate one of our sales offices worldwide at http://www.pervasive.com/company/contact/index.asp.

© 2001-2002 Pervasive Software Inc., Pervasive, Pervasive.SQL, Btrieve, and the Pervasive logo are registered trademarks of Pervasive Software Inc. All other product names are trademarks of their respective companies. All rights reserved worldwide.

Disclaimer: The performance demonstrated in this document is for reference purposes only and does not constitute a performance guarantee even if under similar configurations.

15