USING the SAS @DATA STEP LANGUAGE to READ Dbase III FILES Michael C
Total Page：16
File Type：pdf, Size：1020Kb
Host Systems and Environments 217 USING THE SAS @DATA STEP LANGUAGE TO READ dBASE III FILES Michael C. Harris ARC Information Systems Division Abstract The dBASE III file fannat is arguably the most enduring • Date fields with data stored in YYYYMMDD in the PCIMS-DOS world. Originated by Ashton-Tate in format. the 19805, it has won widespread support among third party vendors of both enduser products and application • Logical fields hold BooJean data. They are repre development tools. The tenn "xBase" has been coined sented in xBase as one byte character fields that to describe third-party products supporting the dBASE may contain only the values "t", "T", "y", "Y", f file fonnat. Today, specialized research databases are n"', "N", "f' or "F '. available in the dBASE .DBF tbnnat. They may contain many tens of thousands ofrecords and occupy multiple • Memo fields are variable length character strings and megabytes of disk space. This paper discusses a tech are not stored in the .DBF file. The .DBF file itself nique for reading DBF files into SAS data sets in the contains a ten byte pointer into another specially PCIMS-DOS and UNIX Sun-OS environments. The structured file having the same name as the .DBF concepts may be applicable to other operating systems as file and a .DBT extension. The xBase language au well. tomatically creates and manages .DBT files and many users are not aware of their existence. Be Introduction cause the actual data can have maximum sizes of Native support for the dBASE file format exists in PC four kilobytes to sixty-four kilobytes depending SAS as PROC DBF. This procedure reads and writes upon implementation, handling memo fields in SAS both dBASE III and dBASE IV files. However, it cannot can be very difficult. read files containing more than 32,767 observations, a constraint that limits the usefulness of the procedure The dBASE ill File Structure when reading files several megabytes in size. The data A dBASE file consists of a variable length binary header step code we will discuss was originally written to followed by fixed length data records. The header overcome this limitation in the PCIMS-DOS environ consists of two types of records: ment. An inquiry from a version 6.07 SAS user operat ing under Sun-OS prompted us to port the code to that • One thirty-two byte record containing a file identi operating system. Briefly, the task at band was to read fier, the date the file was last modified, the number large (ten megabytes or more) files in dBASE III .DBF of records. the size of each record and an unused fonnat from CD-ROMs into SAS data sets. Note that area. PROC DBF exists only in PC-SAS. dBASE Field Types Fields in the xBase world are analogous to variables in • A variable number of thirty-two byte field descriptor SAS. records containing field names, types, lengths, and While only character and numeric data exist in SAS. decimal places for numeric fields. xBase supports a rich variety of data types. With one exception however, they can all be represented in SAS as Each data record is preceded by a one- byte field xBase characters or numerics. Here are the data types that most uses internally to mark records as logically deleted. xBase products support, along with some of their dBASE files are unlike most data we are accustomed to characteristics. reading with SAS in that there are no "lines" ofdata: The filesaresimplystringsofbytes. Theyarenotrawdataflles. • Character fields contain alphanumeric data. The They resemble SASdatasetsinthatthey containdataand maximum length is implementation dependent. all the infonnationyouneedto read it correctIy, provided you knowhowtodeciphertheheader. Tables 1 and2showthe • Numeric fields hold up to 19 bytes including structure of each component of the dBASE header. positions for the sign and decimal point. NESUG '92 Proceedings 218 Host Systems and Environments Table 1: Initial 32 Byte Header be discarded. The second section is a code generator. It reads each field descriptor record, extracts variable ~ Contents Length names and adjusts them if necessary, extracts field 00 .DBF ID (00 or 83 hex) 1 widths and types and constructs intormats, and writes a 01 year of last update 1 data step program to a tile. The .OBF header is read into 02 month of last update 1 one or more variables and dropped, along with the field 03 day of last update 1 indicating logical deletion. We initially attempted to 04 number of records 4 ignore the header and simply begin reading the dBASE 08 header length 2 file beyond the header using @ pointer controls. That OA record size 2 didn't work, because it wasn't possible to advance the OC unused 20 pointer beyond byte 32,767. Table 2: Field Descriptors After writing all the necessary statements to a file, that file is %INCLUDEO into the current program and run. ~ Contents Length We considered writing macro code but decided this 00 field name 11 approach would be more tlexible. if the user is going to OB field type 1 be processing several files baving the same structure, OC unused 4 which is not unusual, the data step code contained in the 10 field length 1 file we wrote can easily be reused by changing the 11 field decimals 1 names of the input and output files. It isn't necessary to 12 unused 14 run the entire program again. Implementation Notes ~esign Goals Below are two sets of source code, one for PClMS-DOS, Our primary goal was to create a program that would and the other for Sun OS. The most obvious difference read dBASE files and represent data types not native to between them is in the handling of binary fields two or SAS in the same maD.Iler as PROC OBF. Here are the more bytes in length. The 80x86 t3.mi.ly of microproces representations we chose: sors for which dBASE was originally written stores integers with the least significant byte at the lower • dBASE character fields become SAS character address, the reverse of Sun's family ofRISC processors: variables. Therefore, header fields COI1taining the number of records, the header length, and the record length must • dBASE numeric fields become SAS nmneric have thejr byte ordering reversed on the Sun workstation. variables. dBASE flIes of seven or more fields have headers greater than two hundred bytes long. We checkthe header length • <!BASE date fields become SAS dates. anddynamicallygenerate asmanyvariablenamesand informat5as we need to read it. Atthe same time, we create • dBASE logical fields become SAS character vari a list ofvariables to drop and add each newly created header ables. variable name to the list • Memo fields will not be supported. A warning Field names in dBASE can be up to ten bytes long, while will be issued if memo fields are found, but the valid SAS variable names are limited to eight bytes. fields will not become part of the SAS data set. We handle this incompatibility by checking the length of Also, we chose to ignore the field indicating logical each field name. if it is greater than eight bytes, we deletion and read every record into the new SAS truncate and compare it to a list Ofall variable names data set. processed thus far. If truncation creates a duplicate name, we dynamically create a new name by concatenat ing the character string 'VAR' to the alphanumeric The Strategy representation of the variable's position in the program This program consists of two parts. The first section data vector. We notify the of any changes in reads the. first thirty-two bytes of the header and det« user mines how many SAS variables need to be created, variable names. including variables to hold the entire header, wbich will NESUG '92 Proceedings Host Systems and Environments 219 COnclusion Acknowledgements It has been said that the SAS data step language can be SAS is a registered trademark of SAS Institute Inc., used for reading any kind of data. We developed a Cary NC, USA. program for reading structured biliary data in the form of dBASE ill files using PC-SAS and ported it to a new dBASE and dBASE III are registered trademarks of operating system \¥ith minimal effort. The techniques Borland!Ashton-Tate. employed may be useful for reading other types of files as well. Further Information For further infonnation, the author can be reached at: References 1. Professional SAS Programming Secrets McGraw-Hili, Atlantic Research Corp. Aster/Seidman Information Systems Division 1301 Piccard Drive, 2nd floor 2. Clipper Programming Guide, 2nd Edition Microtrend, Rockville, MD 20850 Spence (301)258-5300 3. SAS Language Guide for Personal Computers, Version 6 Edition, SAS Institute, Inc. 4. SAS Companion for the UNIX Environment and De rivatives SAS Institute, Inc. NESUG '92 Proceedings 220 Host Systems and Environments 1* NAME: DBF.SAS */ end; /* AurHORSHrP: ARCIPSG */ /* FUNCTION: This program reads */ put dbyear= dbmon= dbday= numrecs=headlen= 1* dBase III files containing any */ reclen=; 1* nwnber of records. it is a */ /* replacement for PROC DBF when the */ 1* 1* nwnber of records is greater than 32,767 */ calculate nwnber of dbf fields - they will become SAS /* ASSUMPTIONS: The user must enter a fully */ variables /* qualified filename and the file must exist and be */ */ /* a valid dBase file */ numflds=(headlen-34)/32; /* INPUTS: dBase III or dBase III Plus files *1 putnumtlds=; /* OUTPUTS: A SAS data step program and a SAS */ /* data set *1 pos=33; /* pointer variable to read field descriptors *1 /* REFERENCE: Michael C. Hanis */ /* /* the user supplies this infonnation *1 Nowthatweknowsomethingabout thedBase file, write %let fbame="; a data step program to read it properly %let dsname='''; */ filename dbfile &fname; filename pinc 'pinc.sas'; /* the file to be created by this tilepinc; /*filereffornew datastep program */ data step */ /* read and ignore the header */ options source2; /* show %INCLUDED code */ if headlen Ie 200 then do; /* if we can get the entire header into one SAS variable */ 1* Get information about the me we want to read ,.