Creating External Files Using SAS Software
Creating External Files Using SAS© Software
Clinton S. Rickards Oxford Health Plans, Norwalk, CT
ABSTRACT how often might it be changed? A one-time, quickie This paper will review the techniques for creating job may allow you to cut some corners that you external files for use with other software. This proc- wouldn’t be able to with a more dynamic or long-lived ess is the flip-side of reading external files. We will program. Where will your data come from? Can the review the business need, statements for pointing to file be made to serve more than one purpose? What and creating the file, and common options. A number environment (e.g. PC, UNIX, MVS) will your program of specific applications will be used to illustrate the run in and where will the file be used? various techniques. Also, don’t neglect understanding the source of data. INTRODUCTION What does it look like, are there idiosyncrasies you need to be aware of? How often is the source cre- The SAS system is a powerful application develop- ated? ment, data analysis, and data manipulation tool. But its structures are proprietary, making usage by other FILE DESIGN CONSIDERATIONS software difficult. This may require you to reformat Again, investing time at this stage will help you de- the data into a form that the other software can han- velop a program that fits your needs. Some ques- dle. For the purposes of this paper, we will limit our- tions to ask: Is the layout determined already? What selves to “traditional” flat files; that is, files that are is the layout? How much flexibility is there in chang- supported by the native operating system. We will ing the layout? What kind of translations and other not concern ourselves with relational databases, manipulations are needed? Does the order of the xBase, or other proprietary data storage systems; records matter? How many records are involved? SAS Institute and other vendors offer tools to work Are there different record formats in the file? Are with those systems. there group fields (i.e. arrays)? How are number fields and dates formatted? Packed? Display? Are This paper provides an overview of the tools SAS there variable areas? Will you use disk or tape? makes available to you for creating external files in a DATA step. Although other areas of SAS (the Dis- play Manager, SAS/AF, various procedures) can be POINTING TO EXTERNAL FILES used to create external files, you will eventually have There are three ways to point SAS to external files: a need to use the venerable DATA step to accom- operating system control statements, FILENAME plish this task. statements, and FILE statements.
We will first discuss the upfront planning and analy- 1. Operating system commands sis tasks you should take. Then we will examine the Some operating systems provide commands or ways you tell SAS about the file to be created. We statements that allow you to create a fileref. A fileref will describe the PUT statement, which actually (file reference) is a nickname used in place of the full writes the file, in some detail, and finally consider operating system filename. For example, MVS batch some complete examples. allows gives you the Job Control Language DD statement, and CMS has the FILEDEF command. All programs in this paper were created under SAS 6.11 running on Windows 95. The techniques pre- Generally, the fileref is created before a SAS session sented here are applicable to versions 6.06 and is started. Some platforms permit the use of the SAS higher. X statement to assign a fileref after the session has started. Please check the SAS Companion for your DETERMINE YOUR NEEDS operating system for details. A disadvantage of as- The first step toward successfully creating an exter- signing filerefs with this method is that the filerefs will nal file is to determine your needs. Spending the not be available in the Display Manager FILENAME necessary time at this stage will greatly reduce the window. number of problems you encounter later on during coding and testing. Questions to ask: What is the 2. FILENAME statement purpose of the file? Can the file serve more than one This statement creates a fileref that names a link purpose? How often will your program be run and between the physical file and the SAS system. For
1 operating systems that do not support filerefs (e.g. PC-DOS and Windows), the FILENAME statement fileref gives the fileref of an external file. The fil- or the FILE statement (below) must be used. The eref must be created before the DATA step by filename statement is executed within your SAS ses- using an operating system definition or a sion but before the DATA step begins. FILENAME statement.
The syntax of the FILENAME statement is: 'external-file' gives the name of an external file. This form is the functional equivalent of a sepa- FILENAME fileref
2 are separated by a space. It is the easiest style to 33 b = a; 34 format a mmddyy10.; code but offers the least control over where and how 35 put a= b=; the data is actually placed in the file. Consider this 36 run; example, which writes to the LOG since no FILE A=05/21/1954 B=-2051 statement is used. Note that defined formats are applied automatically. 1 data _null_; 2 a = '21may1954'd; Should you need to apply a different format, specify 3 b = a; the new format after the equal sign: 4 format a mmddyy10.; 5 put a b; 6 run; 37 data _null_; 38 a = '21may1954'd; 05/21/1954 -2051 39 b = a; 40 format a mmddyy10.; 41 put a=yymmdd8. b=; If a format has been defined for a variable, it is ap- 42 run; plied automatically before the data is written; other- A=54-05-21 B=-2051 wise the BEST format is used for numeric data or the $ format for character. The special keyword _all_ used in a put statement instructs SAS to write all variables in the DATA step List out also allows you to write constant data sur- using named put. rounded by single or double quotes. Note, however, that there is no blank following the constant text. EXAMPLES Assume that you need to create a file containing the 1 data _null_; 2 a = '21may1954'd; name, birth date, and gender of the students in a 3 b = a; class and that the data are stored in a SAS data set. 4 format a mmddyy10.; 5 put a A complete program (under Windows 6.11) would 6 "THIS IS A CONSTANT" be: 7 b ; 8 run; 126 filename rstrfile 'a:\file1.dat'; 05/21/1954 THIS IS A CONSTANT-2051 127 data _null_; 128 set roster; 129 file rstrfile; COLUMN PUT specifies the columns to be written 130 put name birthdte gender; 131 run; for each variable. For example: NOTE: The file RSTRFILE is: 9 data _null_; FILENAME=a:\file1.dat, 10 a = '21may1954'd; RECFM=V,LRECL=256 11 b = a; 12 format a mmddyy10.; NOTE: 3 records were written to the file 13 put a 1-10 b 11-20; RSTRFILE. 14 run; The minimum record length was 9. The maximum record length was 11. -2051 -2051 NOTE: The DATA statement used 0.22 seconds. Note that is a variable’s defined format is ignored when using column put. As you can see, SAS places a great deal of useful information in the log for you. The resulting file looks FORMATTED PUT includes a format for the vari- like this: ables. This form of put is the only form that will work Dan 157 M when non-standard display data representations Mary 74 F (e.g. packed, hexadecimal, date/time) are needed. Mary Anne 3168 M For example: This file presents some difficulties to the program 25 data _null_; that will be actually reading it. First, the placement of 26 a = '21may1954'd; 27 b = a; BIRTHDTE and GENDER varies. This could poten- 28 put a mmddyy10. tially be overcome by having the reading program 29 b yymmdd8.; 30 run; use blanks as delimiters but the record for Mary Anne has an imbedded blank. Second, the value for 05/21/195454-05-21 BIRTHDTE is meaningless unless the program knows how SAS stores date values internally. All of NAMED PUT is of the form variable=. these problems could have been avoided had the This style of put places the variable name with the source data set ROSTER carried the proper formats. data, as in: Since it doesn’t, this is a better program:
31 data _null_; 170 filename rstrfile 'a:/file2.dat'; 32 a = '21may1954'd; 171 data _null_;
3 172 set roster; @ moves the pointer to the column number result- 173 file rstrfile; 174 put name $10. ing from expression. expression can be a nu- 175 birthdte mmddyy10. meric constant, a numeric variable, or a formula 176 gender $1. 177 ; enclosed in parenthesis. For example, @44, 178 run; @START, @(START+COUNT*5) are valid ex- NOTE: The file RSTRFILE is: pressions. FILENAME=a:\file2.dat, RECFM=V,LRECL=256 + moves the pointer left or right the number of col- NOTE: 3 records were written to the file umns resulting from expression. expression can RSTRFILE. be a numeric constant, a numeric variable, or a The minimum record length was 21. The maximum record length was 21. formula enclosed in parenthesis. Expressions NOTE: The DATA statement used 0.11 seconds. resulting in positive numbers move the pointer to the right, negative results move the pointer to the Dan 06/06/1960M Mary 03/15/1960F left. Negative numeric constants must be en- Mary Anne 09/03/1968M closed in (). For example, +5, +START, +(-5), and +(START+COUNT*5) are valid expressions. MISSING NUMERIC DATA When SAS encounters missing numeric data, what it These examples accomplish the same task. Note in writes depends on the format being used and on the example 3 that PEARS is written before APPLES by MISSING= system option. If the format has an using the +5 and +(-10) column pointer controls. “other” or “low” defined, that value is used. Most of- 1. PUT @1 VENDOR $CHAR20. ten, MISSING=” “ or MISSING=”.” is used, telling @20 APPLES 5. SAS to write that character in place of missing val- @25 PEARS 5.; ues. Reading programs may or may not have diffi- 2. START = 1; culties with blanks but most will have difficulties with PUT @START VENDOR $CHAR20. periods, so consider this carefully. APPLES 5. PEARS 5.; ADVANCED PUT STATEMENT 3. PUT VENDOR $20. +5 PEARS 5. FEATURES +(-10) APPLES 5.; The last program manifests a couple of additional problems with the put techniques we have examined LINE pointer controls: so far. First, the location of GENDER is not immedi- #expression ately obvious - you must add the formatted lengths of / NAME and BIRTHDTE to determine where GENDER begins. Second, if BIRTHDTE is not # moves the pointer to a specific line number needed but GENDER must remain in its current lo- based on the result of expression. expression cation, you must consume precious resources writ- can be a numeric constant, a numeric variable, ing 10 spaces. Although acceptable in that small ex- or a formula enclosed in parenthesis. ample, these problems could be unmanageable when writing large files. This section provides the / moves the pointer to the next line. You can use tools to overcome these and other problems. multiple / if needed.
POINTER CONTROLS These PUT statements are functionally equivalent: SAS maintains two pointers, the column pointer 1. PUT #1 @1 LASTNAME $CHAR25. and the line pointer, to keep track of where data will #2 @16 PHONENUM 9. be written during the execution of a PUT statement. #4 @30 CITY $25.; You can change these pointers, for example, to re- 2. PUT @ 1 LASTNAME $CHAR25. write data, change the order in which data is written, / @16 PHONENUM 9. or to create logical records that are defined by multi- // @30 CITY $25.; ple physical records. Use of these pointer controls 3. CITYLOC = 4; also makes the PUT statement more self- PUT LASTNAME $CHAR25. #2 @16 PHONENUM 9. documenting. #CITYLOC @30 CITY $25.;
COLUMN pointer controls: LINE HOLD SPECIFIER @expression A line hold specifier is used to "hold" the current line +expression in the external file through multiple PUT statements. Placed at the end of the PUT statement, it instructs SAS to not write a new record when the next PUT
4 statement is executed. This capability is the key formatted put will ensure that the data is written ex- element of techniques used to writing more complex actly as I want it to be written without any surprises files or to improving efficiency. caused by the default processing SAS could take. Finally, the code is self-documenting to some extent @ (trailing at-sign) tells SAS to keep this record by fully defining where the data is and its structure. current until another PUT is executed, even if that However, you must choose the techniques appropri- PUT is in a different iteration of the DATA step. The ate to your applications. record is written when a PUT statement without a trailing @ is processed, the column pointer moves FILE STATEMENT OPTIONS past the end of the line, or the DATA step ends. For FILE has many options, some specific to the host example, this program writes different data depend- operating system and some generic to any SAS ap- ing on the results of a merge operation. plication. In this section, we will explore the more commonly used SAS options. data _null_; file trash; merge policy (in=a) DROPOVER, FLOWOVER, STOPOVER are mutu- agents (in=b); by agtcode; ally exclusive options that define what action SAS is if a; to take when the program attempts to write past the put polnum @; if b then end of a record. put agtcode agtstats; DROPOVER tells SAS to drop remaining variables else from the record and to continue processing the put agtcode "NOT FOUND"; run; DATA step.
The @@ (double trailing at-sign) is equivalent to @. FLOWOVER tells SAS to continue writing vari- ables on succeeding lines. This is the default ac- WARNING: When using @, care must be taken to tion. Note that a new record will be written when ensure that the record is released when you want it SAS attempts to write past the end of the record released. It is easy to attempt to write past the end of even if a trailing @ on the PUT statement is used. the record or to only write one record. For example, this program will write only 1 record to the file STOPOVER tells SAS to stop processing the STREETS, when data set ADDRESS is exhausted. It DATA step after writing the current record. SAS will have the street of the last observation in sets _ERROR_ equal to 1. ADRESS. N=n tells SAS how many lines to make available to DATA _null_; the PUT statement. Set this value if there is more set address; file streets; than 1 record to be written at a time. The default is 1. put street $ 1-20 @; The line pointer controls / and # would be used. See run; Example 5.
Remember that using line hold specifiers may re- LRECL=, BLKSIZE=, RECFM= are used to describe quire you to either 1) explicitly end execution of the the logical record length, block size, and record for- DATA step with a STOP statement; or 2) release the mat, respectively, of the external file. These options line with a PUT statement without line hold a speci- control the overall size and structure of your file. fier. MOD, OLD are mutually exclusive options that tell GROUPING VARIABLES AND FORMATS SAS to append records to the file (MOD) or to re- You may group variables and formats into lists to place the existing file (OLD). OLD is the default. reduce the size of the PUT statement. SAS recycles the format list until the variable list is exhausted. This The following options are generally used when cre- technique, shown in Program 3, is particularly useful ating reports in a DATA step. Although this paper when you are writing arrays or numbered variables. doesn’t deal with this aspect of DATA step pro- gramming, they are presented for completeness. COMPARISON OF METHODS In general, I tend to use formatted put in conjunction LINESIZE= sets the number of columns available to with column and line pointer controls. Some types of SAS. LRECL= sets the overall record length but data (e.g. packed decimal, signed numeric, hexa- LINESIZE= tells SAS how much of that line to use. decimal, or dates) can be written only with formatted put and this technique gives me maximum control PAGESIZE= sets the number of lines per page in over the writing of the data without excessive worry- your report. After reaching this limit, SAS automati- ing about coding errors in the data. In addition, the cally moves to line 1 of the next page.
5 471 put @25 deptcode $5.; 472 totrecs + 1; /* count records */ NOTITLES, PRINT|NOPRINT controls whether title 473 return; lines are printed and whether printer carriage control 474 475 control: /* write rest of control record */ characters are used, respectively. Usually, 476 totrecs + 1; /* count records */ NOTITLES and NOPRINT are used when creating 477 put @1 '9' /* control record type */ 478 @2 totrecs z9.; files and PRINT is used for reports. 479 return; HEADER= defines a label that is executed whenever 480 run; SAS begins a new output page. NOTE: The file OUTTOOLS is: FILENAME=a:\out2.dat, SPECIFIC APPLICATIONS RECFM=V,LRECL=256 NOTE: 3 records were written to the file OUTOOLS. PROGRAM 1: CREATING A BASIC FIXED The minimum record length was 10. FORMAT FILE The maximum record length was 29. NOTE: The data set WORK.TOOLS has 1 observations This example presents the author’s favorite tech- and 9 variables. nique: using pointer controls and formatted puts to NOTE: The DATA statement used 0.17 seconds. achieve maximum control over the placement and format of the data. A date record (sometimes called a header record) is created the first time through the DATA step. Every file outtools 'a:\outfile1.dat'; record is written to the file. Note the line-hold speci- data tools; set nesug.tools; fier after WRKSALDT. This allows the substitution of file outtools; DEPTCODE in place of CLERK when needed. Also put @1 item $char10. @11 quantity 5. note the slightly convoluted means to achieve a date @16 saledate mmddyy8. in ccyymmdd format; SAS does not supply a format @22 clerk $char5. @22 clrkgrp 2. matching this need. When the end of @24 initials $char3. ; WRITING.TOOLS is reached then a control record is run; written.
PROGRAM 2: CREATING A MULTIPLE FORMAT PROGRAM 3: REPEATING FIELD PATTERNS FILE Program 3 demonstrates one way to write a file with This program illustrates the use of the line hold repeated fields. Each record has an ACCOUNT fol- specifier @. The output data has a date record, then lowed by 4 occurrences of a set of repeated fields several detail records, and a final control record, pertaining to cash flow into the policy. All fields are each with a different layout. The first character of separated by blanks. each record identifies the record type. 545 filename out3 439 filename outtools 546 'a:\out3.dat'; 440 'c:\data\clint\writing-files\out2.dat'; 547 data _null_; 441 data tools; 548 file out3 noprint notitles; 442 set writing.tools end=eof; 549 set writing.sales; 443 file outtools noprint notitles; 550 put @1 policy $char8. 444 if _n_ eq 1 then 551 @10 (fund1-fund3) ($char3. +14) 445 link daterec; 552 @14 (date1-date3) (yymmdd6. +11) 446 553 @21 (amount1-amount3) (5.2 12); 447 link detail; 554 run; 448 449 if eof then NOTE: The file OUT3 is: 450 link control; FILENAME=a:\out3.dat, 451 return; RECFM=V,LRECL=256 452 453 daterec: /* write rest of date record */ NOTE: 1 record was written to the file OUT3. 454 created = compress( The minimum record length was 59. put(today(),mmddyy10.),'/'); The maximum record length was 59. 455 asofdate = compress( NOTE: At least one W.D format was too small for put(today()-1,mmddyy10.),'/'); the number to be printed. The decimal may be 456 put @1 '0' /* date record type */ shifted by the "BEST" format. 457 @2 created $8. NOTE: The DATA statement used 0.33 seconds. 458 @10 asofdate $8.; 459 totrecs + 1; /* count records */ 460 return; The PUT statement writes the data from the vari- 461 462 detail: /* write rest of detail record */ ables by using grouped format lists. Grouped for- 463 wrksaldt = compress( mat lists are a very compact way to write repeating put(saledate,mmddyy10.),'/'); fields because the format lists are recycled until all of 464 put @1 '1' /* detail record type */ 465 @2 item $char10. the variables are exhausted. Grouped format lists 466 @12 quantity z5. consist of 2 lists, each enclosed in parentheses: the 467 @17 wrksaldt $8. @; 468 if clerk ne ' ' then first is the list of variables and the second is the for- 469 put @25 clerk $char5.; mat list to be recycled. The format lists can also in- 470 else
6 clude column pointer controls, vital in this case due 928 #2 @33 agent $40. 929 #4 @65 state $3. ; to the intermixing of the data. 930 run;
NOTE: The file 'a:\out5.dat' is: To illustrate, consider the fund information. FUND1 FILENAME=A:\out5.dat, is written using the $CHAR3. format and then the RECFM=V,LRECL=72 +14 moves the pointer past DATE1, AMOUNT1, and NOTE: 4 records were written to the file the intervening blanks. The pointer is now positioned 'a:\out5.dat'. at the data for FUND2. Looking at the variable list, The minimum record length was 0. The maximum record length was 72. SAS then writes FUND2 using the recycled format NOTE: The DATA statement used 3.18 seconds. list, which leaves the pointer at the data for FUND3. This recycling continues until the variable list is ex- The #1, #2, and #4 line pointer controls moves the hausted. Pointer controls are not used after the last pointer to those lines before the next part of the PUT variable in the list is written. SAS then processes the statement is executed. Line 3 is completely blank next part of the put, which writes the date and amount fields in similar fashion. PROGRAM 6: COMMA DELIMITED, QUOTED TEXT FILE The program also illustrates that care must be taken This type of file is frequently referred to as a CSV when deciding on the formats to use. In this case, file, or comma separated values, file. You must in- the variable AMOUNT3 has a value 100, which when clude the quotes that surrounds text fields and the formatted with 2 decimal places (100.00) cannot fit commas that separate the variables as constant text. into 5 characters. SAS substitutes the BEST format in order to give you as much precision as possible. data _null_; file 'a:\out6.dat' noprint notitles; Unfortunately, the note does not tell you the name of length policy $15 agent $4 state $2; the offending variable; you have to figure that out on informat issuedte yymmdd8.; your own. input policy issuedte agent state; workdate = put(compress(put(issuedte,mmddyy10.),'/'),$8.); PROGRAM 4: REPEATING FIELD PATTERNS put '"' policy $ +(-1) '",' workdate ',' This program shows how to create a file with re- '"' agent $ +(-1) '",' peating fields from a SAS data set with one observa- '"' state $ +(-1) '"'; tion for each group of fields. The data elements are cards; JOHN 19950228 A123 CT separated by blanks. Note the use of trailing-@ to ALFRED 19950515 A459 MA hold the record until an account break is reached. run;
801 filename sales 'a:\outfile4.dat'; "JOHN",02281995 ,"A123","CT" 802 data _null_; "ALFRED",05151995 ,"A459","MA" 803 file sales noprint notitles lrecl=80; 804 set writing.fundinfo; 805 by account; The use of the +(-1) column pointer controls is nec- 806 if first.account then essary since list put is being used. In Observation 1, 807 put @1 account $char8. @; for example, the length of variable POLICY is 4 808 put +1 fund $char3. 809 +1 date yymmdd6. characters. SAS writes the 4 characters then 1 810 +1 amount 5.2 @ ; blank, leaving the column pointer at 6. The +(-1) 811 if last.account then 812 put; /* release the record */ moves the column pointer back to 5 before the dou- 813 run; ble quote and comma are written.
NOTE: The file SALES is: "JOHN",02281995 ,"A123","CT" FILENAME=a:\outfile4.dat, "ALFRED",05151995 ,"A459","MA" RECFM=V,LRECL=80 NOTE: 2 records were written to the file SALES. The minimum record length was 76. CONCLUSION The maximum record length was 76. This paper has only touched upon the basics of cre- NOTE: The DATA statement used 1.32 seconds. ating external files using SAS software. Obtain and use the SAS appropriate documentation for your PROGRAM 5: MULTIPLE RECORDS PER system. There are also numerous NESUG and SUGI OBSERVATION papers that will be helpful. The author highly recom- This program segment writes a policy information file mends Marge Scerbo’s paper “DATA Step Report- that spans multiple physical records. ing” from NESUG 95 as an excellent introduction to that topic. 921 data _null_; 922 file 'a:\out5.dat' 923 noprint notitles n=4 Any given file layout can probably have several solu- 924 lrecl=72 blksize=7200; 925 set writing.policy; tions that will work, each with their own advantages 926 put #1 @7 policy $8. and disadvantages. Take the time to play and ex- 927 @28 issuedte yymmdd8. periment with different techniques. 7 Electronic mail may be sent to the author at [email protected].
SAS is a registered trademark of SAS Institute Inc., Cary, NC
References:
SAS Language: Reference, Version 6.06, First Edi- tion, SAS Institute, Inc., Cary, NC
SAS Technical Report P-222, Changes and En- hancements to Base SAS Software. Release 6.07, SAS Institute, Inc., Cary, NC
SAS Technical Report P-242, SAS Software: Changes and Enhancements. Release 6.08, SAS Institute, Inc., Cary, NC
Scerbo, Marge, “DATA Step Reporting”, NESUG Proceedings, 1995
8