GPERF A Perfect Hash Function Generator
Douglas C. Schmidt [email protected]
http://www.cs.wustl.edu/ schmidt/ Department of Computer Science Washington University, St. Louis 63130
1 Introduction a sample input keyfile; Section 4 highlights design patterns and implementation strategies used to develop gperf; Sec- Perfect hash functions are a time and space efficient imple- tion 5 shows the results from empirical benchmarks between mentation of static search sets. A static search set is an ab- gperf-generated recognizers and other popular techniques stract data type (ADT) with operations initialize, insert,and for reserved word lookup; Section 6 outlines the limita- retrieve. Static search sets are common in system software tions with gperfand potential enhancements; and Section 7 applications. Typical static search sets include compiler and presents concluding remarks. interpreter reserved words, assembler instruction mnemonics, shell interpreter built-in commands, and CORBA IDL compil- ers. Search set elements are called keywords. Keywords are 2 Static Search Set Implementations inserted into the set once, usually off-line at compile-time. gperf is a freely available perfect hash function generator There are numerous implementations of static search sets. written in C++ that automatically constructs perfect hash func- Common examples include sorted and unsorted arrays and tions from a user-supplied list of keywords. It was designed in linked lists, AVL trees, optimal binary search trees, digital the spirit of utilities like lex [1] and yacc [2] to remove the search tries, deterministic finite-state automata, and various drudgery associated with constructing time and space efficient hash table schemes, such as open addressing and bucket chain- keyword recognizers manually. ing [5].
gperf translates an n element list of user-specified key- Different static search structure implementations offer words, called the keyfile, into source code containing a k ele- trade-offs between memory utilization and search time effi- ment lookup table and the following pair of functions: ciency and predictability. For example, an n element sorted array is space efficient. However, the average- and worst-case
hash uniquely maps keywords in the keyfile onto the
time complexity for retrieval operations using binary search
1 k n k = n
range 0..k ,where .If hash is con-
log n on a sorted array is proportional to O [5]. sidered a minimal perfect hash function. In contrast, chained hash table implementations locate a ta-
in word set uses hash to determine whether a par- 1 ble entry in constant, i.e., O , time on the average. However, ticular string of characters occurs in the keyfile,usingat hashing typically incurs additional memory overhead for link
most one string comparison in the common case. pointers and/or unused hash table buckets. In addition, hash-
2
n gperf is designed to run quickly for keyfiles containing ing exhibits O worst-case performance [5]. several thousand keywords. gperf generates efficient ANSI A minimal perfect hash function is a static search set imple- and K&R C and C++ source code as output. It has been used to mentation defined by the following two properties:
generate reserved keyword recognizers in lexical analyzers for 1 The perfect property: Locating a table entry requires O several production and research compilers and language pro- time, i.e., at most one string comparison is required to perform cessing tools, including GNU C/C++ [3] and the TAO CORBA keyword recognition within the static search set. IDL compiler [4]. This paper is organized as follows: Section 2 outlines The minimal property: The memory allocated to store the alternative static search set implementations and compares keywords is precisely large enough for the keyword set and no them with gperf-generated hash tables; Section 3 presents larger.
1 %{ Minimal perfect hash functions provide a theoretically op- #include
Near-perfect hash functions: Near-perfect hash functions do not possess the perfect property since they allow non- Figure 1: An Example Keyfile for Months of the Year unique keyword hash values [9] (they may or may not possess the minimal property, however). This technique is a compro- mise that trades increased generated-code-execution-time for month, as well as the months’ ordinal numbers, i.e., january = decreased function-generation-time. Near-perfect hash func- 1,february=2,...,december = 12. tions are useful when main memory is at a premium since they gperf’s input format is similar to the UNIX utilities lex tend to produce much smaller lookup tables than non-minimal and yacc. It uses the following input format: perfect hash functions. declarations and text inclusions gperf can generate minimal perfect, non-minimal perfect, %% keywords and optional attributes and near-perfect hash functions, as described below. %% auxiliary code
3 Interacting with GPERF A pair of consecutive % symbols in the first column sepa- rate declarations from the list of keywords and their optional This section explains how end-users can interact with gperf. attributes. C or C++ source code and comments are included By default, gperf reads a keyword list and optional associ- verbatim into the generated output file by enclosing the text in- ated attributes from the standard input keyfile.Keywords side %{ %} delimiters, which are stripped off when the output are specified as arbitrary character strings delimited by a user- file is generated, e.g.: specified field separator that defaults to ’,’. Thus, keywords may contain spaces and any other ASCII characters. Associ- %{ #include
2 An optional user-supplied struct declaration may be however, describe the design and implementation of a general- placed at the end of the declaration section, just before the %% purpose perfect hashing generator tool in detail. This section separator. This feature enables “typed attribute” initialization. describes the data structures, algorithms, output format, and For example, in Figure 1 struct months is defined to have reusable components in gperf.
four fields that correspond to the initializer values given for the gperf is written in 4,000 lines of C++ source code. C++ month names and their respective associated values, e.g.: was chosen as the implementation language since it supports struct months { data abstraction better than C, while maintaining C’s efficiency char *name; and expressiveness [16]. int number; int days; gperf’s three main phases for generating a perfect or near- int leap_days; perfect hash function are shown in Figure 2: Figure 6 illus- }; gperf %% trates ’s overall program structure. and described be- Lines containing keywords and associated attributes appear KEYFILE GPERF C/C++ CODE in the keywords and optional attributes section of the keyfile. january Key_List int hash The first field of each line always contains the keyword itself, february (const char *str, unsigned int len) left-justified against the first column and without surround- ... { ing quotation marks. Additional attribute fields can follow december Asso_Values // ... the keyword. Attributes are separated from the keyword and from each other by field separators, and they continue up to the gperf “end-of-line marker,” which is the newline character (’\n’) by Figure 2: ’s Processing Phases default. low: Attribute field values are used to initialize components of the user-supplied struct appearing at the end of the decla- 1. Process command-line options, read keywords and at- ration section, e.g.: tributes (the input format is described in Section 3), and january, 1, 31, 31 initialize internal objects (described in Section 4.1). february, 2, 28, 29 march, 3, 31, 31 2. Perform a non-backtracking, heuristically guided search ... for a perfect hash function (described in Section 4.2.1 and Section 4.2.2). As with lex and yacc, it is legal to omit the initial dec- laration section entirely. In this case, the keyfile begins with 3. Generate formatted C or C++ code according to the the first non-comment line (lines beginning with a "#" char- command-line options (output format is described in Sec- acter are treated as comments and ignored). This format style tion 4.3). is useful for building keyword set recognizers that possess no The following section outlines gperf’s perfect hash func- associated attributes. For example, a perfect hash function for tion generation algorithms and internal objects, examines its frequently occurring English words can efficiently filter out generated source code output, describes several reusable class uninformative words, such as “the,” “as,” and “this,” from con- components, and discusses the program’s current limitations sideration in a key-word-in-context indexing application [5]. and future enhancements. Again, as with lex and yacc, all text in the optional third auxiliary code section is included verbatim into the generated output file, starting immediately after the final %% and extend- 4.1 Internal Objects ing to the end of the keyfile. It is the user’s responsibility to gperf’s implementation centers around two internal objects: ensure that the inserted code is valid C or C++. In the Fig- the keyword signatures list (Key List)andtheassociated ure 1 example, this auxiliary code provides a test driver that is values array (asso values), both of which are described conditionally included if the DEBUG symbol is enabled when below. compiling the generated C or C++ code. 4.1.1 The Keyword Signatures List 4 Design and Implementation Strate- Every user-specified keyword and its attributes are read from gies the keyfile and stored in a node on a linked list, called Key List. gperf only considers a subset of each key- Many articles describe perfect hashing [10, 7, 11, 12] and min- words’ characters while it searches for a perfect hash function. imal perfect hashing algorithms [8, 13, 6, 14, 15]. Few articles, The subset is called the “keyword signature,” or keysig.
3 The keysig represents the particular subset of characters hash_value = asso_values[keyword[0]] used by the automatically generated recognition function to + asso_values[keyword[1]] compute a keyword’s hash value. Keysigs are created and + asso_values[keyword[length - 1]] cached in each node in the Key List when the keyfile is ini- + length; tially processed by gperf. Developers can control the generated hash function’s con- tents using the "-k" option to explicitly specify the keyword 4.1.2 Associated Values Array index positions used as keysig elements by gperf. The de- The associated values array, asso values, is an object that fault is "-k 1,$",wherethe’$’ represents the keyword’s is closely related to keysigs. In fact, it is indexed by keysig final character. characters. The array is constructed internally by gperf and Table 1 shows the keywords, keysigs, and hash value for referenced frequently while gperf searches for a perfect hash each month shown in the Figure 1 keyfile. These keysigs were function. During the C/C++ code generation phase of gperf,an Keyword Keysig Hash Value january an 3 ASCII representation of the associated array is output in the february be 9 static generated hash function as a local array. This array march ar 4 is declared as u int asso values[MAX ASCII SIZE]. april pr 2 When searching for a perfect hash function, gperf repeatedly may ay 8 reassigns different values to certain asso values elements june nu 1 specified by keysig entries. At every step during the search july lu 6 for the perfect hash function solution, the asso values ar- august gu 7 ray’s contents represent the current associated values’ config- september ep 0 uration. october ct 10 When configured to produce minimal perfect hash functions november ov 11 (which is the default), gperf searches for an associated val- december ce 5
ues configuration that maps all n keysigs onto non-duplicated hash values. A perfect hash function is produced when gperf Table 1: Keywords, Keysigs, and Hash Values for the Months finds a configuration that assigns each keysig to a unique loca- Example tion within the generated lookup table. The resulting perfect
hash function returns an unsigned int value in the range produced using the -k2,3 option.
:: k 1 k = maximum k ey w or d hash v al ue +1
0 ,where ( ). Keysigs are multisets since they may contain multiple oc-
= n
When k a minimal perfect hash function is produced. currences of certain characters. This approach differs from
n n For k larger than , the lookup table’s load factor is
k other perfect hash function generation techniques [8] that only number of k ey w or ds
( ). +
total table siz e consider first/last characters length when computing a key- A keyword’s hash value is typically computed by combin- word’s hash value. ing the associated values of its keysig with its length.1 By The hash function generated by gperf properly handles default, the hash function adds the associated value of a key- keywords shorter than a specified index position by skipping word’s first index position plus the associated value of its last characters that exceed the keyword’s length. In addition, users index position to its length, i.e.: can instruct gperf to include all of a keyword’s characters in its keysig via the "-k*" option. hash_value = asso_values[keyword[0]] + asso_values[keyword[length - 1]] + length; 4.2 Generating Perfect Hash Functions This subsection describes how gperf generates perfect hash Other combinations are often necessary in practice. For exam- functions. ple, using the default hash function for C++ reserved words causes a collision between delete and double.Toresolve this collision and generate a perfect hash function for C++ re- 4.2.1 Main Algorithm served words, an additional character must be added to the
gperf iterates sequentially through the list of i keywords,
keysig, as follows:
i n n 1 ,where equals the total number of keywords. 1The "-n" option instructs gperf not include the length of the keyword During each iteration gperf attempts to extend the set of when computing the hash function. uniquely hashed keywords by 1. It succeeds if the hash value
4
i 1 computed for keyword i does not collide with the previous collision between january and march by incrementing uniquely hashed keywords. Figure 3 outlines the algorithm. asso value[’n’] by 1. As shown in Table 2, this is its
final value.
1
for i to n loop Keysig Associated Frequency of
th st st
1 i 1 if hash (i key) collides with any hash ( key ... key) Characters Values Occurrence then ’a’ 2 3 modify disjoint union of associated values to resolve collisions ’b’ 9 1 based upon certain collision resolution heuristics ’c’ 5 2 end if ’e’ 0 3 end loop ’g’ 7 1 ’l’ 6 1 Figure 3: Gperf’s Main Algorithm ’n’ 1 2 ’o’ 1 1 The algorithm terminates and generates a perfect hash func- ’p’ 0 2
’r’ 2 2
= n tion when i and no unresolved hash collisions remain. ’t’ 5 1
Thus, the best-case asymptotic time-complexity for this algo- ’u’ 0 3
n rithm is linear in the number of keywords, i.e., . ’v’ 0 1 ’y’ 6 1 4.2.2 Collision Resolution Strategies As outlined in Figure 3, gperf attempts to resolve keyword Table 2: Associated Values and Occurrences for Keysig Char- hash collisions by incrementing certain associated values. The acters following discusses the strategies gperf uses to speedup col- lision resolution. Search heuristics: gperf uses several search heuristics to reduce the time required to generate a perfect hash function. Disjoint union: To avoid performing unnecessary work, For instance, characters in the disjoint union are sorted by in- gperf is selective when changing associated values. In par- creasing frequency of occurrence, so that less frequently used ticular, it only considers characters comprising the disjoint characters are incremented before more frequently used char-
union of the colliding keywords’ keysigs. The disjoint union acters. This strategy is based on the assumption that incre-
Ag fB g fA [ B g fA\Bg of two keysigs f and is defined as .menting infrequently used characters first decreases the nega- To illustrate the use of disjoint unions, consider the key- tive impact on keywords that are already uniquely hashed with words january and march from Figure 1. These key- respect to each other. Table 2 shows the associated values and words have the keysigs ‘‘an’’ and ‘‘ar’’, respectively, frequency of occurrences for all the keysig characters in the as shown in Table 1. Thus, when asso values[’a’], months example. asso values[’n’],andasso values[’r’] all equal 2 0, a collision will occur during gperf’s execution. To re- gperf generates a perfect hash function if increments to solve this collision, gperf only considers changing the as- the associated values configuration shown in Figure 3 and de- sociated values for ’n’ and/or ’r’. Changing ’a’ by any scribed above eliminate all keyword collisions when the end increment cannot possibly resolve the collision since ’a’ oc- of the Key List is reached. The worst-case asymptotic time-
curs the same number of times in each keysig. 3
n l l complexity for this algorithm is O ,where is the number By default, all asso values are initialized to 0. When a of characters in the largest disjoint union between colliding collision is detected gperf increases the corresponding asso- keyword keysigs. After experimenting with gperf on many ciated value by a “jump increment.” The command-line option keyfiles it appears that such worst-case behavior occurs rarely "-j" can be used to increase the jump increment by a fixed in practice. or random amount. In general, selecting a smaller jump incre- Many perfect hash function generation algorithms [6, 7] are "-j 1" ment, e.g., decreases the size of the generated hash sensitive to the order in which keywords are considered. To gperf table, though it may increase ’s execution time. mitigate the effect of ordering, gperf will optionally reorder In the months example in Figure 1, the "-j 1" op- keywords in the Key List if the "-o" command-line op- tion was used. Therefore, gperf quickly resolves the tion is enabled. This reordering is done in a two-stage pre- 2Note that since the "-n" option is used in the months example, the dif- pass [8] before gperf invokes the main algorithm shown in ferent keyword lengths are not considered in the resulting hash function. Figure 3. First, the Key List is sorted by decreasing fre-
5 quency of keysig characters occurrence. The second reorder- #include
HASH VA L U E RANGE = TOTAL KEYWORDS and DUPLI- if (len <= MAX_WORD_LENGTH
&& len >= MIN_WORD_LENGTH) { 0 CATES = .Anon-minimal perfect hash function oc- int key = hash (str, len);
if (key <= MAX_HASH_VALUE > curs when DUPLICATES = 0andHASH VA L U E RANGE && key >= MIN_HASH_VALUE) { TOTAL KEYWORDS. Finally, a near-perfect hash function char *s = wordlist[key].name;
if (*str == *s occurs when DUPLICATES > 0andDUPLICATES TO- && !strcmp (str + 1, s + 1)) TAL KEYWORDS. return &wordlist[key]; } } return 0; 4.3.2 The Generated Lookup Table } By default, when gperf is given a keyfile as input it at- tempts to generate a perfect hash function that uses at most Figure 4: Minimal Perfect Hash Function Generated by one string comparison to recognize keywords in the lookup ta- gperf ble. gperf can implement the lookup table either an array or a switch statement, as described below.
6 { asso values array lookup table: gperf generates an const struct months *rw; array by default, emphasizing run-time speed over minimal switch (key) memory utilization. This array is called asso values,as { shown in the hash function in Figure 4. The asso values case 0: rw = &wordlist[0]; break; case 1: rw = &wordlist[1]; break; array is used by the two generated functions that compute hash case 2: rw = &wordlist[2]; break; values and perform table lookup. case 3: rw = &wordlist[3]; break; case 4: rw = &wordlist[4]; break; gperf also provides command-line options that allow de- case 5: rw = &wordlist[5]; break; velopers to select trade-offs between memory size and execu- case 6: rw = &wordlist[6]; break; case 7: rw = &wordlist[7]; break; tion time. For example, expanding the range of hash values case 8: rw = &wordlist[8]; break; produces a sparser lookup table. This generally yields faster case 9: rw = &wordlist[9]; break; case 10: rw = &wordlist[10]; break; keyword searches but requires additional memory. case 11: rw = &wordlist[11]; break; The array-based asso values scheme works best when default: return 0; } the HASH VA L U E RANGE is not considerably larger than the if (*str == *rw->name TOTAL KEYWORDS. When there are a large number of key- && !strcmp (str + 1, rw->name + 1)) return rw; words, and an even larger range of hash values, however, the return 0; wordlist array in is month function in Figure 4 may be- } come extremely large. Several problems arise in this case:
Figure 5: The switch-based Lookup Table
The time to compile the sparsely populated array is ex- cessive; 4.3.3 The Generated Functions
The array size may be too large to store in main memory; gperf generates a hash function and a lookup function. By A large array may lead to increased “thrashing” of virtual memory in the OS. default, they are called hash and in word set, although a different name may be given for in word set using the "-N" command-line option. Both functions require two ar- guments, a pointer to a NUL-terminated (’\0’) array of Switch-based lookup table: To handle the problems de- characters, const char *str, and a length parameter, scribed above, gperf can also generate one or more switch unsigned int len. statements to implement the lookup table. Depending on the underlying compiler’s switch optimization capabilities, the The generated hash function (hash): Figure 4 shows the switch-based method may produce smaller and faster code, hash function generated from the input keyfile shown in Fig- compared with the large, sparsely filled array. Figure 5 shows ure 1. The command-line option "-k 2,3" was enabled for how the switch statement code appears if the months exam- this test. This instructs hash to return an unsigned int
ple is generated with gperf’s "-S 1" option. hash value that is computed by using the ASCII values of the
nd rd 3 Since the months example is somewhat contrived, the trade- 2 and characters from its str argument into the local off between the array and switch approach is not particularly static array asso values.3 The two resulting numbers obvious. However, good C++ compilers generate assembly are added to calculate str’s hash value. code implementing a “binary-search-of-labels” scheme if the The asso values array is generated by gperf using the switch statement’s case labels are sparse compared to the algorithm in Section 4.1.2. This array maps the user-defined range between the smallest and largest case labels [3]. This keywords onto unique hash values. All asso values array technique can save a great deal of space by not emitting un- entries with values greater than MAX HASH VA L U E (i.e.,all necessary empty array locations or jump-table slots. The exact the “12’s” in the asso values array in Figure 4) represent time and space savings of this approach varies according to the ASCII characters that do not occur as either the second or third underlying compiler’s optimization strategy. characters in the months of the year. The is month function gperf generates source code that constructs the array or in Figure 4 uses this information to quickly eliminate input switch statement lookup table at compile-time. Therefore, strings that cannot possibly be month names. initializing the keywords and any associated attributes requires Generated lookup function (in word set): The little additional execution-time overhead when the recognizer in word set function is the entry point into the perfect function is run. The “initialization” is automatically per- hash lookup function. In contrast, the hash function is formed as the program’s binary image is loaded from disk into main memory. 3Note that C arrays start at 0, so str[1] is actually the second character.
7 declared static and cannot be invoked by application programs directly. If the function’s first parameter, char ASSO *str, is a valid user-define keyword, in word set returns GEN a pointer to the corresponding record containing each keyword VALUES PERF and its associated attributes; otherwise, a NULL pointer is returned. Figure 4 shows how the in word set function can be KEY renamed to is month using the "-N" command-line op- GPERF LIST tion. Note how gperf checks the len parameter and COMPONENTS resulting hash function return value against the symbolic constants for MAX WORD LENGTH, MIN WORD LENGTH, ACE MAX HASH VA L U E,andMIN HASH VA L U E. This check COMPONENTS quickly eliminates many non-month names from further con- BOOL sideration. If users know in advance that all input strings are READ HASH ARRAY valid keywords, gperf will suppress this addition checking with the "-O" option. BUFFER TABLE SINGLETON If gperf is instructed to generate an array-based lookup table the generated code is quite concise, i.e., once it is deter- mined that the hash value lies within the proper range the code is simply: Figure 6: gperf’s Software Architecture
{ char *s = wordlist[key]; This one function, Gen Perf::affects previous,de- if (*s == *str && !strcmp (str + 1, s + 1)) termines how changes to associated values affect previously return s; hashed keywords. In particular, it identifies duplicate hash val- } ues that occur during program execution. Since this function is called so frequently, it is im- The *s == *str expression quickly detects when the portant to minimize its execution overhead. Therefore, computed hash value indexes into a “null” table slot since *s gperf employs a novel boolean array component called is the NUL character (’\0’) in this case. This check is useful ACE Bool Array to expedite this process. The C++ inter- when searching a sparse keyword lookup table, where there is face for the ACE Bool Array class is depicted in Figure 7. a higher probability of locating a null entry. If a null entry is Since only one copy is required, BOOL ARRAY is typedef’d located, there is no need to perform a full string comparison. to be a Singleton using the ACE Singleton adapter. This Since the months’ example generates a minimal perfect template automatically transforms a class into a Singleton us- hash function null enties never appear. The check is still ing the Singleton and Adapter patterns [18]. useful, however, since it avoids calling the string comparison The in set method efficiently detects duplicate keyword function when the str’s first letter does not match any of the hash values for a given associated values configuration. It re- keywords in the lookup table. turns non-zero if a value is already in the set and zero other- wise. Whenever a duplicate is detected, the reset method 4.4 Reusable Components and Patterns is called to reset all the array elements back to “empty” for ensuing iterations of the search process. Figure 6 illustrates the key components used in gperf’s If many hash collisions occur, the reset method is exe- software architecture. gperf is constructed from reusable cuted frequently during the duplicate detection and elimina- components from the ACE framework [17]. Each compo- tion phase of gperf’s algorithm. Processing large keyfiles, nent evolved “bottom-up” from special-purpose utilities into e.g., containing more than 1,000 keywords, tends to require
reusable software components. Several noteworthy reusable n a maximum hash value k that is often much larger than , classes include the following components: the total number of keywords. Due to the large range, it be- ACE Bool Array: Earlier versions of gperf were instru- comes expensive to explicitly reset all elements in array mented with a run-time code profiler on large input key- back to empty, especially when the number of keywords actu- files that evoke many collisions. The results showed that ally checked for duplicate hash values is comparatively small. gperf spent approximately 90 to 99 percent of its time in To address this issue, gperf uses a pattern called gener- a single function when performing the algorithm in Figure 3. ation numbering, which optimizes the search process by not
8 class ACE\_Bool_Array { unsigned short integer, which occurs infrequently in public: practice. // Constructor. ACE\_Bool_Array (void); A design strategy employed throughout gperf’s imple- // Returns dynamic memory to free store. ˜ACE_Bool_Array (void); mentation is “first determine a clean set of operations and in- terfaces, then successively tune the implementation.” In the // Allocate a k element dynamic array. init (u_int k); case of generation numbering, this policy of optimizing per- formance, without compromising program clarity, decreased // Checks if
9 Executable Input File
Program ET++.in NIH.in g++.in idraw.in cfront.in libg++.in
j j j j j
control.exe 38.8 j 1.00 15.4 1.00 15.2 1.00 8.9 1.00 5.7 1.00 4.5 1.00
j j j j j
trie.exe 59.1 j 1.52 23.8 1.54 23.8 1.56 13.7 1.53 8.6 1.50 7.0 1.55
j j j j j
flex.exe 60.5 j 1.55 23.9 1.55 23.9 1.57 13.8 1.55 8.9 1.56 7.1 1.57
j j j j j
gperf.exe 64.6 j 1.66 26.0 1.68 25.1 1.65 14.6 1.64 9.7 1.70 7.7 1.71
j j j j j
chash.exe 69.2 j 1.78 27.5 1.78 27.1 1.78 15.8 1.77 10.1 1.77 8.2 1.82
j j j j j
patricia.exe 71.7 j 1.84 28.9 1.87 27.8 1.82 16.3 1.83 10.8 1.89 8.7 1.93
j j j j j
binary.exe 72.5 j 1.86 29.3 1.90 28.5 1.87 16.4 1.84 10.8 1.89 8.8 1.95
j j j j j comp-flex.exe 80.1 j 2.06 31.0 2.01 32.6 2.14 18.2 2.04 11.6 2.03 9.2 2.04
Table 3: Raw and Normalized CPU Processing Time g++ . The function returns 1 if a given input string is identi- automata (DFA)-based recognizers. Not using compaction fied as a reserved word and 0 otherwise. maximizes speed in the generated recognizer, at the ex- The seven test programs are described below. They are pense of much larger tables. For example, the uncompacted listed by increasing order of execution time, as shown in Ta- flex.exe program is almost 5 times larger than the com- ble 3. The input files used for the test programs are described pacted comp-flex.exe program, i.e., 117,808 bytes versus in Table 4. Table 5 shows the number of bytes for each test 24,416 bytes.
Input File Identifiers Keywords Total gperf.exe: a gperf-generated recognizer created with the ET++.in 624,156 350,466 974,622 "-a -D -S 1 -k 1,$" options. These options mean NIH.in 209,488 181,919 391,407 “generate ANSI C prototypes ("-a"), handle duplicate key- g++.in 278,319 88,169 366,488 words ("-D"), via a single switch statement ("-S 1"), and idraw.in 146,881 74,744 221,625 make the keysig be the first and last character of each key- cfront.in 98,335 51,235 149,570 word.” libg++.in 69,375 50,656 120,031 chash.exe: a dynamic chained hash table lookup function similar to the one that recognizes reserved words for AT&T’s Table 4: Total Identifiers and Keywords for Each Input File cfront 3.0 C++ compiler. The table’s load factor is 0.39, the cfront program’s compiled object file, listed by increasing size (both same as it is in 3.0. patricia.o and chash.o use dynamic memory, so their patricia.exe: a PATRICIA trie recognizer, where PATRICIA overall memory usage depends upon the underlying free store stands for “Practical Algorithm to Retrieve Information Coded mechanism). in Alphanumeric.” A complete PATRICA trie implementation is available in the GNU libg++ class library distribution [19]. Object Byte Count File text data bss dynamic total binary.exe: a carefully coded binary search function that control.o 88 0 0 0 88 minimizes the number of complete string comparisons. binary.o 1,008 288 0 0 1,296 gperf.o 2,672 0 0 0 2,672 comp-flex.exe: a flex-generated recognizer created with chash.o 1,608 304 8 1,704 3,624 the default "-cem" options, providing the highest degree patricia.o 3,936 0 0 2,272 6,208 of table compression. Note the obvious time/space trade- comp-flex.o 7,920 56 16,440 0 24,416 trie.o 79,472 0 0 0 79,472 off between the uncompacted flex.exe (which is faster flex.o 3,264 98,104 16,440 0 117,808 and larger) and the compacted comp-flex.exe (which is smaller and much slower). Table 5: Size of Object Files in Bytes In addition to these seven test programs, a simple C++ pro- gram called control.exe measures and controls for I/O trie.exe: a program based upon an automatically generated overhead, i.e.: table-driven search trie created by the trie-gen utility included with the GNU libg++ distribution. int main (void) { char buf[BUFSIZ]; flex.exe: a flex-generated recognizer created with the while (gets (buf)) "-f" (no table compaction) option. Note that both the printf ("%s", buf); flex.exe and trie.exe are uncompacted, deterministic finite }
10 All of the above reserved word recognizer programs were piler. It is more difficult, on the other hand, to partially in- compiled by the GNU g++ 2.7.2 compiler with the "-O2 tegrate flex or lex into a lexical analyzer since they are -finline-functions" options enabled. They were then generally used in an “all or nothing” fashion. Furthermore, tested on an otherwise idle SPARCstation 20 model 712 with neither flex nor lex are capable of generating recognizers 128 megabytes of RAM. extremely large keyfiles because the size of the state machine All six input files used for the tests contained a large is too big for their internal DFA state tables. number of words, both user-defined identifiers and g++ re- served words, organized with one word per line. This for- mate was automatically created by running the UNIX com- 6 Current Limitations and Future mand "tr -cs A-Za-z_ ’\012’" on the preprocessed source code for several large C++ systems, including the Work ET++ windowing toolkit (ET++.in), the NIH class library (NIH.in), the GNU g++ 2.7.2 C++ compiler (g++.in), the gperf has been freely distributed for many years along with idraw figure drawing utility from the InterViews 2.6 distri- the GNU libg++ library and the ACE network programming bution (idraw.in), the AT&T cfront 3.0 C++ compiler toolkit at www.cs.wustl.edu/˜schmidt/ACE.html. (cfront.in), and the GNU libg++ 2.8 C++ class library Although gperf has proven to be quite useful in practice, (libg++.in). Table 4 shows the relative number of iden- there are several limitations. This section describes the trade- tifiers and keywords for the test input files. offs and compromises with its current algorithms and outlines Table 3 depicts the amount of time each search set imple- how it can be improved. Since gperf is open source soft- mentation spent executing the test programs, listed by increas- ware, however, it is straightforward to add enhancements and ing execution time. The first number in each column repre- extensions. sents the user-time CPU seconds for each recognizer. The sec- ond number is “normalized execution time,” i.e., the ratio of 6.1 Tradeoffs and Compromises user-time CPU seconds divided by the control.exe pro- gram execution time. The normalized execution time for each Several other hash function generation algorithms utilize some technique is very consistent across the input test file suite, il- form of backtracking when searching for a perfect or minimal lustrating that the timing results are representative for different perfect solution [6, 8, 9]. For example, Cichelli’s [8] algorithm source code inputs. recursively attempts to find an associated values configuration
Several conclusions result from these empirical bench- that uniquely maps all n keywords to distinct integers in the ::n marks: range 1 . In his scheme, the algorithm “backs up” if com- puting the current keyword’s hash value exceeds the minimal Time/space tradeoffs are common: The uncompacted, perfect table size constraint at any point during program exe- DFA-based trie (trie.exe)andflex(flex.exe)imple- cution. Cichelli’s algorithm then proceeds by undoing selected mentations are both the fastest and the largest implementa- hash table entries, reassigning different associated values, and tions, illustrating the time/space trade-off dichotomy. Appli- continuing to search for a solution. cations where saving time is more important than conserving Unfortunately, the exponential growth rate associated with space may benefit from these approaches. the backtracking search process is simply too time consum- gperf can provide the best of both worlds: While the ing for large keyfiles. Even “intelligently-guided” exhaustive trie.exe and flex.exe recognizers allow programmers search quickly becomes impractical for more than several hun- to trade-off space for time, the gperf-generated perfect hash dred keywords. function gperf.exe is comparatively time and space ef- To simplify the algorithm in Figure 3, and to improve ficient. Empirical support for this claim can be calculated average-case performance, gperf does not backtrack when from the data for the programs that did not allocate dy- keyword hash collisions occur. Thus, gperf may process the namic memory, i.e., trie.exe, flex.exe, gperf.exe, entire keyfile input, without finding a unique associated values binary.exe,andcomp-flex.exe. The number of iden- configuration for every keyword, even if one exists. If a unique tifiers scanned per-second, per-byte of executable program configuration is not found, users have two choices: overhead was 5.6 for gperf.exe, but less than 1.0 for trie.exe, flex.exe,andcomp-flex.exe. 1. They can run gperf again, enabling different options in search of a perfect hash function; or Since gperf generates a stand-alone recognizer, it is eas- ily incorporated into an otherwise hand-coded lexical analyzer, 2. They can guarantee a solution by instructing gperf to such as the ones found in the GNU C and GNU C++ com- generate an near-perfect hash function.
11 Near-perfect hash functions permit gperf to operate on and secondary keys. In the latter case, if the primary keywords keyword sets that it otherwise could not handle, e.g.,ifthe are distinguishable only via secondary key comparisons, the keyfile contains duplicates or there are a very large number of user may edit the generated code by hand or via an automated keywords. Although the resulting hash function is no longer script to completely disambiguate the search key. “perfect,” it handles keyword membership queries efficiently 4 since only a small number of duplicates usually remain. 6.2 Enhancements and Extensions Both duplicate keyword entries and unresolved keyword collisions are handled by generalizing the switch-based Fully automating the perfect hash function generation process scheme described in Section 3. gperf treats duplicate is gperf’s most significant unfinished extension. One ap- keywords as members of an equivalence class and gener- proach is to replace gperf’s current algorithm with more ex- ates switch statement code containing cascading if-else haustive approaches [9, 7]. Due to gperf’s object-oriented comparisons within a case label to handle non-unique key- program design, such modifications will not disrupt the overall word hash values. program structure. The perfect hash function generation mod- For example, if gperf is run with the default keysig selec- ule, class Gen Perf, is independent from other program tion command-line option "-k 1,$" on a keyfile contain- components; it represents only about 10 percent of gperf’s ing C++ reserved words, a hash collision occurs between the overall lines of source code. delete and double keywords, thereby preventing a perfect A more comprehensive, albeit computationally expensive, hash function. Using the "-D" option produces a near-perfect approach could switch over to a backtracking strategy when hash function, that allows at most one string comparison for the initial, computationally less expensive, non-backtracking all keywords except double, which is recognized after two first pass fails to generate a perfect hash function. For many comparisons. Figure 8 shows the relevant fragment of the gen- common uses, where the search sets are relatively small, the erated near-perfect hash function code. program will run successfully without incurring backtracking overhead. In practice, the utility of these proposed modifica- { char *rw; tions remains an open question. ... Another potentially worthwhile feature is enhancing switch (hash (str, len)) { ... gperf to automatically select the keyword index positions. case 46: This would assist users in generating time or space efficient rw = "delete"; if (*str == *rw hash functions quickly and easily. Currently, the user must && !strcmp (str + 1, rw + 1, len - 1)) use the default behavior or explicitly select these positions via return rw; command-line arguments. Finally, gperf’s output functions rw = "double"; if (*str == *rw can be extended to generate code for other languages, e.g., && !strcmp (str + 1, rw + 1, len - 1)) Java, Ada, Smalltalk, Module 3, Eiffel, etc. return rw; return 0; case 47: rw = "default"; break; case 49: 7 Concluding Remarks rw = "void"; break; ... gperf was originally designed to automate the construction } if (*str == *rw of keyword recognizers for compilers and interpreter reserved && !strcmp (str + 1, rw + 1, len - 1)) word sets. The various features described in this paper en- return rw; return 0; able it to achieve its goal, as evidenced by its use in the GNU } compilers. In addition, gperf has been used in the following applications: Figure 8: The Near-Perfect Lookup Table Fragment
The TAO CORBA IDL compiler [4] uses gperf to gen- A simple linear search is performed on duplicate keywords erate the operation dispatching tables [21] used by server- that hash to the same location. Linear search is effective side skeletons. since most keywords still require only one string comparison. A hash function for 15,400 “Medical Subject Headings” Support for duplicate hash values is useful in several circum- used to index journal article citations in MEDLINE, a stances, such as large input keyfiles (e.g., dictionaries), highly large bibliographic database of the biomedical literature similar keyword sets (e.g., assembler instruction mnemonics), maintained by the National Library of Medicine. Gener- 4The exact number depend on the keyword set and the command-line op- ating this hash function takes approximately 10 minutes tions. of CPU time on a SPARC 20 workstation.
12 The GNU indent C code reformatting program, where [12] M. Dietzfelbinger, A. Karlin, K. Mehlhorn, F. M. auf der Heid, the inclusion of perfect hashing sped up the program by H. Rohnert, and R. Tarjan, “Dynamic Perfect Hashing: Up- an average of 10 percent. per and Lower Bounds,” SIAM Journal of Computing, vol. 23, pp. 738–761, Aug. 1994.
A public domain program converting double precision [13] G. Jaeschke, “Reciprocal Hashing: A Method for Generating FORTRAN source code to/from single precision uses Minimal Perfect Hashing Functions,” Communications of the gperf to modify function names that depend on the ACM, vol. 24, pp. 829–833, Dec. 1981. types of their arguments, e.g., replacing sgefa with [14] T. Sager, “A Polynomial Time Generator for Minimal Per- dgefa in the LINPACK benchmark. Each name cor- fect Hash Functions,” Communications of the ACM, vol. 28, responding to a function is recognized via gperf and pp. 523–532, Dec. 1985. substituted with the version for the appropriate precision. [15] C. C. Chang, “A Scheme for Constructing Ordered Minimal Perfect Hashing Functions,” Information Sciences, vol. 39,
A speech synthesizer system, where there is a cache be- pp. 187–195, 1986. tween the synthesizer and a larger, disk-based dictionary. A word is hashed using gperf, and if the word is already [16] Bjarne Stroustrup, The C++ Programming Language, 3 rd Edi- tion. Addison-Wesley, 1991. in the cache it is not looked up in the dictionary. [17] D. C. Schmidt, “ACE: an Object-Oriented Framework for
Since automatic static search set generators perform well in Developing Distributed Applications,” in Proceedings of the th practice and are widely and freely available, there seems little 6 USENIX C++ Technical Conference, (Cambridge, Mas- sachusetts), USENIX Association, April 1994. incentive to code keyword recognition functions manually for most applications. [18] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Pat- terns: Elements of Reusable Object-Oriented Software. Read- ing, MA: Addison-Wesley, 1995.
References [19] D. Lea, “libg++, the GNU C++ Library,” in Proceedings of the st
1 C++ Conference, (Denver, CO), pp. 243–256, USENIX, [1] M. Lesk and E. Schmidt, LEX - A Lexical Analyzer Generator. Oct. 1988. Bell Laboratories, Murray Hill, N.J., Unix Programmers Man- [20] J. Kegler, “A Polynomial Time Generator for Minimal Perfect ual ed. Hash Functions,” Communications of the ACM, vol. 29, no. 6, [2] S. Johnson, YACC - Yet another Compiler Compiler. Bell Lab- pp. 556–557, 1986. oratories, Murray Hill, N.J., Unix Programmers Manual ed. [21] A. Gokhale and D. C. Schmidt, “Evaluating the Performance [3] R. M. Stallman, Using and Porting GNU CC. Free Software of Demultiplexing Strategies for Real-time CORBA,” in Pro- Foundation, GCC 2.7.2 ed. ceedings of GLOBECOM ’97, (Phoenix, AZ), IEEE, November 1997. [4] A. Gokhale, D. C. Schmidt, and S. Moyer, “Tools for Automat- ing the Migration from DCE to CORBA,” in Proceedings of ISS 97: World Telecommunications Congress, (Toronto, Canada), IEEE Communications Society, September 1997. [5] D. E. Knuth, The Art of Computer Programming, vol. 1: Search- ing and Sorting. Reading, MA: Addison Wesley, 1973. [6] C. R. Cook and R. R. Oldehoeft, “A Letter Oriented Minimal Perfect Hashing Function,” SIGPLAN Notices, vol. 17, pp. 18– 27, Sept. 1982. [7] A. Tharp and M. Brain, “Using Tries to Eliminate Pattern Col- lisions in Perfect Hashing,” IEEE Transactions on Knowledge and Data Engineering, vol. 6, no. 2, pp. 329–347, 1994. [8] R. J. Cichelli, “Minimal Perfect Hash Functions Made Simple,” Communications of the ACM, vol. 21, no. 1, pp. 17–19, 1980. [9] M. Brain and A. Tharp, “Near-perfect Hashing of Large Word Sets,” Software – Practice and Experience, vol. 19, no. 10, pp. 967–978, 1989. [10] R. Sprugnoli, “Perfect hashing functions: A single probe re- trieving method for static sets,” Communications of the ACM, pp. 841–850, Nov. 1977. [11] G. V. Cormack, R. Horspool, and M. Kaiserwerth, “Practical Perfect Hashing,” Computer Journal, vol. 28, pp. 54–58, Jan. 1985.
13