Harry Version 0.4.2 — User Manual — Konrad Rieck April 16, 2016 Contents 1 NAME 2 2 SYNOPSIS 2 3 DESCRIPTION 2 4 CONFIGURATION 3 5 OPTIONS 13 6 FILES 14 7 LIMITATIONS 15 8 REFERENCES 15 9 COPYRIGHT 16 1 1 NAME harry - A tool for measuring string similarity 2 SYNOPSIS harry [options][-c config] input [input] output 3 DESCRIPTION harry is a small tool for measuring the similarity of strings. The tool supports com- mon distance and kernel functions for strings as well as some exotic similarity mea- sures. The focus of harry lies on implicit similarity measures, that is, comparison functions that do not give rise to an explicit vector space. Examples of such similarity measures are the Levenshtein distance, the Jaro-Winkler distance or the normalized compression distance. During operation harry loads a set of strings from input, computes the specified simi- larity measure and writes a matrix of similarity values to output. If one input is given, harry computes the similarities between all strings in input. If two input sources are provided, harry computes only the similarities between the two sources. The sim- ilarity measure can be computed based on the granularity of bits, bytes as well as tokens contained in the strings. The configuration of this process, such as the input format, the similarity measure and the output format, are specified in the file config and can be additionally refined using command-line options. .---------. .----------. | lines | | text | | dir | \ .---------. / | libsvm | | stdin | -- | Harry | -- | matlab | | ... | / ’---------’ \ | ... | ’---------’ ’----------’ Input Similarity Measures Output harry is implemented using OpenMP, such that the computation time for a set of strings scales linear with the number of available CPU cores. Moreover, efficient implementations of several similarity measures, effective caching of similarity values and low-overhead locking further speedup the computation. harry complements the tool sally(1) that embeds strings in a vector space and allows computing vectorial similarity measures, such as the cosine distance and the bag-of- words kernel. 2 4 CONFIGURATION The configuration of harry is provided by a configuration file. This file is structured into the three sections input, measures and output, which define the parameters of the input format, the similarity measures and the output format, respectively. If no configuration file is provided, harry resorts to a default configuration. This default configuration can be printed using the command-line option -D (see OPTIONS). Input formats harry supports different formats for reading sets of strings, which may range from plain files to directories and other structured resources. The input format is specified in the configuration file of harry, but can also be defined on the command line using the option -i (see OPTIONS). Following is a list of supported input formats: input = f input format = ”lines”; This parameter specifies the input format. ”lines” The input strings are available as lines in a text file. The name of the file is given as input to harry. The lines need to be separated by new- line and may not contain the NUL character. Labels can be extracted from each line using a regular expression (see lines regex). ”stdin” The input strings are provided on standard input (stdin) as text lines. The lines need to be separated by newline and may not contain the NUL character. Labels can be extracted from each line using a regular expression (see lines regex). This input format is also enabled when input is set to -, otherwise input is ignored. ”dir” The input strings are available as binary files in a directory and the name of the directory is given as input to harry. The suffixes of the files are used as labels for the strings. ”arc” The input strings are available as binary files in a compressed archive, such as a zip or tgz archive. The name of the archive is given as input to harry. The suffixes of the files are used as labels for the strings. ”fasta” The input strings are available in FASTA format. The name of the file is given as input to harry. Labels can be extracted from the description of each sequence using a regular expression (see fasta regex). Com- ments are allowed if they are preceded by either ’;’ or ’>’. ”raw” The input strings are provided on standard input (stdin) in ”raw” format. This input module is designed to efficiently interface with other environments. The binary format for strings has the form 3 | len (uint32) | array (uint8) ... | | len (uint32) | array (uint8) ... | | ... where len is a 32-bit unsigned integer in host byte order indicating the length of the following array containing the string data in bytes. La- bels cannot be extracted from this representation. This input format is also enabled when input is set to =, otherwise input is ignored. chunk size = 256; To enable an efficient processing of large data sets, harry loads strings in chunks. This parameter defines the number of strings in one of these chunks. Depending on the lengths and type of the strings, this parameter can be adjusted to improve loading times. decode str = false; If this parameter is set to true, harry automatically decodes strings that contain URI-encoded characters. That is, substrings of the form %XX are replaced with the byte corresponding to the hexadecimal number XX. fasta regex = ” (nn+j-)?[0-9]+”; The FASTA format allows to equip each string with a short description. In several data sets this description contains a numerical label which can be used for supervised learning tasks. The parameter defines a regular expression that matches these numerical labels, such as +1 and -1. lines regex = ”ˆ(nn+j-)?[0-9]+”; If the strings are available as text lines, the parameter can be used to ex- tract a numerical label from the strings. The parameter is a regular ex- pression matching labels, such as +1 and -1. reverse str = false; If this parameter is set to true, the characters of all input strings will be reversed. Such reversing might help in situations where the reading di- rection of the input strings is unspecified. stoptoken file = ””; Stop tokens (irrelevant tokens) can be filtered from the strings by provid- ing a file containing these tokens; one per line. Non-printable characters can be escaped using URI encoding (%XX). Stop tokens can only be fil- tered, if the granularity is set to tokens. soundex = false; All tokens in the strings are mapped to the soundex index. For example, ”Pfister” is mapped to ”P236” and ”Jackson” to ”J250”. The soundex in- dex has been originally designed for comparing names, however, in harry it can be applied to all sorts of tokens, if they are composed of alphabetic letters. Punctation characters are ignored and thus the string ”Hey, I am here with Harry!”, gets mapped to ”H000 I000 A500 H600 W300 H600”. g; 4 Similarity Measures harry supports different similarity measures for comparing string, including com- mon distance and kernel functions. The similarity measure can be specified in the configuration file as well as on the command line using the option -m (see OP- TIONS). The name of each similarity measure is prefixed by its type (dist for dis- tances, kern for kernels and sim for similarity measures). For convenience, this pre- fix can be omitted. Moreover, the names of some similarity measures are aliased, for example, the normalized compression distance is available as dist compression and dist ncd. Parameters of the similarity measures are organized in individual configuration groups. For instance, parameters of the Levenshtein distance are defined in the group dist levenshtein, while parameters for the Jaro and Jaro-Winkler distance are given in dist jarowinkler. measures = f type = ”dist levenshtein” The parameter type specifies the similarity measure that is used for com- paring the strings. Supported similarity measures are: ”dist hamming” Hamming distance. See configuration group dist hamming. ”dist levenshtein”, ”dist edit” Levenshtein distance. See configuration group dist levenshtein. ”dist damerau” Damerau-Levenshtein distance. See configuration group dist damerau. ”dist osa” Optimal string alignment (OSA) distance. See configuration group dist osa. ”dist jaro” Jaro distance. See configuration group dist jarowinkler. ”dist jarowinkler” Jaro-Winkler distance. See configuration group dist jarowinkler. ”dist lee” Lee distance. See configuration group dist lee ”dist compression”, ”dist ncd” Normalized compression distance (NCD). See configuration group dist compression. ”dist bag” Bag distance. See configuration group dist bag. ”dist kernel” Kernel substitution distance. See configuration group dist kernel. ”kern subsequence”, ”kern ssk” Subsequence kernel (SSK). See configuration group kern subsequence. ”kern spectrum”, ”kern ngram” Spectrum kernel (also n-gram kernel). See configuration group kern spectrum. 5 ”kern wdegree”, ”kern wdk” Weighted-degree kernel (WDK) with shifts. See configuration group kern wdegree. ”kern distance”, ”kern dsk” Distance substitution kernel (DSK). See configuration group kern distance. ”sim simpson” Simpson coefficient. See configuration group sim coefficient. ”sim jaccard” Jaccard coefficient. See configuration group sim coefficient. ”sim braun” Braun-Blanquet coefficient. See configuration group sim coefficient. ”sim dice”, ”sim czekanowski” Dice-coefficient (Czekanowsi coefficient) See configuration group sim coefficient. ”sim sokal”, ”sim anderberg” Sokal-Sneath coefficient (Anderberg coefficient). See configuration group sim coefficient. ”sim kulczynski” Second Kulczynski coefficient. See configuration