A Hybrid Approach to Functional Dependency Discovery Thorsten Papenbrock Felix Naumann Hasso Plattner Institute (HPI) Hasso Plattner Institute (HPI) 14482 Potsdam, Germany 14482 Potsdam, Germany
[email protected] [email protected] ABSTRACT concrete metadata. One reason is that they depend not only Functional dependencies are structural metadata that can on a schema but also on a concrete relational instance. For be used for schema normalization, data integration, data example, child ! teacher might hold for kindergarten chil- cleansing, and many other data management tasks. Despite dren, but it does not hold for high-school children. Conse- their importance, the functional dependencies of a specific quently, functional dependencies also change over time when dataset are usually unknown and almost impossible to dis- data is extended, altered, or merged with other datasets. cover manually. For this reason, database research has pro- Therefore, discovery algorithms are needed that reveal all posed various algorithms for functional dependency discov- functional dependencies of a given dataset. ery. None, however, are able to process datasets of typical Due to the importance of functional dependencies, many real-world size, e.g., datasets with more than 50 attributes discovery algorithms have already been proposed. Unfor- and a million records. tunately, none of them is able to process datasets of real- world size, i.e., datasets with more than 50 columns and We present a hybrid discovery algorithm called HyFD, which combines fast approximation techniques with efficient a million rows, as a recent study showed [20]. Because validation techniques in order to find all minimal functional the need for normalization, cleansing, and query optimiza- dependencies in a given dataset.