Inductive Programming – Example-driven construction of functional programs

Ute Schmid, Martin Hofmann, Emanuel Kitzelmann We developed an efficient, analytical approach for learning recursive functional programs from examples. Igor2 is a realization of this approach as constructor term rewriting system. We can show that our approach compares very well to other systems of inductive programming and we will discuss applications in the domains of cognitive modelling, end-user programming and automated software engineering.

1 Introduction programs: In a first step, input/output examples were rewritten into traces, in a second step recurrent patterns were searched-for Inductive programming research addresses the problem of learn- in the traces and the found regularities were generalised to a re- ing computer programs from incomplete specifications, typically cursive function. Thesys was restricted to structural problems samples of the desired input/output behaviour and possibly addi- on lists such as unpacking elements or reverting. Due to that tional constraints (Biermann, Guiho, & Kodratoff, 1984; Flener restriction rewriting of input/output examples could be realized & Schmid, to appear). Induced programs are typically in a with a deterministic algorithm where inputs where characterised declarative – functional or logic – programming language. As by the list structure using empty as only predicate. Folding of a special application of (Mitchell, 1997), in- traces was restricted to linear based on Summers’ syn- ductive programming creates program hypotheses, that is, gen- thesis theorem which stated that a Kleene sequence of traces eralised, typically recursive, programs. In contrast to classifica- can be interpreted as being generated by the kth sequence of tion learning, program hypotheses must cover all given exam- unfoldings of the searched-for linear recursive program. ples correctly since for programs it is expected that a desired For Thesys and all later analytical approaches it is enough input/output relation holds for all possible inputs. In contrast to present a small set of only positive input/output examples. to deductive approaches to automated program construction, These examples must be the first representants of the underlying there is no guarantee that the induced program meets the in- data-type of the input parameter. In contrast, in search-based tension of the user. Inductive programming approaches can be approaches, an arbitrary set of positive examples can be pre- characterised by learning biases, especially by a language bias sented. Typically, more examples than for analytical approaches which restricts the syntactic form of programs which can be are necessary. In addition negative examples can be used to characterised by a program scheme and by a search bias which eliminate unsuitable hypotheses. determines the sequence in which hypotheses are constructed. An extension of Thesys to a larger class of recursive pro- In contrast to deductive approaches, the barrier for application is grams is Igor1 (Schmid & Wysotzki, 1998; Schmid, 2003; lower since the system user is not required to become an expert Kitzelmann & Schmid, 2006). Rewriting of input/output ex- in the domain of formal specifications. Therefore, inductive pro- amples allowed nested expressions and folding was guided by a gramming approaches are good candidates for the development more general program scheme allowing for linear and tree recur- of programming assistants (Flener & Partridge, 2001). sions and especially for inducing sets instead of single recursive There are two general approaches to inductive programming: functions. Another analytical system in the context of ILP is analytical and search-based methods. Search-based methods Dialogs (Flener, 1996). Here synthesis relies on interaction enumerate syntactically correct programs and test these pro- with a user who selects a suitable program scheme for the given grams against the given examples guided by some search strat- problem. The most recent analytical approach to inductive pro- egy. The inductive (ILP) system Foil con- gramming is Igor2 which will be presented in the following. structs sets of Horn clauses by sequential covering, that is by a first construct then test approach. This approach was also ap- plied to learning recursive rule sets (Quinlan & Cameron-Jones, 2 IGOR2 1995). The most successful search-based system is Adate (Ols- son, 1995) which constructs ML programs using evolutionary Igor2 (Kitzelmann & Schmid, 2007; Kitzelmann, 2008; Kitzel- principles. The system MagicHaskeller (Katayama, 2005) mann & Hofmann, 2008; Kitzelmann, 2009) is a new analytical enumerates Haskell programs with higher-order functions. approach to inductive programming which overcomes inherent Analytical methods are guided by the structure underlying limitations of systems in the Thesys tradition, includes pow- the given input/output examples. Beginning of research on in- erful concepts introduced in inductive logic programming, and ductive programming in the nineteen-seventies was concerned relates to the state of art in languages. with such analytical strategies for learning Lisp programs from The two step approach of Thesys results in an inherent bottle- small sets of positive input/output examples (Biermann et al., neck to due to the first step – rewriting in- 1984). The most influential of the early systems is Thesys put/output examples to traces. A rewriting algorithm relies on a (Summers, 1977). It realized a two step approach to synthesise set of predefined predicates to discern different inputs and on a

Page 1 set of predefined operators such as list constructors and decon- (* specification omitted *) eq Reverse([]) = [] structors to express the construction of outputs from the inputs. eq Reverse(cons(X, [])) = cons(X,[]) Only if both sets are kept very small and if discerning of in- eq Reverse(cons(X, cons(Y, []))) = cons(Y, cons(X, [])) puts is possible without nested conditions an efficient algorithm eq Reverse(cons(X, cons(Y, cons(Z, [])))) = with unique results can be defined. In consequence, folding of cons(Z, cons(Y, cons(X, []))) traces into recursive programs is restricted to such simple trace eq Reverse(cons(X, cons(Y, cons(Z, cons(V, []))))) = structures also. To allow the generation of more complex traces, cons(V, cons(Z, cons(Y, cons(X, [])))) the rewriting algorithm needs to employ more complex pattern Reverse([]) = [] recognition abilities and partially has to rely on the same infer- Reverse(cons(X0,X1)) = cons(Sub1(cons(X0,X1)), ence mechanisms which are used in the folding step of synthesis. Reverse(Sub2(cons(X0,X1)))) Igor2 therefore relies on a single-step algorithm to generalise input/output examples to recursive programs. (* last *) The idea of allowing the use of background knowledge in Sub1(cons(X0, [])) = X0 Sub1(cons(X0, cons(X1, X2))) = Sub1(cons(X1, X2)) learning as it was introduced in ILP (Muggleton & De Raedt, 1994) was also successfully applied in ILP approaches to induc- (* init *) tive programming (Muggleton & Feng, 1990). More complex Sub2(cons(X0,[])) = [] programs can be learned by providing information of functions Sub2(cons(X0, cons(X1, X2))) = cons(X0, Sub2(cons(X1, X2))) which can be used in program construction. For example, quick- sort can be learned by providing not only examples for sorting Figure 1: Induction of reverse by Igor2 but also examples for partitioning lists and for appending lists. One drawback of search-based approaches is, however, that the background knowledge needs to be restricted to such functions of the example outputs with according parameter substitutions which are necessary to induce the target program because oth- and empty condition. If the resulting rule bodies contain un- erwise there is combinatorial explosion. Igor2 allows the use of bound variables, successor hypotheses are computed applying background knowledge but the introduction of such additional the following three methods: (1) Partitioning of the inputs by functions is restricted by matching against the given examples replacing one pattern by a set of disjoint more specific patterns due to the analytical strategy used. or by adding a predicate to the condition. (2) Replacing the Contemporary functional languages such as ML or Haskell body by a (recursive) call of a defined function, where finding allow to define functions using patterns for case distinction and the argument of the function call is treated as a new induc- rely on the definition of data types. These characteristics are re- tion problem. (3) Replacing the sub-terms in which unbound flected in constructor term rewriting systems (Baader & Nipkow, variables occur by a call to new sub-programs. In cases (2) 1998). Igor2 is defined in this framework and implemented in and (3) auxiliary functions are invented, abducing input/output Maude (Clavel et al., 2003). Input for Igor2 is examples for them. The algorithm is described in more detail • a set of non-recursive equations specifying the input/output in Kitzelmann (2009) the system can be downloaded from the examples of the target function, project website. • definitions of the data types for the target program (which can be very general, e.g. just sort, or more specific) by means of constructors, 3 Comparison of IP Systems • optional definitions of background knowledge, also in form Igor2 of non-recursive equations. To get closer insights in the strengths and weaknesses of The output of Igor2 is a program hypothesis in form of we characterised it in relation to other approaches to induc- a set of recursive term rewriting rules. The returned hypoth- tive programming with respect to: the algorithmic strategy, its esis has the following guaranteed characteristics (Kitzelmann, scope and its efficiency (Hofmann, Kitzelmann, & Schmid, 2008, 2009): All input/output examples are covered. The hypothesis 2009). We used conditional higher-order term rewriting as a is a recursive generalisation which is minimal with respect to the framework to characterise both logical and functional programs number of case distinctions, the number of rules and the syn- and characterised the currently available systems with respect tactical complexity of rule bodies. The learned target function to their language and search biases, the kind and number of is guaranteed to terminate. examples necessary for induction, the possibility to make use As Igor1, Igor2 can induce target functions together with of background knowledge and whether the systems are able to additional functions necessary to solve the problem. For exam- invent functions or predicates. Furthermore, we performed em- ple, to construct a function for reverting lists, Igor2 induces pirical evaluations of the different systems. A selection of results the additional functions last and init without any hint from the is given in Table 1. system user. Such kind of function invention is the equiva- As problems we have chosen such which are often presented lent problem to necessary predicate invention (Flener & Yilmaz, in inductive programming papers and additionally such which 1999) as discussed in ILP. The specification and the induced highlight specific strengths and weaknesses: multlast replaces solution of reverse is given in Figure 1. all elements with the last and shiftr makes a right-shift of all The algorithm of Igor2 can be outlined as follows: First ini- elements in a list. Therefore it is necessary to access the last tial rules are constructed as least general generalisation (Plotkin, element for further computations. Further functions are lasts 1969) of the example equations with resulting patterns as gener- which applies last on a list of lists, isort which is insertion-sort, alisation of the example inputs and rule bodies as generalisation and weave alternates elements from two lists into one. The

Page 2 Another kind of programming assistance is to support profes- Table 1: Systems’ run times in secs sional software developers working with tools such as the Ratio-

I II nal Software Architect. We are currently exploring possibilities to combine model-driven and example-driven programming. If Adate Ffoil MagH. Igor Igor the software engineer gives input/output examples as annota- lasts 365.62 0.7⊥ 19.43 0.051 5.695 tion to methods, these examples can be generalised to function- last 1.0 0.1 0.01 0.005 0.007 style methods. Obviously this strategy can only be applied to a member 2.11 0.1⊥ 1.07 — 0.152 sub-class of methods but this might be a useful way to relieve ⊥ odd/even — < 0.1 — — 0.019 programmers from routine coding tasks. A more natural appli- multlast 5.69 < 0.1 0.30 0.331 0.023 cation for example-driven programming is to support functional 83 41 × 0 01 0 105 isort . . — . programming (Koopman & Plasmeijer, 2006). We plan to focus reverse 30.24 — 0.08 0.324 0.103 ⊥ on this application in the future. weave 27.11 0.2 ⊙ 0.001 0.022 shiftr 20.14 < 0.1⊥ 157.32 0.041 0.127 From a broader perspective, analytical inductive program- ming can be viewed as a generic approach to extracting regu- — not tested × stack overflow ⊙ time out ⊥ wrong larities from experience. We applied Igor2 to several domains considered in cognitive modelling such as problem solving, rea- soning and natural language processing (Schmid, Hofmann, & functions in odd/even are mutually recursive and need more than Kitzelmann, 2009) and we could show that Igor2 can learn op- two rules; lasts, multlast, isort, and reverse suggest function timal rule sets for solving problems such as Tower of Hanoi or invention, but only reverse needs necessary function invention to blocks-world problems from small sets of positive examples. produce the desired program. lasts also splits up in more than two rules if no function invention is applied. To solve member pattern matching is required, because equality is not provided as 5 Further Work predicate. The function weave is especially interesting, because it demands either for iterating over more than one argument The empirical comparison showed that for the tested problems resulting in more than one base case, or swapping the arguments Igor2 performed very well with respect to synthesis time and at each recursive call. scope. Since Adate as evolutionary approach is in principle Because Ffoil usually performs better with more examples, able to synthesise all possible programs but is comparatively slow whereas MagicHaskeller and Igor II do better with less, to construct even small programs we currently investigate how each system got as much examples as necessary up to certain analytical preprocessing of examples with Igor2 can speed-up complexity, but then exhaustively, so no specific cherry-picking synthesis time for Adate with promising initial results (Crossley, was allowed. For synthesising isort all systems had a function Kitzelmann, Hofmann, & Schmid, 2009). to insert into a sorted list, and the predicate < as background A current restriction of Igor2 is that it cannot induce rules knowledge. For all systems except MagicHaskeller the def- with a call to another recursive function as outer function. This inition of the background knowledge was extensional. Igor II restriction is inherent to Igor2’s strategy of abducing new in- was allowed to use variables. MagicHaskeller had paramor- put/output examples for function invention. For outer functions phic functions to iterate over a data type. Note that we did the input/output examples are the originally given ones. There- not test a system with a problem which it per se cannot solve fore, we plan to extend Igor2 such that higher-order functions due to its language bias. This is indicated with ‘—’ instead of can be introduced in rule bodies. Since Maude does not pro- a runtime. A timeout after ten minutes is indicated with ⊙. vide higher-order concepts and for higher visibility of Igor2 in Tests have been conducted on a Intel Dual Core 2.33 GHz with the functional programming community we currently provide a 4GB memory. 2.2, Adate version 0.50 and MagicHaskeller Haskell implementation. 0.8.3-1. The input files for the presented and further problems Acknowledgement. can be obtained from the project web-page. The research project “Efficient Algorithms for Inductive Program Synthesis” is supported by DFG grant SCHM 1239/6-1 since October 2007. 4 Applications

The natural application for inductive programming is program- References Igor2 ming assistance. We applied to support end-user pro- Baader, F., & Nipkow, T. (1998). Term rewriting and all that. gramming in XSL (Schmid & Waltermann, 2004; Hofmann, Cambridge University Press. 2008). We provide AutoXSL, an Eclipse plug-in which can Biermann, A. W., Guiho, G., & Kodratoff, Y. (Eds.). (1984). Auto- be downloaded from the project web-page. It supports users matic program construction techniques. New York: Macmillan. to write XSL transformations for XML files who have no back- Clavel, M., Dur´an, F., Eker, S., Lincoln, P., Mart´ı-Oliet, N., Meseguer, J., & Talcott, C. (2003). The Maude 2.0 system. In ground in programming. Users edit tag contents of an XML R. Nieuwenhuis (Ed.), Rewriting Techniques and Applications file, the edit traces are rewritten into input/output examples (RTA 2003) (p. 76-87). Springer. and given to Igor2. The generalised rules are then transformed Crossley, N., Kitzelmann, E., Hofmann, M., & Schmid, U. (2009). into recursive XSL transformations and applied to the XML doc- Combining analytical and evolutionary inductive programming. ument. The bottleneck is to provide the correct set of examples In B. Goerzel, P. Hitzler, & M. Hutter (Eds.), Proceedings of for Igor2. Therefore, the user has to select the relevant trans- the Second Conference on Artificial General Intelligence (AGI- formations from a list of I/O pairs calculated from the traces.

Page 3 09, Arlington, Virginia, March 6-9 2009) (pp. 19–24). Ams- 287–312. terdam: Atlantis Press. Schmid, U. (2003). Inductive synthesis of functional programs – Flener, P. (1996). Inductive logic program synthesis with Dialogs. Learning domain-specific control rules and abstract schemes. In S. Muggleton (Ed.), Proceedings of the 6th International Heidelberg: Springer. Workshop on Inductive Logic Programming (p. 28-51). Stock- Schmid, U., Hofmann, M., & Kitzelmann, E. (2009). Analytical in- holm University, Royal Institute of Technology. ductive programming as a cognitive rule acquisition devise. In Flener, P., & Partridge, D. (2001). Inductive programming. Auto- B. Goerzel, P. Hitzler, & M. Hutter (Eds.), Proceedings of the mated Software Engineering, 8(2), 131-137. Second Conference on Artificial General Intelligence (AGI-09, Flener, P., & Schmid, U. (to appear). Inductive programming. In Arlington, Virginia, March 6-9 2009) (pp. 162–167). Amster- C. Sammut & G. Webb (Eds.), Encyclopedia of machine learn- dam: Atlantis Press. ing. Springer. Schmid, U., & Waltermann, J. (2004). Automatic synthesis of XSL- Flener, P., & Yilmaz, S. (1999). Inductive synthesis of recursive transformations from example documents. In Artificial Intel- logic programs: Achievements and prospects. Journal of Logic ligence and Applications Proceedings (AIA 2004, Innsbruck, Programming, 41(2–3), 141–195. Austria, February 16-18) (p. 252-257). Acta Press, Series on Hofmann, M. (2008). Automated construction of xsl-templates: An Artificial Intelligence and Soft Computing. inductive programming approach. Saarbr¨ucken: VDM. Schmid, U., & Wysotzki, F. (1998). Induction of recursive pro- Hofmann, M., Kitzelmann, E., & Schmid, U. (2008). Analysis and gram schemes. In Proc. 10th European Conference on Machine evaluation of inductive programming systems in a higher-order Learning (ECML-98) (Vol. 1398, p. 214-225). Springer. framework. In A. Dengel, K. Berns, T. M. Breuel, F. Bomarius, Summers, P. D. (1977). A methodology for LISP program construc- & T. R. Roth-Berghofer (Eds.), KI 2008: Advances in Artificial tion from examples. Journal ACM, 24(1), 162-175. Intelligence (p. 78-86). Berlin: Springer. Hofmann, M., Kitzelmann, E., & Schmid, U. (2009). A unifying framework for analysis and evaluation of inductive program- Contact ming systems. In B. Goerzel, P. Hitzler, & M. Hutter (Eds.), Proceedings of the Second Conference on Artificial General In- Prof. Dr. Ute Schmid, telligence (AGI-09, Arlington, Virginia, March 6-9 2009) (pp. Dipl.-Inf. Emanuel Kitzelmann, 55–60). Amsterdam: Atlantis Press. Dipl.-Wirtsch.Inf (E.M.B.Sc.) Martin Hofmann Katayama, S. (2005). Systematic search for lambda expressions. In Fakult¨atWirtschaftsinformatik und Angewandte Informatik Trends in functional programming (p. 111-126). Otto-Friedrich-Universit¨atBamberg Kitzelmann, E. (2008). Data-driven induction of functional programs. 96045 Bamberg In M. Ghallab, C. Spyropoulos, N. Fakotakis, & N. N. Avouris (Eds.), ECAI 2008, 18th European Conference on Artificial Tel.: +49 (0)951-863-2861 Intelligence, Proceedings (p. 781-782). IOS. Fax: +49 (0)951-863-2862 Kitzelmann, E. (2009). Analytical inductive functional programming. EMail: {schmid, kitzelmann, hofmann}@uni-bamberg.de In M. Hanus (Ed.), Proceedings of the 18th International Sym- Website: http://www.cogsys.wiai.uni-bamberg.de/effalip/ posium on Logic-Based Program Synthesis and Transforma- tion (LOPSTR 2008, Valencia, Spain) (Vol. 5438, p. 87-102). Springer. Bild Ute Schmid holds a diploma in psychology Kitzelmann, E., & Hofmann, M. (2008). IGOR2: an inductive func- and a diploma in computer science, both tional programming prototype. In M. Ghallab, C. Spyropou- from Technical University Berlin (TUB). She los, N. Fakotakis, & N. N. Avouris (Eds.), Proceedings of the received her doctoral degree in computer sci- System Demonstrations of the 18th European Conference on ence from TUB in 1994 and her habilitation Artificial Intelligence (p. 29-31). Patras, Greece. Kitzelmann, E., & Schmid, U. (2006). Inductive synthesis of in computer science in 2002. She worked as functional programs: An explanation based generalization ap- assistant and lecturer at TUB and Univer- proach. Journal of Machine Learning Research, 7(Feb), 429– sity of Osnabr¨uck. Since 2004 she is profes- 454. sor for Applied Computer Science/Cognitive Kitzelmann, E., & Schmid, U. (2007). Inducing constructor systems Systems at University of Bamberg. from example-terms by detecting syntactical regularities. Elec- tronic Notes in Theoretical Computer Science, 174(1), 49-63. Koopman, P., & Plasmeijer, R. (2006). Systematic synthesis of func- Bild Emanuel Kitzelmann studied computer sci- tions. In H. Nilsson (Ed.), Proceedings Seventh Symposium ence in Passau and at Technical University on Trends in Functional Programming, TFP 2006 (p. 35-54). Berlin where he received his diploma in 2004. Intellect Books UK. He is working as teaching associate in the Mitchell, T. M. (1997). Machine learning. New York: McGraw-Hill. Cognitive Systems Group at University of Muggleton, S., & De Raedt, L. (1994). Inductive logic programming: Bamberg and currently is finishing his doc- Theory and methods. Journal of Logic Programming, Special Issue on 10 Years of Logic Programming, 19-20, 629-679. toral degree. Muggleton, S., & Feng, C. (1990). Efficient induction of logic pro- grams. In Proc. of the Workshop on Algorithmic Learning Theory by the Japanese Society for AI (p. 368-381). Tokyo. Olsson, R. (1995). Inductive functional programming using incre- mental program transformation. Artificial Intelligence, 74(1), 55-83. Plotkin, G. D. (1969). A note on inductive generalization. In Machine intelligence (Vol. 5, pp. 153–163). Edinburgh University Press. Quinlan, J., & Cameron-Jones, R. (1995). Induction of logic pro- grams: FOIL and related systems. New Generation Comput- ing, Special Issue on Inductive Logic Programming, 13(3-4),

Page 4 Bild Martin Hofmann studied business informa- tion systems and business management at University of Bamberg and University of Wales, Swansea. He graduated 2007 and currently is research associate in the project “Efficient algorithms of inductive program- ming”.

Page 5