Proof Artifact Co-training for Theorem Proving with Language Models
Jesse Michael Han 1 Jason Rute 2 Yuhuai Wu 3 Edward W. Ayers 4 Stanislas Polu 5
Abstract curriculum of theorems, the agent will inevitably saturate Labeled data for imitation learning of theorem and suffer from data starvation. proving in large libraries of formalized mathe- Data scarcity is a particularly thorny obstruction for ap- matics is scarce as such libraries require years plying large language models (LLMs) to neural theorem of concentrated effort by human specialists to be proving. LLMs have achieved spectacular success in data- built. This is particularly challenging when apply- rich regimes such as plain text (Brown et al., 2020), im- ing large Transformer language models to tactic ages (Dosovitskiy et al., 2020), and joint text-image model- prediction, because the scaling of performance ing (Radford et al.), and the performance of decoder-only with respect to model size is quickly disrupted Transformers has been empirically shown to obey scaling in the data-scarce, easily-overfitted regime. We power laws in model and data size (Henighan et al., 2020). propose PACT (Proof Artifact Co-Training), a However, existing datasets of human proof steps for neural general methodology for extracting abundant self- theorem proving are extremely small and exist at scales at supervised data from kernel-level proof terms for which overfitting occurs extremely rapidly, disrupting the co-training alongside the usual tactic prediction scaling of performance with respect to model size (Kaplan objective. We apply this methodology to Lean, et al., 2020). an interactive proof assistant which hosts some of the most sophisticated formalized mathematics to We make two contributions towards addressing the problem date. We instrument Lean with a neural theorem of data scarcity in the context of formal mathematics. First, prover driven by a Transformer language model we introduce PACT (Proof Artifact Co-Training), a general and show that PACT improves theorem proving methodology for extracting self-supervised auxiliary tasks success rate on a held-out suite of test theorems for co-training a language model alongside a tactic predic- from 32% to 48%. tion objective for interactive theorem proving. Second, we present LEANSTEP, a collection of datasets and a machine learning environment for the Lean 3 theorem prover with 1. Introduction support for PACT, supervised learning of tactic prediction, theorem proving evaluation, and reinforcement learning. Deep learning-driven automated theorem proving in large libraries of formalized mathematics (henceforth “neural We train large language models on these data and demon- theorem proving”) has been the focus of increased atten- strate that PACT significantly improves theorem proving tion in recent years. Labeled data for imitation learning success rate on a held-out suite of test theorems, from 32% of theorem proving is scarce—formalization is notoriously to 48%. We then embark on a careful study of the effects labor-intensive, with an estimated cost of 2.5 man-years of pre-training vs. co-training and show that PACT com- arXiv:2102.06203v1 [cs.AI] 11 Feb 2021 per megabyte of formalized mathematics (Wiedijk, 2000), bined with WebMath pre-training (Polu & Sutskever, 2020) and complex projects require years of labor from human achieves the lowest validation loss and theorem proving specialists. Within a fixed corpus of (possibly unproven) success rate. Finally, on an out-of-distribution collection of theorem statements, it is possible to augment a seed dataset thousands of theorems (some involving novel definitions) of human proofs with new successful trajectories using rein- added to Lean’s mathematical library after we extracted our forcement learning or expert iteration. However, this is quite train/test data, we achieve a theorem proving success rate of computationally intensive, and without a way to expand the 37%, suggesting strong generalization and usefulness at the frontier of formalized mathematics. 1University of Pittsburgh, Pittsburgh, PA, USA 2CIBO Tech- nologies, Cambridge, MA, USA 3University of Toronto, ON, Canada 4University of Cambridge, Cambridge, UK 5OpenAI, San 2. Background Francisco, CA, USA. Correspondence to: Jesse Michael Han
Metamath is an another interactive theorem prover which Self-supervised learning methods Recently, self- does not use tactics, but instead relies on a low-level prov- supervised methods have revolutionized machine learning ing framework. Neural provers for Metamath such as models across various domains, such as natural language Holophrasm (Whalen, 2016), MetaGen (Wang & Deng, understanding (Logeswaran & Lee, 2018; Radford et al., 2020), and GPT-f (Polu & Sutskever, 2020) all use sequence 2018; Devlin et al., 2019; Song et al., 2019; Dong et al., or language models to generate tactics. Specifically, our 2019; Raffel et al., 2020; Conneau & Lample, 2019), work builds on Metamath GPT-f (Polu & Sutskever, 2020) computer vision (Vincent et al., 2008; Doersch et al., (MM GPT-f), which uses a Transformer architecture to gen- 2015; Noroozi & Favaro, 2016; Pathak et al., 2016; erate proof steps. Unlike MM GPT-f, which trained pri- Gidaris et al., 2018; van den Oord et al., 2018), and marily on the Metamath proof step objective (i.e. guessing reinforcement learning (Jaderberg et al., 2017; Jang the next lemma to be applied to a goal, and subsumed by et al., 2018; Laskin et al., 2020). These methods rely NEXTLEMMA our task, c.f. Section 3.2), we co-train on a on automatically generating labels for a large quantity diverse suite of self-supervised tasks extracted from Lean of cheaply mined data, which is then used to create proof terms and demonstrate significant improvements in auxiliary tasks. Training on the auxiliary tasks has been theorem proving performance. shown to greatly improve performance sample efficiency on downstream tasks, and even improve the adversarial Reasoning with Transformers Besides theorem proving, robustness of models (Hendrycks et al., 2019; Carmon a number of recent papers have shown that language mod- et al., 2019; Chen et al., 2020). For text, the most popular els, especially Transformers, are capable of something like approach is based on language modeling with next-word mathematical and logical reasoning in integration (Lample prediction tasks (Radford et al., 2018; Devlin et al., 2019). & Charton, 2020), differential equations (Charton et al., The advent of the Transformer architecture (Vaswani et al., 2017) and BERT style pretraining (Devlin et al., 2019) such goals are the user-facing representation of the tactic achieved a huge improvement in many natural language state. The user enters instructions for modifying the tactic understanding tasks. Since then, an explosion of research state called tactic commands using Lean’s tactic DSL. If activity around better pretraining tasks has further improved the tactic command is successful, the user will be prompted the quality of language models. with one or more new goals. The user repeats this process until all goals are closed, thereby solving the theorem. Our proposed method also makes use of automatically gen- erated labels on a large quantity of cheaply-mined data, i.e. Our human tactic proof step dataset consists of source-target extant proof artifacts. The notable differences from existing pairs of strings, one for each tactic command in the Lean methods are (1) our auxiliary tasks are specifically designed core library and in mathlib. The source string is the for theorem proving, and (2) most of the existing text-based pretty-printed tactic state. The target string is the tactic com- self-supervised methods use auxiliary tasks for pretraining, mand as entered by a human in the source code to modify whereas we investigate whether co-training can bring further the tactic state. We train language models autoregressively benefits and make better use of auxiliary tasks. to complete the source with the target Section 4.1. We refer to the task of predicting the next human tactic proof step given a tactic state as the proofstep objective. Machine learning with proof artifacts The idea of min- ing low-level proof artifacts was previously explored in the Lean’s parser is special in that syntax can change signif- context of automated lemma extraction (Kaliszyk & Urban, icantly mid-code. While this makes it possible to utilize 2015c; Kaliszyk et al., 2015). It has also been previous ob- custom mathematical syntax and domain specific tactic lan- served that training on fully elaborated Coq terms (Nie et al., guages, it makes it near impossible to parse Lean code out- 2020) helps with a downstream theorem naming task. How- side of using Lean itself. Our solution was a mixed approach ever, similar to previous work on skip-tree training, their where we used Lean’s extensive meta-programming frame- dataset focuses solely on theorem statements, i.e. types, work to hook into the Lean parser and tactic framework, does not cover the far more complex proof terms, and does extracting the tactic goals and various parser position infor- not evaluate the effect of such training on theorem proving mation. Then we used a simple homemade Lean parser to evaluations. combine this information and extract the tactic commands. While there exist environments and datasets for other formal 3.2. Proof artifact co-training mathematics libraries (Kaliszyk et al., 2017; Li et al., 2021; Huang et al., 2018; Kaliszyk & Urban, 2015b), LEANSTEP In this section, we describe the PACT task suite and how is the first and only tactic proof dataset for the Lean theorem training examples for these tasks are extracted. prover. This makes available a large set of formal math- ematical data to researchers covering a diverse and deep For every proof term τ, we record the type Γ of τ, its name nm, and a list ps of all premises (i.e. named references to spectrum of pure mathematics. Moreover, LEANSTEP is unique in that it contains both high-level human-written tac- other lemmas in the library) which are used in τ. We then bs tics as well as kernel-level proof terms, which enables the recurse through τ, tracking a list of bound variables extraction of self-supervised tasks for PACT (Section 3.2). which we update whenever navigating into the body of a λ-expression. At every sub-term τ 0 ⊆ τ we record τ 0, its type Γ0, the current state of bs, and the following data: 3. The LEANSTEP datasets and machine learning environment 1. A tactic state, where the goal is set to be Γ0 and the list of hypotheses in the local context is set to be the list We describe our datasets and instrumentation of the Lean bs, i.e. those bound variables in scope at τ 0. theorem prover, including data extraction procedures and a generic best-first search algorithm for automated theorem 2.A partial proof term, i.e. τ with τ 0 masked out. proving with a language model. While our data pipeline can 3. A premise selection bitmask, i.e. Boolean labels for be instantiated at any Lean project, enabling the generation every p in ps indicating whether p is used in τ 0. of bespoke, domain-specific datasets, we henceforth focus on mathlib, Lean’s mathematical components library. 4. A local context bitmask, i.e. similar Boolean labels for every b in bs indicating whether b is used in τ 0. 3.1. Human tactic proof steps 5. An optional next lemma: if the first step of τ 0 is to As seen in Figure 3, Lean tactic proofs consist of comma- apply a premise p in ps, we record p. separated tactic commands. At the start of the proof the user is prompted with a goal to prove. That goal is proceeded by Whenever we record a term, we record both pretty-printed a list of hypotheses and declared local variables. In Lean, and far more explicit fully elaborated versions of it. The l emma i f f _of _t r ue { P : Pr op} GOAL P : Pr op ? ( P ? t r ue) ? P PROOFSTEP i nt r o H : ( P ? t r ue) ? P : = begi n GOAL P : Pr op, H : P ? t r ue ? P PROOFSTEP cases H wi t h H? H? i nt r o H, cases H wi t h H? H?, appl y H?, GOAL P : Pr op, H? : P ? t r ue, H? : t r ue ? P ? P PROOFSTEP appl y H? t r i vi al end GOAL P : Pr op, H? : P ? t r ue, H? : t r ue ? P ? t r ue PROOFSTEP t r i vi al
Masked proof term Tactic state Term that fulfils goal . . .
? H, i f f . dcases_on H _ ? ( P ? t r ue) ? P ( ? H? H?, H? t r i vi al ) RESULT ? { P : Pr op} ( H : P ? t r ue) , PREDI CT H ( ? ( H? : P ? t r ue) ( H? : t r ue ? P) , H? t r i vi al ) TYPE ( P ? t r ue) ? ( ( P ? t r ue) ? ( t r ue ? P) H : P ? t r ue i f f . dcases_on H ? H, _ ? P) ? P ? P ( ? H? H?, H? t r i vi al ) H? : P ? t r ue ? H, i f f . dcases_on H H? : t r ue ? P H? t r i vi al ( ? H? H?, _) TYPE ? { P : Pr op} , ( P ? t r ue) ? P NAME i f f _of _t l ean ? P H? : P ? t r ue ? H, i f f . dcases_on H H? : t r ue ? P t r i vi al ( ? H? H?, H? _) GOAL P : Pr op, H : P ? t r ue ? ( P ? t r ue) ? ( ( P ? t r ue) ? ( t r ue ? P) ? t r ue ? P) ? P NEXT_LEMMA i f f . dcases_on H : P ? t r ue ? ( P ? t r ue) ? ? H, _ H ( ? H? H?, ( ( P ? t r ue) ? i f f . dcases_on H? t r i vi al ) RESULT ? { P : Pr op} ( H : P ? t r ue) , PREDI CT H ( ? ( H? : P ? t r ue) ( H? : ( t r ue ? P) ? P) t r ue ? P) , H? t r i vi al ) TYPE ( P ? t r ue) ? ( ( P ? t r ue) ? ( t r ue ? P) ? P ? P) ? P
Figure 3. An illustration of our data extraction procedures. The entry point is the top-level theorem iff_of_true. (Top) We gather human proof step data by stepping through the supplied tactic proof script, recording the tactic state and subsequent tactic application. We train a Transformer language model on the sequence GOAL ... PROOFSTEP .... When using our model for proof search, we only prompt it using GOAL ... PROOFSTEP to generate tactic applications. (Bottom) We gather data for PACT by recursing through all subterms of the proof term produced by the tactic script. We generate training examples in a self-supervised fashion, creating many auxiliary tasks which we disambiguate from the primary PROOFSTEP task using specially chosen prompt keywords. fully elaborated terms explicitly display enormous amounts 9. Theorem naming. Given the type Γ of the top-level of type information which are usually silently inferred by proof term τ, predict the name nm. Lean. From these data, we assemble the following language modeling tasks: We formulate the premise classification task in terms of predicting a bit given the tactic state and premise instead 1. Next lemma prediction. Given the tactic state, predict of predicting the sublist of all positive premises according the next lemma to be applied. to the premise selection bitmask because there are often an order of magnitude more premises than hypotheses in the 2. Proof term prediction. Given the tactic state, predict local context. the entire proof term τ 0. We remark that our next lemma prediction task is precisely 3. Skip-proof. Given the partial proof term, predict the the low-level PROOFSTEP objective studied in (Polu & masked-out subterm τ 0. Sutskever, 2020), and our skip-proof task superficially re- sembles, but is much more difficult than the skip-tree task 4. Type prediction. Given the partial proof term, predict studied in (Rabe et al., 2020), as proof terms tend to be 0 0 the type Γ of the masked-out subterm τ . far more complicated than the syntax trees of the theorem statement. Our classification tasks are closely related to 5. Tactic state elaboration. Given the tactic state, pre- the proof step classification examples studied by (Kaliszyk dict the fully elaborated tactic state. et al., 2017). We also note that by mathlib convention, 6. Proof term elaboration. Given τ, predict the fully theorem names are compact snake-cased English descrip- elaborated version of τ. tions of the type signature, so the theorem naming task is akin to a translation/summarization task. 7. Premise classification. Given the tactic state and a premise p ∈ ps, predict either
Figure 4. Auto-regressive objectives used for each task described in Section 3. Placeholders represented with brackets (such as
Model Tokens total Early-stop mix1 mix2 tactic Pass-rate Baselines refl 1.1% tidy-bfs 9.9% WebMath > tactic 32B 1B 1.02 32.2% Pre-training WebMath > mix1 32B 11B 0.08 WebMath > mix2 32B 16B 0.08 WebMath > mix1 + mix2 32B 22B 0.11 0.08 WebMath > mix1 > tactic 32B 1B 1.00 39.8% WebMath > mix1 + mix2 > tactic 32B 1B 0.97 44.0% Co-training (PACT) WebMath > mix1 + tactic 32B 18B 0.08 0.94 40.0% WebMath > mix1 + mix2 + tactic 96B 71B 0.09 0.09 0.91 48.4% Pre-training and co-training WebMath > mix2 > mix1 + tactic 32B 18B 0.08 0.93 46.9%
Figure 5. Comparison of pre-training and co-training on mix-1 and mix-2. > denotes a pre-training step and + denotes a co-training. As an example, WebMath > mix2 > mix1 + tactic signifies a model successively pre-trained on WebMath then mix2 and finally co-trained as a fine-tuning step on mix1 and tactic. Columns mix1, mix2, tactic report the min validation loss achieved on these respective datasets. Model Tokens total Early-stop mix1 mix2 tactic Baselines tactic 32B 1B 1.59 Pre-training mix1 32B 20B 0.12 mix2 32B 25B 0.10 mix1 + mix2 32B 27B 0.13 0.10 mix1 > tactic 32B 1B 1.26 mix1 + mix2 > tactic 32B 1B 1.16 Co-training mix1 + tactic 32B 27B 0.11 1.12 mix1 + mix2 + tactic 96B 71B 0.10 0.11 1.07 Pre-training and co-training mix2 > mix1 + tactic 32B 26B 0.11 1.09
Figure 6. Validation losses achieved in the pre-training and co-training setups without WebMath pre-training. See Figure 5 for a description of the columns and the models nomenclature used.
Model Tokens total Early-stop mix1 mix2 tactic Pass-rate 121M 96B 82B 0.13 0.10 1.23 35.1% 163M 96B 80B 0.12 0.09 1.11 39.8% 837M 96B 71B 0.09 0.09 0.91 48.4%
Figure 7. Validation losses and pass-rates achieved for various model sizes using PACT. See Figure 5 for a description of the columns. The setup used is WebMath > mix1 + mix2 + tactic. (simply attempting a proof by the refl tactic) closes 328 human experts to provide feedback and additional training proofs (11.6%), demonstrating an important skew towards signal in a virtuous cycle. trivial boilerplate lemmas generally defined to provide alternate interfaces to new definitions. The tidy-bfs Future directions There are many elaborations on the baseline closed 611 proofs (21.8%), and our best model training data, training methodology, and tree search wrap- wm-tt-m1-m2 closed 1043 proofs (37.1%), proving 94% ping lean-gptf which can be reasonably expected to of the refl lemmas. improve its performance at theorem proving. Our dataset We attribute the weaker performance to heavy distribution can be synthetically augmented using similar methods as shift: by the nature of the dataset, the future-mathlib (Polu & Sutskever, 2020). Merely making the decoded theorems frequently involve new definitions and concepts rewrites robust by only using the largest prefix of successful which the model was never exposed to during training. Nev- rewrites significantly boosts the success rate of suggested ertheless, the success rate remains high enough to suggest rewrites. In a similar vein, predicted lemmas generated strong generalization and usefulness at the frontier of for- as arguments to unsuccessful tactic applications could be malized mathematics. cached and re-used as hints for an intermittently-queried hammer. The increased success rate of chained tactic pre- dictions mentioned above shows the feasibility of having 5. Discussion language models perform multiple reasoning steps in a sin- Chained tactic predictions In Lean, multiple tactic com- gle query, potentially improving the efficiency of the proof mands can be chained together using semicolons. Our data search. From the experiments described in Section 4, it is pipeline treats these tactic chains as a single sequence in clear that the composition of the dataset used for co-training our training data, and they are occasionally predicted by significantly affects performance on theorem proving. Al- the model. Such chained tactic applications are difficult though we uniformly sampled across all co-training tasks, it for human formalizers to synthesize on their own, as they would be interesting to optimize a dynamic mixture sched- require reasoning about the semantics of multiple tactics ule, perhaps annealing towards a desired task. in sequence and their effects on the tactic state, and the examples present in the training data are usually optimized Conclusion There is a sense in which PACT is merely by hand from longer, less succinct proofs. We observed that an application of the well known principle that compute in PACT significantly boosts the capability of our models to the form of search should be exchanged for training signal successfully predict longer chained tactic applications. This whenever possible. In Lean, typeclass inference relies on a occurs in spite of the fact that this tactic chaining idiom is backtracking Prolog-style search; the elaborator performs specific to the tactic proofstep dataset and does not appear in search to disambiguate overloaded notation and infer types; the PACT training data whatsoever. We supply more detail Lean tactics have complex semantics precisely because they in the appendix. can perform search to find subproofs automatically. The work done by these subroutines is preserved in the proof artifacts, and PACT can be viewed as a way of extracting Impact on Lean community Lean’s mathlib is one of this information offline for more training signal. the most active open-source software projects in the world, achieving explosive growth in recent years (mathlib, 2020). We have presented PACT as a way of addressing the data Our work has been welcomed by members of this commu- scarcity issue for learning theorem proving from human tac- nity, with Lean power users describing some of the new tic scripts in proof assistant libraries. Another well-studied proofs found by GPT-f as “nontrivial” and “clever”. More solution for this is expert iteration and reinforcement learn- than one-third of the proofs found by our models are shorter ing. In the setting of HOL Light, and under the assumption and produce smaller proof terms (sometimes by several of a hardcoded finite action space of tactics, DeepHOLZero orders of magnitude) than the ground truth. Manually in- in conjunction with supervised seed data was able to achieve specting a portion of these shorter proofs has led to 36 GPT-f up to 70% proof success rate on the HOList theorem proving co-authored commits to mathlib, some of which reduce task. Similarly, in a set-up much closer to ours, MM GPT-f proof term sizes and theorem compilation times by an order demonstrated the feasibility of expert iteration when using of magnitude. We supply more detail in the appendix. generative language models for theorem proving. Within a fixed corpus of theorems (and hence proof terms), Lean GPT-f interactive frontend We have released a however, both PACT and RL are fundamentally constrained simplified version of the proof search described in Sec- by a lack of exploration—as the performance of the theo- tion 3.3 as a tactic called gptf to the Lean community in a rem proving agent improves, it will eventually saturate and public beta, opening the way for our models to directly ac- become starved for data, and its curriculum will need to celerate the development of formalized mathematics and for be expanded. Although self-supervised methods such as PACT represent a way to significantly improve the data- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, efficiency of reinforcement learning loops over existing the- J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., orem prover libraries, the development of continuously self- Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., improving and infinitely scalable neural theorem provers Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, remains contingent on sufficiently powerful exploration and J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., automated curriculum generation; we consider these chal- Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., lenges to be of paramount importance. Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Acknowledgements Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: We thank the members of the Lean community, in particu- Annual Conference on Neural Information Processing lar Kevin Buzzard, Simon Hudon, Johan Commelin, Mario Systems 2020, NeurIPS 2020, December 6-12, 2020, Carneiro, Bhavik Mehta, and Gabriel Ebner for their valu- virtual, 2020. URL https://proceedings. able feedback on our work. We are indebted to Markus Rabe neurips.cc/paper/2020/hash/ and Christian Szegedy for many hours of helpful discussion. 1457c0d6bfcb4967418bfb8ac142f64a-Abstract. We also thank Daniel Selsam, Tom Hales, and Josef Urban html. for feedback on earlier drafts of this paper. Buzzard, K., Hughes, C., Lau, K., Livingston, A., Mir, R. F., and Morrison, S. Schemes in lean. arXiv preprint References arXiv:2101.02602, 2019. Alama, J., Heskes, T., Kuhlwein,¨ D., Tsivtsivadze, E., and Buzzard, K., Commelin, J., and Massot, P. Formalis- Urban, J. Premise selection for mathematics by corpus ing perfectoid spaces. In Blanchette, J. and Hritcu, analysis and kernel methods. J. Autom. Reason., 52(2): C. (eds.), Proceedings of the 9th ACM SIGPLAN 191–213, 2014. doi: 10.1007/s10817-013-9286-5. International Conference on Certified Programs and Bansal, K., Loos, S. M., Rabe, M. N., and Szegedy, C. Proofs, CPP 2020, New Orleans, LA, USA, January Learning to reason in large theories without imitation. 20-21, 2020, pp. 299–312. ACM, 2020. doi: 10. CoRR, abs/1905.10501, 2019a. URL http://arxiv. 1145/3372885.3373830. URL https://doi.org/ org/abs/1905.10501. 10.1145/3372885.3373830. Bansal, K., Loos, S. M., Rabe, M. N., Szegedy, C., Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, J. C., and Wilcox, S. Holist: An environment for ma- and Liang, P. Unlabeled data improves adversarial chine learning of higher order logic theorem proving. robustness. In Wallach, H. M., Larochelle, H., Beygelz- In Chaudhuri, K. and Salakhutdinov, R. (eds.), Pro- imer, A., d’Alche-Buc,´ F., Fox, E. B., and Garnett, ceedings of the 36th International Conference on Ma- R. (eds.), Advances in Neural Information Processing chine Learning, ICML 2019, 9-15 June 2019, Long Systems 32: Annual Conference on Neural Information Beach, California, USA, volume 97 of Proceedings Processing Systems 2019, NeurIPS 2019, December of Machine Learning Research, pp. 454–463. PMLR, 8-14, 2019, Vancouver, BC, Canada, pp. 11190– 2019b. URL http://proceedings.mlr.press/ 11201, 2019. URL https://proceedings. v97/bansal19a.html. neurips.cc/paper/2019/hash/ 32e0bd1497aa43e02a42f47d9d6515ad-Abstract. Blaauwbroek, L., Urban, J., and Geuvers, H. The tacti- html. cian - A seamless, interactive tactic learner and prover for coq. In Benzmuller,¨ C. and Miller, B. R. (eds.), Intel- Charton, F., Hayat, A., and Lample, G. Deep differential ligent Computer Mathematics - 13th International Con- system stability - learning advanced computations from ference, CICM 2020, Bertinoro, Italy, July 26-31, 2020, examples. CoRR, abs/2006.06462, 2020. URL https: Proceedings, volume 12236 of Lecture Notes in Com- //arxiv.org/abs/2006.06462. puter Science, pp. 271–277. Springer, 2020. doi: 10. 1007/978-3-030-53518-6\ 17. URL https://doi. Chen, T., Liu, S., Chang, S., Cheng, Y., Amini, L., and org/10.1007/978-3-030-53518-6_17. Wang, Z. Adversarial robustness: From self-supervised pre-training to fine-tuning. In 2020 IEEE/CVF Confer- Blanchette, J. C., Kaliszyk, C., Paulson, L. C., and Ur- ence on Computer Vision and Pattern Recognition, CVPR ban, J. Hammering towards QED. J. Formaliz. Rea- 2020, Seattle, WA, USA, June 13-19, 2020, pp. 696– son., 9(1):101–148, 2016. doi: 10.6092/issn.1972-5787/ 705. IEEE, 2020. doi: 10.1109/CVPR42600.2020.00078. 4593. URL https://doi.org/10.6092/issn. URL https://doi.org/10.1109/CVPR42600. 1972-5787/4593. 2020.00078. Conneau, A. and Lample, G. Cross-lingual language December 8-14, 2019, Vancouver, BC, Canada, pp. model pretraining. In Wallach, H. M., Larochelle, H., 13042–13054, 2019. URL https://proceedings. Beygelzimer, A., d’Alche-Buc,´ F., Fox, E. B., and neurips.cc/paper/2019/hash/ Garnett, R. (eds.), Advances in Neural Information c20bb2d9a50d5ac1f713f8b34d9aac5a-Abstract. Processing Systems 32: Annual Conference on Neural html. Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 7057– Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, 7067, 2019. URL https://proceedings. D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, neurips.cc/paper/2019/hash/ M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. c04c19c2c2474dbf5f7ac4372c5b9af1-Abstract.An image is worth 16x16 words: Transformers for image html. recognition at scale. CoRR, abs/2010.11929, 2020. URL https://arxiv.org/abs/2010.11929. Cummins, C., Fisches, Z. V., Ben-Nun, T., Hoefler, T., and Leather, H. Programl: Graph-based deep learn- Ebner, G., Ullrich, S., Roesch, J., Avigad, J., and de Moura, ing for program optimization and analysis. CoRR, L. A metaprogramming framework for formal veri- abs/2003.10536, 2020. URL https://arxiv.org/ fication. Proc. ACM Program. Lang., 1(ICFP):34:1– abs/2003.10536. 34:29, 2017. doi: 10.1145/3110278. URL https: //doi.org/10.1145/3110278. de Moura, L. M., Kong, S., Avigad, J., van Doorn, F., and von Raumer, J. The Lean Theorem Prover (system de- Evans, R., Saxton, D., Amos, D., Kohli, P., and Grefenstette, scription). In Felty, A. P. and Middeldorp, A. (eds.), E. Can neural networks understand logical entailment? Automated Deduction - CADE-25 - 25th International In 6th International Conference on Learning Represen- Conference on Automated Deduction, Berlin, Germany, tations, ICLR 2018, Vancouver, BC, Canada, April 30 - August 1-7, 2015, Proceedings, volume 9195 of Lecture May 3, 2018, Conference Track Proceedings. OpenRe- Notes in Computer Science, pp. 378–388. Springer, 2015. view.net, 2018. URL https://openreview.net/ doi: 10.1007/978-3-319-21401-6\ 26. URL https:// forum?id=SkZxCk-0Z. doi.org/10.1007/978-3-319-21401-6_26. Finkbeiner, B., Hahn, C., Rabe, M. N., and Schmitt, F. Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: Teaching temporal logics to neural networks. CoRR, pre-training of deep bidirectional transformers for lan- abs/2003.04218, 2020. URL https://arxiv.org/ guage understanding. In Burstein, J., Doran, C., and abs/2003.04218. Solorio, T. (eds.), Proceedings of the 2019 Conference of Ganesalingam, M. and Gowers, W. T. A fully auto- the North American Chapter of the Association for Com- matic theorem prover with human-style output. J. Au- putational Linguistics: Human Language Technologies, tom. Reason., 58(2):253–291, 2017. doi: 10.1007/ NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, s10817-016-9377-1. URL https://doi.org/10. 2019, Volume 1 (Long and Short Papers), pp. 4171–4186. 1007/s10817-016-9377-1. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423. URL https://doi.org/ Gauthier, T., Kaliszyk, C., Urban, J., Kumar, R., and 10.18653/v1/n19-1423. Norrish, M. Learning to prove with tactics. CoRR, abs/1804.00596, 2018. URL http://arxiv.org/ Doersch, C., Gupta, A., and Efros, A. A. Unsupervised abs/1804.00596. visual representation learning by context prediction. In 2015 IEEE International Conference on Computer Vi- Gidaris, S., Singh, P., and Komodakis, N. Unsupervised rep- sion, ICCV 2015, Santiago, Chile, December 7-13, 2015, resentation learning by predicting image rotations. In 6th pp. 1422–1430. IEEE Computer Society, 2015. doi: International Conference on Learning Representations, 10.1109/ICCV.2015.167. URL https://doi.org/ ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 10.1109/ICCV.2015.167. 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum? Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, id=S1v4N2l0-. Y., Gao, J., Zhou, M., and Hon, H. Unified language model pre-training for natural language understanding Han, J. M. and van Doorn, F. A formal proof of the inde- and generation. In Wallach, H. M., Larochelle, H., pendence of the continuum hypothesis. In Blanchette, J. Beygelzimer, A., d’Alche-Buc,´ F., Fox, E. B., and and Hritcu, C. (eds.), Proceedings of the 9th ACM SIG- Garnett, R. (eds.), Advances in Neural Information PLAN International Conference on Certified Programs Processing Systems 32: Annual Conference on Neural and Proofs, CPP 2020, New Orleans, LA, USA, Jan- Information Processing Systems 2019, NeurIPS 2019, uary 20-21, 2020, pp. 353–366. ACM, 2020. doi: 10. 1145/3372885.3373826. URL https://doi.org/ Jang, E., Devin, C., Vanhoucke, V., and Levine, S. 10.1145/3372885.3373826. Grasp2vec: Learning object representations from self- supervised grasping. In 2nd Annual Conference on Hendrycks, D., Mazeika, M., Kadavath, S., and Song, D. Robot Learning, CoRL 2018, Zurich,¨ Switzerland, 29- Using self-supervised learning can improve model ro- 31 October 2018, Proceedings, volume 87 of Proceed- bustness and uncertainty. In Wallach, H. M., Larochelle, ings of Machine Learning Research, pp. 99–112. PMLR, H., Beygelzimer, A., d’Alche-Buc,´ F., Fox, E. B., and 2018. URL http://proceedings.mlr.press/ Garnett, R. (eds.), Advances in Neural Information v87/jang18a.html. Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Kaliszyk, C. and Urban, J. Femalecop: Fairly effi- December 8-14, 2019, Vancouver, BC, Canada, pp. cient machine learning connection prover. In Davis, 15637–15648, 2019. URL https://proceedings. M., Fehnker, A., McIver, A., and Voronkov, A. (eds.), neurips.cc/paper/2019/hash/ Logic for Programming, Artificial Intelligence, and a2b15837edac15df90721968986f7f8e-Abstract.Reasoning - 20th International Conference, LPAR-20 html. 2015, Suva, Fiji, November 24-28, 2015, Proceed- ings, volume 9450 of Lecture Notes in Computer Sci- Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., ence, pp. 88–96. Springer, 2015a. doi: 10.1007/ Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, 978-3-662-48899-7\ 7. URL https://doi.org/ S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ry- 10.1007/978-3-662-48899-7_7. der, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. Scaling laws for autoregressive gener- Kaliszyk, C. and Urban, J. Mizar 40 for mizar 40. J. ative modeling. CoRR, abs/2010.14701, 2020. URL Autom. Reason., 55(3):245–256, 2015b. doi: 10.1007/ https://doi.org/10. https://arxiv.org/abs/2010.14701. s10817-015-9330-8. URL 1007/s10817-015-9330-8. Huang, D., Dhariwal, P., Song, D., and Sutskever, I. Kaliszyk, C. and Urban, J. Learning-assisted theorem Gamepad: A learning environment for theorem proving. proving with millions of lemmas. J. Symb. Com- CoRR, abs/1806.00608, 2018. URL http://arxiv. put., 69:109–128, 2015c. doi: 10.1016/j.jsc.2014.09. org/abs/1806.00608. 032. URL https://doi.org/10.1016/j.jsc. 2014.09.032. Irving, G., Szegedy, C., Alemi, A. A., Een,´ N., Chollet, F., and Urban, J. DeepMath—deep sequence models for Kaliszyk, C., Urban, J., and Vyskocil, J. Lemmati- premise selection. In Advances in Neural Information zation for stronger reasoning in large theories. In Processing Systems, pp. 2235–2243, 2016. Lutz, C. and Ranise, S. (eds.), Frontiers of Combin- ing Systems - 10th International Symposium, FroCoS Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., 2015, Wroclaw, Poland, September 21-24, 2015. Pro- Leibo, J. Z., Silver, D., and Kavukcuoglu, K. Rein- ceedings, volume 9322 of Lecture Notes in Computer forcement learning with unsupervised auxiliary tasks. Science, pp. 341–356. Springer, 2015. doi: 10.1007/ In 5th International Conference on Learning Repre- 978-3-319-24246-0\ 21. URL https://doi.org/ sentations, ICLR 2017, Toulon, France, April 24-26, 10.1007/978-3-319-24246-0_21. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum? Kaliszyk, C., Chollet, F., and Szegedy, C. Holstep: A id=SJ6yPD5xg. machine learning dataset for higher-order logic theorem proving. In 5th International Conference on Learning Jakubuv, J. and Urban, J. ENIGMA: efficient learning- Representations, ICLR 2017, Toulon, France, April 24-26, based inference guiding machine. In Geuvers, H., Eng- 2017, Conference Track Proceedings. OpenReview.net, land, M., Hasan, O., Rabe, F., and Teschke, O. (eds.), 2017. URL https://openreview.net/forum? Intelligent Computer Mathematics - 10th International id=ryuxYmvel. Conference, CICM 2017, Edinburgh, UK, July 17-21, 2017, Proceedings, volume 10383 of Lecture Notes in Kaliszyk, C., Urban, J., Michalewski, H., and Olsak,´ Computer Science, pp. 292–302. Springer, 2017. doi: 10. M. Reinforcement learning of theorem proving. In 1007/978-3-319-62075-6\ 20. URL https://doi. Bengio, S., Wallach, H. M., Larochelle, H., Grauman, org/10.1007/978-3-319-62075-6_20. K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Con- Jakubuv,˚ J. and Urban, J. Hammering Mizar by learning ference on Neural Information Processing Systems 2018, clause guidance. arXiv preprint arXiv:1904.01677, 2019. NeurIPS 2018, December 3-8, 2018, Montreal,´ Canada, pp. 8836–8847, 2018. URL https://proceedings. Nie, P., Palmskog, K., Li, J. J., and Gligoric, M. Deep neurips.cc/paper/2018/hash/ generation of coq lemma names using elaborated terms. 55acf8539596d25624059980986aaa78-Abstract.In Peltier, N. and Sofronie-Stokkermans, V. (eds.), Au- html. tomated Reasoning - 10th International Joint Confer- ence, IJCAR 2020, Paris, France, July 1-4, 2020, Pro- Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., ceedings, Part II, volume 12167 of Lecture Notes in Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Computer Science, pp. 97–118. Springer, 2020. doi: Amodei, D. Scaling laws for neural language models. 10.1007/978-3-030-51054-1\ 6. URL https://doi. CoRR, abs/2001.08361, 2020. URL https://arxiv. org/10.1007/978-3-030-51054-1_6. org/abs/2001.08361. Noroozi, M. and Favaro, P. Unsupervised learning of Lample, G. and Charton, F. Deep learning for symbolic visual representations by solving jigsaw puzzles. In mathematics. In 8th International Conference on Learn- Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), ing Representations, ICLR 2020, Addis Ababa, Ethiopia, Computer Vision - ECCV 2016 - 14th European Confer- April 26-30, 2020. OpenReview.net, 2020. URL https: ence, Amsterdam, The Netherlands, October 11-14, 2016, //openreview.net/forum?id=S1eZYeHFDS. Proceedings, Part VI, volume 9910 of Lecture Notes in Computer Science, pp. 69–84. Springer, 2016. doi: Laskin, M., Srinivas, A., and Abbeel, P. CURL: con- 10.1007/978-3-319-46466-4\ 5. URL https://doi. trastive unsupervised representations for reinforcement org/10.1007/978-3-319-46466-4_5. learning. In Proceedings of the 37th International Con- ference on Machine Learning, ICML 2020, 13-18 July Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., 2020, Virtual Event, volume 119 of Proceedings of Ma- Ng, N., Grangier, D., and Auli, M. fairseq: A fast, chine Learning Research, pp. 5639–5650. PMLR, 2020. extensible toolkit for sequence modeling. In Ammar, URL http://proceedings.mlr.press/v119/ W., Louis, A., and Mostafazadeh, N. (eds.), Proceed- laskin20a.html. ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- Lederman, G., Rabe, M. N., Seshia, S., and Lee, E. A. Learn- tics: Human Language Technologies, NAACL-HLT 2019, ing heuristics for quantified boolean formulas through Minneapolis, MN, USA, June 2-7, 2019, Demonstra- reinforcement learning. In 8th International Confer- tions, pp. 48–53. Association for Computational Lin- ence on Learning Representations, ICLR 2020, Addis guistics, 2019. doi: 10.18653/v1/n19-4009. URL Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, https://doi.org/10.18653/v1/n19-4009. https://openreview.net/forum? 2020. URL Paliwal, A., Loos, S. M., Rabe, M. N., Bansal, K., and id=BJluxREKDB . Szegedy, C. Graph representations for higher-order logic and theorem proving. In The Thirty-Fourth AAAI Con- Li, W., Yu, L., Wu, Y., and Paulson, L. C. Isarstep: a ference on Artificial Intelligence, AAAI 2020, The Thirty- benchmark for high-level mathematical reasoning. In Second Innovative Applications of Artificial Intelligence International Conference on Learning Representations, Conference, IAAI 2020, The Tenth AAAI Symposium on 2021. URL https://openreview.net/forum? Educational Advances in Artificial Intelligence, EAAI id=Pzj6fzU6wkj. 2020, New York, NY, USA, February 7-12, 2020, pp. 2967– 2974. AAAI Press, 2020. URL https://aaai.org/ Logeswaran, L. and Lee, H. An efficient framework ojs/index.php/AAAI/article/view/5689. for learning sentence representations. In 6th Interna- tional Conference on Learning Representations, ICLR Pathak, D., Krahenb¨ uhl,¨ P., Donahue, J., Darrell, T., and 2018, Vancouver, BC, Canada, April 30 - May 3, Efros, A. A. Context encoders: Feature learning by in- 2018, Conference Track Proceedings. OpenReview.net, painting. In 2016 IEEE Conference on Computer Vi- 2018. URL https://openreview.net/forum? sion and Pattern Recognition, CVPR 2016, Las Vegas, id=rJvJXZb0W. NV, USA, June 27-30, 2016, pp. 2536–2544. IEEE Com- puter Society, 2016. doi: 10.1109/CVPR.2016.278. URL mathlib. The lean mathematical library. In Blanchette, J. https://doi.org/10.1109/CVPR.2016.278. and Hritcu, C. (eds.), Proceedings of the 9th ACM SIG- PLAN International Conference on Certified Programs Pfenning, F. and Paulin-Mohring, C. Inductively de- and Proofs, CPP 2020, New Orleans, LA, USA, Jan- fined types in the calculus of constructions. In Main, uary 20-21, 2020, pp. 367–381. ACM, 2020. doi: 10. M. G., Melton, A., Mislove, M. W., and Schmidt, D. A. 1145/3372885.3373824. URL https://doi.org/ (eds.), Mathematical Foundations of Programming Se- 10.1145/3372885.3373824. mantics, 5th International Conference, Tulane Univer- sity, New Orleans, Louisiana, USA, March 29 - April Scholze, P. Liquid tensor experiment. https: 1, 1989, Proceedings, volume 442 of Lecture Notes in //xenaproject.wordpress.com/2020/ Computer Science, pp. 209–228. Springer, 1989. doi: 12/05/liquid-tensor-experiment/, 2020. 10.1007/BFb0040259. URL https://doi.org/10. Formalization available at https://github.com/ 1007/BFb0040259. leanprover-community/lean-liquid.
Polu, S. and Sutskever, I. Generative language modeling Selsam, D., Lamm, M., Bunz,¨ B., Liang, P., de Moura, L., for automated theorem proving. CoRR, abs/2009.03393, and Dill, D. L. Learning a SAT solver from single-bit 2020. URL https://arxiv.org/abs/2009. supervision. In 7th International Conference on Learning 03393. Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https://openreview. Rabe, M. N., Lee, D., Bansal, K., and Szegedy, C. Mathe- net/forum?id=HJMC_iA5tm. matical reasoning via self-supervised skip-tree training. CoRR, abs/2006.04757, 2020. URL https://arxiv. Selsam, D., Han, J. M., de Moura, L., and Godefroid, P. org/abs/2006.04757. Universal policies for software-defined mdps. CoRR, abs/2012.11401, 2020. URL https://arxiv.org/ Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., abs/2012.11401. Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Song, K., Tan, X., Qin, T., Lu, J., and Liu, T. MASS: et al. Learning Transferable Visual Models From Natural masked sequence to sequence pre-training for language Language Supervision. generation. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Radford, A., Narasimhan, K., Salimans, T., Proceedings of the 36th International Conference on and Sutskever, I. Improving language un- Machine Learning, ICML 2019, 9-15 June 2019, Long derstanding by generative pre-training. URL Beach, California, USA, volume 97 of Proceedings of https://s3-us-west-2. amazonaws. com/openai- Machine Learning Research, pp. 5926–5936. PMLR, assets/researchcovers/languageunsupervised/language 2019. URL http://proceedings.mlr.press/ understanding paper. pdf, 2018. v97/song19d.html. Urban, J. MPTP - motivation, implementation, first ex- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., periments. J. Autom. Reason., 33(3-4):319–339, 2004. Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the doi: 10.1007/s10817-004-6245-1. URL https:// limits of transfer learning with a unified text-to-text trans- doi.org/10.1007/s10817-004-6245-1. former. J. Mach. Learn. Res., 21:140:1–140:67, 2020. URL http://jmlr.org/papers/v21/20-074. Urban, J. and Jakubuv, J. First neural conjecturing datasets html. and experiments. In Benzmuller,¨ C. and Miller, B. R. (eds.), Intelligent Computer Mathematics - 13th Interna- Rocktaschel,¨ T. and Riedel, S. End-to-end differentiable tional Conference, CICM 2020, Bertinoro, Italy, July proving. In Guyon, I., von Luxburg, U., Bengio, 26-31, 2020, Proceedings, volume 12236 of Lecture S., Wallach, H. M., Fergus, R., Vishwanathan, S. Notes in Computer Science, pp. 315–323. Springer, 2020. V. N., and Garnett, R. (eds.), Advances in Neural doi: 10.1007/978-3-030-53518-6\ 24. URL https:// Information Processing Systems 30: Annual Conference doi.org/10.1007/978-3-030-53518-6_24. on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 3788– Urban, J., Vyskocil, J., and Stepanek,´ P. Malecop ma- 3800, 2017. URL https://proceedings. chine learning connection prover. In Brunnler,¨ K. and neurips.cc/paper/2017/hash/ Metcalfe, G. (eds.), Automated Reasoning with Ana- b2ab001909a8a6f04b51920306046ce5-Abstract.lytic Tableaux and Related Methods - 20th Interna- html. tional Conference, TABLEAUX 2011, Bern, Switzerland, July 4-8, 2011. Proceedings, volume 6793 of Lecture Sanchez-Stern, A., Alhessi, Y., Saul, L. K., and Lerner, Notes in Computer Science, pp. 263–277. Springer, 2011. S. Generating correctness proofs with neural networks. doi: 10.1007/978-3-642-22119-4\ 21. URL https:// In Sen, K. and Naik, M. (eds.), Proceedings of the doi.org/10.1007/978-3-642-22119-4_21. 4th ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@PLDI van den Oord, A., Li, Y., and Vinyals, O. Representa- 2020, London, UK, June 15, 2020, pp. 1–10. ACM, tion learning with contrastive predictive coding. CoRR, 2020. doi: 10.1145/3394450.3397466. URL https: abs/1807.03748, 2018. URL http://arxiv.org/ //doi.org/10.1145/3394450.3397466. abs/1807.03748. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., A. Datasets Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U., Pre-training datasets Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, We pre-train on WebMath as described in (Polu & S. V. N., and Garnett, R. (eds.), Advances in Neural Sutskever, 2020). WebMath pre-trained models are also Information Processing Systems 30: Annual Conference pre-trained on the mix used by GPT-3 (Brown et al., 2020) on Neural Information Processing Systems 2017, which includes a filtered CommonCrawl, WebText2, December 4-9, 2017, Long Beach, CA, USA, pp. 5998– Book1, Book2 and Wikipedia. WebMath includes 6008, 2017. URL https://proceedings. Python-only GitHub data, as well as arXiv and Math neurips.cc/paper/2017/hash/ StackExchange. 3f5ee243547dee91fbd053c1c4a845aa-Abstract. html. From these datasets, a potential risk for test-set contamina- tion (presence of mathlib) exists for the crawled datasets, Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, namely CommonCrawl, WebText2, and (in case of a P. Extracting and composing robust features with de- filtering bug) Python-only GitHub. The other datasets noising autoencoders. In Cohen, W. W., McCallum, (in particular arXiv and Math StackExchange) may A., and Roweis, S. T. (eds.), Machine Learning, Pro- contain short references of mathlib code but in shape and ceedings of the Twenty-Fifth International Conference forms that would not lead to effective contamination. (ICML 2008), Helsinki, Finland, June 5-9, 2008, vol- ume 307 of ACM International Conference Proceed- To assess the contamination risk related with the ing Series, pp. 1096–1103. ACM, 2008. doi: 10. crawled datasets, we run full-scans searching for the mathlib CommonCrawl 1145/1390156.1390294. URL https://doi.org/ following -specific strings on , WebText2 GitHub 10.1145/1390156.1390294. , and Python-only . Note that we ex- pect these strings to match even if presented in HTML on Wang, M. and Deng, J. Learning to prove theorems the Web as these datasets contain text rendered versions of by learning to generate theorems. In Larochelle, H., the page crawled online. Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), "{ rintro h" Advances in Neural Information Processing Systems 33: "{ rcases h" Annual Conference on Neural Information Processing "irrational_sqrt_two : irrational (sqrt 2)" Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings. Despite the two first strings occurring respectively 266 and neurips.cc/paper/2020/hash/ 101 times in mathlib, we found 0 occurrence of any of d2a27e83d429f0dcae6b937cf440aeb1-Abstract.the three strings in WebText2, Python-only GitHub, or html. CommonCrawl; negating any suspicion of effective test-set contamination. Whalen, D. Holophrasm: a neural automated theorem prover for higher-order logic. CoRR, abs/1608.02644, At the same time we looked for the following Metamath 2016. URL http://arxiv.org/abs/1608. specific and HOL specific strings: 02644. Metamath: "( ph -> A=C)" Wiedijk, F. The De Bruijn factor, 2000. URL "( ph -> A R C )" http://www.cs.ru.nl/F.Wiedijk/factor/ "( sqrt 8 2 ) e/ QQ" factor.pdf. HOL: "apply (rule " Wu, Y., Rabe, M. N., Li, W., Ba, J., Grosse, R. B., and "apply (drule " Szegedy, C. LIME: learning inductive bias for primitives of mathematical reasoning. CoRR, abs/2101.06223, 2021. We found 0 occurrence of the Metamath-related strings but URL https://arxiv.org/abs/2101.06223. interestingly found a non-negligible amount of HOL-related Yang, K. and Deng, J. Learning to prove theorems via documents, which does not constitute a test-set contamina- interacting with proof assistants. In Chaudhuri, K. and tion but potentially benefits the downstream tasks studied in Salakhutdinov, R. (eds.), Proceedings of the 36th Inter- this paper. national Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 Dataset sizes of Proceedings of Machine Learning Research, pp. 6984– • tactic: ≈128K examples. 6994. PMLR, 2019. URL http://proceedings. mlr.press/v97/yang19a.html. • mix1 – Next lemma prediction: ≈2.5M examples "decl_premises":[["absurd", "∀ {a b : – Proof term prediction: ≈2.9M examples Prop}, a → ¬a → b"], ["absurd", "∀ {a b : Prop}, a → ¬a → b"], • mix2 ["decidable.not_imp", "∀ {a b : Prop} – Skip-proof: ≈1.7M examples [_inst_1 : decidable a], ¬(a → b) ↔ a ∧ ¬b"], – Type-prediction: ≈1.7M examples ["iff.mp", "∀ {a b : Prop}, (a ↔ b) → – Tactic state elaboration: ≈346K examples a → b"], ["and.dcases_on", – Proof term elaboration: ≈1.0M examples "∀ {a b : Prop} {C : a ∧ b → Prop} (n – Premise classification: ≈9.3M examples : a ∧ b), (∀ (left : a) (right : b), C – Local context classification: ≈2.0M examples _) → C n"], ["decidable.not_or_of_imp", "∀ {a b : – Theorem naming: ≈32K examples. Prop} [_inst_1 : decidable a], (a → b) → ¬a ∨ b"], ["or.dcases_on", Example datapoints "∀ {a b : Prop} {C : a ∨ b → Prop} (n : a b), ( (h : a), C _) ( (h : We present datapoints extracted from a toy example, namely ∨ ∀ → ∀ b), C _) → C n"], the proof of the Peirce identity, viz. ["em", "∀ (p : Prop), p ∨ ¬p"], lemma peirce_identity{PQ:Prop} : ((P → ["or.elim", "∀ {a b c : Prop}, a ∨ b → Q) → P) → P := (a → c) → (b → c) → c"]], begin "decl_premises_mask":[false, false, apply or.elim(emP), true, false, false, false, false, introsh_, false, false], exacth, "goal":"∀ {b : Prop} [_inst_1 : tauto! decidable P], ¬(P → b) ↔ P ∧ ¬b", end "proof_term":"decidable.not_imp", "result":"λ {P Q : Prop}, (em P).elim (λ (h : P) (αˇ : (P → Q) → P), h) (λ (αˇ : From this, we can extract four tactic datapoints (i.e. ¬P) (αˇ_1 : (P → Q) → P), human-generated tactic proof steps): (decidable.not_or_of_imp αˇ _1).dcases_on (λ (αˇ_1 : ¬(P → Q)), -- GOALPQ: Prop ` ((P → Q) → P) → P ((PREDICT Q (classical.prop_decidable PROOFSTEP apply or.elim(emP) P)).mp αˇ_1).dcases_on (λ (αˇ_1_left : -- GOALPQ: Prop P ((P Q) P) ` → → → → P) (αˇ_1_right : ¬Q), absurd αˇ_1_left αˇ PPQ: Prop P ((P Q) P) `¬ → → → )) (λ (αˇ_1 : P), absurd αˇ_1 αˇ))", P PROOFSTEP introsh_ → "next_lemma":["decidable.not_imp", "∀ {a -- GOALPQ: Prop,h:P, :(P Q) αˇ → → b : Prop} [_inst_1 : decidable a], ¬(a P PPQ: Prop P ((P Q) ` `¬ → → → → b) ↔ a ∧ ¬b"], P) → P PROOFSTEP exacth "goal_is_prop":true, -- GOALPQ: Prop `¬ P → ((P → Q) → P) "verbose_proof_term":"@decidable.not_imp → P PROOFSTEP tauto! P", "verbose_goal":"∀ {b : Prop} [_inst_1 : In contrast, we can extract dozens of raw PACT datapoints. decidable P], ¬(P → b) ↔ P ∧ ¬b", Due to space constraints, we list a representative sample "verbose_result":"λ {P Q : Prop}, (em of four such datapoints, from each of which we can de- P).elim (λ (h : P) (αˇ : (P → Q) → P), h) (λ (αˇ : ¬P) (αˇ_1 : (P → Q) → P), rive the nine self-supervised auxiliary PACT tasks studied (@decidable.not_or_of_imp (P → Q) P in our present work. For example, proof term prediction (classical.prop_decidable (P → Q)) αˇ is precisely predicting the "proof_term" given the con- _1).dcases_on (λ (αˇ_1 : ¬(P → Q)), catenation of "hyps", "`", and the "goal", skip-proof is (@iff.mp (¬(P → Q)) (P ∧ ¬Q) (PREDICT predicting the "proof_term" given "result", etc. Q (classical.prop_decidable P)) αˇ _1).dcases_on (λ (αˇ_1_left : P) DATAPOINT: (αˇ_1_right : ¬Q), @absurd P P αˇ_1_left --- αˇ)) (λ (αˇ_1 : P), @absurd P P αˇ_1 αˇ))"} { "decl_nm":"peirce_identity", --- "decl_tp":"∀ {P Q : Prop}, ((P → Q) → P) → P", DATAPOINT: "hyps":[["P", "Prop"], ["Q", "Prop"], --- ["αˇ", "¬P"], ["αˇ_1", "(P → Q) → P"], { "decl_nm":"peirce_identity", ["αˇ_1", "¬(P → Q)"]], "decl_tp":"∀ {P Q : Prop}, ((P → Q) → "hyps_mask":[true, false, false, false, P) → P", false], "hyps":[["P", "Prop"], ["Q", "Prop"], ["αˇ", "¬P"], ["αˇ_1", "(P → Q) → P"], ["αˇ_1", "¬(P → Q)"]], ["αˇ_1", "¬(P → Q)"]], "hyps_mask":[true, true, false, false, "hyps_mask":[false, true, false, false, false], false], "decl_premises":[["absurd", "∀ {a b : "decl_premises":[["absurd", "∀ {a b : Prop}, a → ¬a → b"], Prop}, a → ¬a → b"], ["absurd", "∀ {a b : Prop}, a → ¬a → ["absurd", "∀ {a b : Prop}, a → ¬a → b"], b"], ["decidable.not_imp", "∀ {a b : Prop} ["decidable.not_imp", "∀ {a b : Prop} [_inst_1 : decidable a], ¬(a → b) ↔ a [_inst_1 : decidable a], ¬(a → b) ↔ a ∧ ¬b"], ∧ ¬b"], ["iff.mp", "∀ {a b : Prop}, (a ↔ b) → ["iff.mp", "∀ {a b : Prop}, (a ↔ b) → a → b"], a → b"], ["and.dcases_on", ["and.dcases_on", "∀ {a b : Prop} {C : a ∧ b → Prop} (n "∀ {a b : Prop} {C : a ∧ b → Prop} (n : a ∧ b), (∀ (left : a) (right : b), C : a ∧ b), (∀ (left : a) (right : b), C _) → C n"], _) → C n"], ["decidable.not_or_of_imp", "∀ {a b : ["decidable.not_or_of_imp", "∀ {a b : Prop} [_inst_1 : decidable a], (a → b) Prop} [_inst_1 : decidable a], (a → b) → ¬a ∨ b"], → ¬a ∨ b"], ["or.dcases_on", ["or.dcases_on", "∀ {a b : Prop} {C : a ∨ b → Prop} (n "∀ {a b : Prop} {C : a ∨ b → Prop} (n : a ∨ b), (∀ (h : a), C _) → (∀ (h : : a ∨ b), (∀ (h : a), C _) → (∀ (h : b), C _) → C n"], b), C _) → C n"], ["em", "∀ (p : Prop), p ∨ ¬p"], ["em", "∀ (p : Prop), p ∨ ¬p"], ["or.elim", "∀ {a b c : Prop}, a ∨ b → ["or.elim", "∀ {a b c : Prop}, a ∨ b → (a → c) → (b → c) → c"]], (a → c) → (b → c) → c"]], "decl_premises_mask":[false, false, "decl_premises_mask":[false, false, true, false, false, false, false, false, false, false, false, false, false, false], false, false], "goal":"∀ [_inst_1 : decidable P], ¬(P → "goal":"Prop", Q) ↔ P ∧ ¬Q", "proof_term":"Q", "proof_term":"decidable.not_imp", "result":"λ {P Q : Prop}, (em P).elim (λ "result":"λ {P Q : Prop}, (em P).elim (λ (h : P) (αˇ : (P → Q) → P), h) (λ (αˇ : (h : P) (αˇ : (P → Q) → P), h) (λ (αˇ : ¬P) (αˇ_1 : (P → Q) → P), ¬P) (αˇ_1 : (P → Q) → P), (decidable.not_or_of_imp αˇ (decidable.not_or_of_imp αˇ _1).dcases_on (λ (αˇ_1 : ¬(P → Q)), _1).dcases_on (λ (αˇ_1 : ¬(P → Q)), (decidable.not_imp.mp αˇ_1).dcases_on ((PREDICT (classical.prop_decidable (λ (αˇ_1_left : P) (αˇ_1_right : ¬Q), P)).mp αˇ_1).dcases_on (λ (αˇ_1_left : absurd αˇ_1_left αˇ)) (λ (αˇ_1 : P), P) (αˇ_1_right : ¬Q), absurd αˇ_1_left αˇ absurd αˇ_1 αˇ))", )) (λ (αˇ_1 : P), absurd αˇ_1 αˇ))", "next_lemma":["Q", "Prop"], "next_lemma":["decidable.not_imp", "∀ {a "goal_is_prop":false, b : Prop} [_inst_1 : decidable a], ¬(a "verbose_proof_term":"Q", → b) ↔ a ∧ ¬b"], "verbose_goal":"Prop", "goal_is_prop":true, "verbose_result":"λ {P Q : Prop}, (em "verbose_proof_term":"@decidable.not_imp P).elim (λ (h : P) (αˇ : (P → Q) → P), P Q", h) (λ (αˇ : ¬P) (αˇ_1 : (P → Q) → P), "verbose_goal":"∀ [_inst_1 : decidable (@decidable.not_or_of_imp (P → Q) P P], ¬(P → Q) ↔ P ∧ ¬Q", (classical.prop_decidable (P → Q)) αˇ "verbose_result":"λ {P Q : Prop}, (em _1).dcases_on (λ (αˇ_1 : ¬(P → Q)), P).elim (λ (h : P) (αˇ : (P → Q) → P), ((@decidable.not_imp P PREDICT h) (λ (αˇ : ¬P) (αˇ_1 : (P → Q) → P), (classical.prop_decidable P)).mp αˇ (@decidable.not_or_of_imp (P → Q) P _1).dcases_on (λ (αˇ_1_left : P) (classical.prop_decidable (P → Q)) αˇ (αˇ_1_right : ¬Q), @absurd P P αˇ_1_left _1).dcases_on (λ (αˇ_1 : ¬(P → Q)), αˇ)) (λ (αˇ_1 : P), @absurd P P αˇ_1 αˇ))"} (@iff.mp (¬(P → Q)) (P ∧ ¬Q) (PREDICT --- (classical.prop_decidable P)) αˇ _1).dcases_on (λ (αˇ_1_left : P) DATAPOINT: (αˇ_1_right : ¬Q), @absurd P P αˇ_1_left --- αˇ)) (λ (αˇ_1 : P), @absurd P P αˇ_1 αˇ))"} { "decl_nm":"peirce_identity", --- "decl_tp":"∀ {P Q : Prop}, ((P → Q) → P) → P", DATAPOINT: "hyps":[["P", "Prop"], ["Q", "Prop"], --- ["αˇ", "¬P"], ["αˇ_1", "(P → Q) → P"], { "decl_nm":"peirce_identity", "decl_tp":"∀ {P Q : Prop}, ((P → Q) → P) → P", Table 1. Counting the number of semicolon-chained tactics pre- "hyps":[["P", "Prop"], ["Q", "Prop"], dicted by our models that appear in successful proofs. Each column ["αˇ", "¬P"], ["αˇ_1", "(P → Q) → P"], headed by a number n; indicates the number of times that a sug- ["αˇ_1", "¬(P → Q)"]], gestion appeared with n occurrences of ‘;’. "hyps_mask":[false, false, false, false, false], "decl_premises":[["absurd", "∀ {a b : MODEL 1; 2; 3; 4; MEAN Prop}, a → ¬a → b"], ["absurd", "∀ {a b : Prop}, a → ¬a → wm-to-tt 215 49 2 0 1.199 b"], wm-to-tt-m1 186 39 5 1 1.225 ["decidable.not_imp", "∀ {a b : Prop} wm-to-tt-m1-m2 328 82 12 3 1.271 [_inst_1 : decidable a], ¬(a → b) ↔ a ∧ ¬b"], ["iff.mp", "∀ {a b : Prop}, (a ↔ b) → a → b"], B. Experiments ["and.dcases_on", "∀ {a b : Prop} {C : a ∧ b → Prop} (n Chained tactic prediction : a ∧ b), (∀ (left : a) (right : b), C _) → C n"], Individual Lean tactics are chained together with commas. ["decidable.not_or_of_imp", "∀ {a b : However, the Lean interactive tactic DSL also includes a Prop} [_inst_1 : decidable a], (a → b) number of other tactic combinators for creating composite → ¬a ∨ b"], tactics. A frequently used combinator is the infix semicolon ["or.dcases_on", t; s which will perform the tactic t and then apply the "∀ {a b : Prop} {C : a ∨ b → Prop} (n s t : a ∨ b), (∀ (h : a), C _) → (∀ (h : tactic to each of the resulting subgoals produced by . b), C _) → C n"], Our data pipeline for human tactic proof steps treats these ["em", "∀ (p : Prop), p ∨ ¬p"], semicolon-chained tactics as a single string for the language ["or.elim", "∀ {a b c : Prop}, a ∨ b → modeling objective. Thus, our models learn to occasionally (a → c) → (b → c) → c"]], emit multiple-step tactic predictions using semicolons. For "decl_premises_mask":[false, false, wm-to-tt-m1-m2 false, false, false, false, false, example, solved the following lemma false, false], in category theory with a single prediction chaining four "goal":"Π (a : Prop), decidable a", tactics in a row: "proof_term":"classical.prop_decidable", "result":"λ {P Q : Prop}, (em P).elim (λ theorem (h : P) (αˇ : (P → Q) → P), h) (λ (αˇ : category_theory.grothendieck.congr ¬P) (αˇ_1 : (P → Q) → P), {XY: grothendieckF}{fg:X −→ Y} (decidable.not_or_of_imp αˇ (h:f=g): _1).dcases_on (λ (αˇ_1 : ¬(P → Q)), f.fiber= eq_to_hom(by substh) (decidable.not_imp.mp αˇ_1).dcases_on g.fiber := (λ (αˇ_1_left : P) (αˇ_1_right : ¬Q), begin absurd αˇ_1_left αˇ)) (λ (αˇ_1 : P), rcasesX; rcasesY; substh; simp absurd αˇ_1 αˇ))", end
"next_lemma":["classical.prop_decidable", One way of measuring the sophistication of predicted tactics "Π (a : Prop), decidable a"], is to consider the number of successful proofs on the evalu- "goal_is_prop":false, ation set which have this composite form using semicolon- "verbose_proof_term":"classical.prop_decidable",chaining. We display this analysis in Table 1, which shows "verbose_goal":"Π (a : Prop), decidable that training with PACT in addition to the human-made tac- a", tics causes longer semicolon-chained tactics to be success- "verbose_result":"λ {P Q : Prop}, (em fully predicted during theorem proving. This is remarkable P).elim (λ (h : P) (αˇ : (P → Q) → P), h) (λ (αˇ : ¬P) (αˇ_1 : (P → Q) → P), because the semicolon idiom is specific to the tactic DSL (@decidable.not_or_of_imp (P → Q) P which does not occur in the PACT data whatsoever, and yet (PREDICT (P → Q)) αˇ_1).dcases_on (λ the co-training causes longer and more frequent successful (αˇ_1 : ¬(P → Q)), composite tactic predictions. ((@decidable.not_imp P Q (PREDICT P)).mp αˇ_1).dcases_on (λ (αˇ_1_left : P) (αˇ_1_right : ¬Q), @absurd P P αˇ Theorem naming case study _1_left αˇ)) (λ (αˇ_1 : P), @absurd P P αˇ _1 αˇ))"} We included theorem naming as part of the PACT task suite. --- By mathlib convention, theorem names are essentially snake-cased, natural language summaries of the type signa- Correct top-1 guesses
∀ {α : Type u_1}{β : Type u_2}[_inst_1: decidable_eq α] [_inst_2: decidable_eq β](s: finset α)(t: finset β), Theorem statement s.productt=s.bUnion (λ (a: α), finset.image( λ (b: β), (a,b))t)
Ground truth finset.product_eq_bUnion
∀ {α : Type u_1}{β : Type u_2}[_inst_1: topological_space α] Theorem statement [_inst_2: topological_space β]{f: α → β}, quotient_mapf → function.surjectivef
Ground truth quotient_map.surjective
∀ {α : Type u_1}{β : Type u_2}(f: α → option β) Theorem statement (x: option α),x.pbind( λ (a: α)(_x:a ∈ x),fa)=x.bindf
Ground truth option.pbind_eq_bind
∀ {C: Typeu 1}[_inst_1: category_theory.categoryC] {G:C ⇒ C}[_inst_2: category_theory.comonadG] ∼ Theorem statement {AB: category_theory.comonad.coalgebraG}(h:A.A = B.A) (w:A.a G.maph.hom=h.hom B.a), (category_theory.comonad.coalgebra.iso_mkhw).hom.f=h.hom
Ground truth category_theory.comonad.coalgebra.iso_mk_hom_f
∀ {k : Type u_1}{E: Type u_2}[_inst_1: is_R_or_C, k] [_inst_2: inner_product_space k E] [_inst_4: normed_space R E][_inst_5: is_scalar_tower R k E] Theorem statement (px:E × E), ⇑(fderiv_inner_clmp)x= has_inner.innerp.fstx.snd+ has_inner.innerx.fstp.snd
Ground truth fderiv_inner_clm_apply
Figure 8. A sample of correct top-1 guesses by our best model wm-to-tt-m1-m2 on the theorem naming task. We performed this experiment on the future-mathlib evaluation set, which comprises entirely unseen theorems added to mathlib only after we last extracted training data. Incorrect guesses
∀ {α : Type u_1}(t: ordnode α)(x: α), Theorem statement t.dual.find_min0 x= ordnode.find_max 0 xt
ordinal.find_min0_eq, ordinal.find_min0_eq_max0, ordinal.find_min0_def, Guesses (top 8) ordinal.find_min0_eq_max, ordinal.find_min0, ordinal.dual_find_min0, ordinal.find_min0_gt, ordinal.find_min0_q 0 Ground truth ordnode.find_min _dual
∀ {α : Type u_1}{β : Type u_3}{γ : Type u_5}[_inst_1: measurable_space α][_inst_3: measurable_space β] [_inst_5: measurable_space γ]{µ : measure_theory.measure α} {ν : measure_theory.measure β} Theorem statement [_inst_8: measure_theory.sigma_finite ν] {f: α × β → γ}, ae_measurablef( µ.prod ν) → (∀m(x: α) ∂µ, ae_measurable( λ (y: β),f(x,y)) ν)
measure_theory.ae_prod, measure_theory.ae_of_ae_prod, measure_theory.ae_eq_prod_of_ae, measure_theory.ae_ae_of_ae_prod, Guesses (top 8) measure_theory.ae_measure_prod_mk_left, measure_theory.ae_prod_of_ae_prod, measure_theory.ae_measure_prod, measure_theory.ae_eq_refl Ground truth ae_measurable.prod_mk_left
∀ {α : Type u_1}{β : Type u_2}{γ : Type u_3} {f: filter α}{h: set α → set β}{m: γ → β} Theorem statement {l: filter γ}, filter.tendstoml(f.lift 0 h) ↔ ∀ (s: set α),s ∈ f → (∀f (a: γ) inl,ma ∈ hs)
Guesses (top 8) filter.tendsto_lift0_iff, filter.tendsto_lift0_def 0 Ground truth filter.tendsto_lift
∀ {R: Type}[_inst_1: comm_ringR] {d: }(f: √d +∗ R), Theorem statement Z Z → ↑(⇑(zsqrtd.lift.symm)f)= ⇑f zsqrtd.sqrtd
zsqrtd.coe_lift_symm, zsqrtd.coe_lift.symm, zsqrtd.lift.coe_symm_apply, Guesses (top 8) zsqrtd.lift_symm_apply, zsqrtd.lift.coe_coe_symm, zsqrtd.lift.coe_symm_coe, zsqrtd.lift.symm_coe_zsqrtd, zsqrtd.lift_symm_to_zsqrtd Ground truth zsqrtd.lift_symm_apply_coe
Figure 9. A sample of incorrect guesses by our best model wm-to-tt-m1-m2 on the theorem naming task. We performed this experiment on the future-mathlib evaluation set, which comprises entirely unseen theorems added to mathlib only after we last extracted training data. Most of the top-8 guesses displayed in the above table are very similar to the ground truth, in some cases being equivalent up to permutation of underscore-separated tokens. Note that for the first example, the concept of ordnode was not in the training data whatsoever and all predictions are in the syntactically similar ordinal namespace. ture of a theorem, and so the theorem naming task is analo- , "solve_by_elim" gous to a formal-to-informal translation task. We evaluate , "norm_cast" the ability of our best model (in terms of theorem proving ] success rate) wm-to-tt-m1-m2 on its ability to guess the- orem names on the completely unseen future-mathlib Unlike the gptf backend, which generates a list of candi- set of theorems. The distribution shift inherent in the dates in parallel independently, tidy enjoys the advantage future-mathlib dataset particularly impacts the the- that the list of tactics it emits is carefully chosen and ordered orem naming task, because many of the ground-truth names in order to optimize the proof search—this is based on the will involve names for concepts that were only defined in “waterfall” technique of the human-style automated theorem mathlib after we extracted our training data. prover described in ((Ganesalingam & Gowers, 2017)).
On the ≈2.8K future-mathlib theorems, we queried Computational resource estimates wm-to-tt-m1-m2 for up to N = 16 candidates. We order these candidates into a list xs by decreasing cumu- For each evaluation loop over the test set, we distributed lative log-probability and calculate the top-K accuracy by the theorems over a pool of 32 CPU workers whose infer- checking if any of the first K candidates of xs match the ence requests were load-balanced over 4 V100 GPUs. Each ground truth exactly. The model wm-to-tt-m1-m2 was evaluation required ≈10 hours with ≈30% GPU utilization. able to achieve 20.1% top-1 accuracy, 21.1% top-3 accuracy, We observed that our evaluation was bottlenecked by infer- 26.7% top-10 accuracy, and 30.0% top-16 accuracy. We ence and in practice, we hosted up to three evaluation loops display a sample of correct top-1 guesses (Figure 8) and at once on a VM with 80 logical cores without achieving a sample of failed guesses in (Figure 9). We note that the full CPU utilization. In addition to the wall-clock timeout failed guesses, while containing no syntactic matches, are of 600s, we also limited the proof search to a logical time- both semantically reasonable and syntactically very similar out of 512 iterations, where one iteration corresponds to a to the ground truth. single expansion of a node of the BFS search tree. In prac- tice, so much time was spent either blocked on inference or Test set evaluation breakdown by module performing the tactic executions in the inner loop of each iteration that we rarely exceeded the logical timeout, usually Lean’s mathlib is organized into top-level modules, exceeding the wall-clock timeout instead. which roughly organize theorems into mathematical sub- ject area. In Figure 10, we break down the evaluation Fine-tuning on our largest dataset mix1 + mix2 + results on our test set between our PACT-trained mod- tactic required 26 hours using 64 A100 GPUs ex- els wm-to-tt-m1-m2 and wm-to-tt-m1 and our base- hibiting high FP16 usage, totalling an estimated ≈1.5K lines wm-to-tt and tidy. We see that full PACT mostly A100(FP16)-hours. This gives an estimated cost of 17.33 dominates over co-training on just the mix1 tasks over A100(FP16)-hours per billion elapsed tokens during train- all subject areas, and that wm-to-tt-m1 dominates the ing. We note that when calculating the number of elapsed model wm-to-tt trained on human tactic proof steps only. tokens for training, we overestimate the actual number of tokens effectively trained on by summing full context win- Baseline description dows (in this case, 2048 tokens).
The tidy backend is determined by a constant oracle C. Example proofs Ω : tactic_state → list(string × float) Lean’s mathlib is one of the most active open-source which always returns the same list of tactics, namely: software projects in the world. More than one-third of the proofs found by our models are shorter and produce smaller meta def tidy_default_tactics: list proof terms than the ground truth, leading to dozens of GPT- (string × float) := list.map(flip prod.mk 0.0) [ f co-authored commits to mathlib. We examine some of "refl" the proofs found by our models in more detail. , "exact dec_trivial" , "assumption" lie algebra.morphism.map bot iff , "tactic.intros1" , "tactic.auto_cases" This proof produces a proof term which is 4X smaller than , "apply_auto_param" the original: , "dsimp at ∗" , "simp at ∗" lemma map_bot_iff:I.mapf= ⊥ ↔ I ≤ , "ext1" f.ker := , "fsplit" by{ rw ← le_bot_iff, apply , "injections_and_clear" lie_ideal.map_le_iff_le_comap} iue10. Figure hspofpoue ro emwihi 2 mle than smaller 12X is which term proof a produces proof This h rgnl ua-rte ro smc ogr viz. longer, much is proof human-written original, The { of_equiv theorem original: the begin the = f .map I : map_bot_iff lemma model the dominate PACT using trained models the and primrec.of end := e primrec primcodable.of_equiv := haveI by f : suffices { [eq_bot_iff, rwa { lie_ideal.map, unfold le_ker_iff, rw tidy exact set.mem_image], set.mem_singleton_iff, }, lie_submodule.lie_span_eq], L h,}, at lie_submodule.bot_coe] set.image_subset_iff, lie_submodule.lie_span_le, h, intros split; := f.ker }, hy, y rintros { split, [lie_submodule.bot_coe, rw x, ext ,} }, }, [hx], simp 0, use , hx intros { 0 [this, rw { ),
aeiears o-ee oue nLean’s in modules top-level across baseline Success Rate 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 radw ftermpoigscesrt nthe on rate success proving theorem of breakdown A
equiv logic h hx hy, y, algebra 00 β = I : {e } order ↑
i data ( rw , ⊥ β
R lie_ideal : category_theory ' ← ↔ ⊥ α h exact hx,
: } control Test SetEvaluationBreakdownByModules α I group_theory mathlib ≤ e; wm-to-tt combinatorics
test topology esethat see We . rie nhmntci ro steps. proof tactic human on trained sfra hyke,ti rc a ee sdbfr nthe in before used never was trick this knew, they as far As hspofdmntae u oe’ irr nweg and knowledge library model’s our demonstrates proof This pack- that of maintainer and proof original the of author The primcodable : letI by end begin : x ( real.tan_eq_sin_div_cos lemma selection. premise at ability real.tan computability commented: age linear_algebra e for set [complex .tan_eq_sin_div_cos, only simp rw primrec.encode encode_iff.1 primcodable.of_equiv of_real_div, of_real_tan] of_real_cos , := of_real_sin, x cos / x sin = x tan x) (e encode as = isomorphism. defined is x function encode encode the when alence translate to way a it’s encode set_theory ← wm-to-tt-m1-m2 wm-to-tt-m1-m2 of_real_inj, analysis eq f. primrec.encode iff.1 geometry sin package. dynamics div , tidy (baseline) wm-to-tt (tacticsteponly) wm-to-tt-m1 (PACTmix1only) wm-to-tt-m1-m2 (PACTfull) primrec
otydominates mostly computability wm-to-tt-m1 cos β
:= number_theory α exact e; measure_theory cosa equiv- an across where
, field_theory wm-to-tt wm-to-tt-m1 sclever, is e R ring_theory sthe is : ) and , Our model was able to predict this entire list of {g: α}: ||g|| ≤ 0 ↔ g=0 := simp lemmas in one shot. Note that the lemma by{ simpa[le_antisymm_iff, norm_nonneg] complex.tan_eq_sin_div_cos in this list is the using @norm_eq_zero α _g} -- ground truth: complex number version of the result, i.e. ∀ (x : ), C -- by{ rw ←[dist_zero_right], tan x = sin x / cos x. The previous human- -- exact dist_le_zero} written version of the proof did not use the more general version of the lemma on complex numbers, demonstrating The lemmas supplied between the square brackets are used our model’s ability to find more general cases of lemmas. to simplify the main goal. The lemma supplied after the We contrast this with the human-written ground truth, which keyword using can further simplify the lemmas supplied is more complex and performs a case analysis using the com- between the square brackets. The @ modifier makes all argu- plex cosine: ments explicit. The string @norm_eq_zero never appeared lemma tan_eq_sin_div_cos: tanx= sinx in our training data but the prediction includes the correct / cosx := number of correctly typed arguments, and even replaces the ifh: complex.cosx=0 then by simp second argument with a placeholder _, correctly guessing ∗ [sin, cos, tan, , complex.tan, that it can be inferred by the elaborator. Finally, this again div_eq_mul_inv] at ∗ else showcases the strength of our models as premise selectors: by rw[sin, cos, tan, complex.tan, ← all three lemmas le_antisymm_iff, norm_nonneg, and of_real_inj, div_eq_mul_inv, mul_re]; norm_eq_zero were not used in the human-supplied proof simp[norm_sq,(div_div_eq_div_mul__ but are necessary for this proof. _).symm, div_selfh]; refl Moving forward, we hope that our neural theorem provers will continue to find ways to improve mathlib and assist sym2.is diag iff proj eq in creating new proofs. More generally, we hope neural The proof of this lemma is longer than the ground truth theorem proving will one day be become a routine part of and was not contributed to mathlib, but we describe it the formalization workflow. here because the proof is original and includes a nontrivial instantiation of an existential quantifier. D. Source code theorem sym2.is_diag_iff_proj_eq(z: α × Our source code will be made available at the following ): α repositories: is_diag z ↔ z.1=z.2 := begin J K intros, Lean theorem proving environment : simp only[is_diag, prod.ext_iff, https://www.github.com/ quot.exists_rep, iff_true, not_true, jesse-michael-han/lean-tpe-public eq_self_iff_true], simp[diag], split, Tactic step data pipeline : { rintros hy, hyi, cases hy; refl}, introh, casesz, existsi z_snd, https://www.github.com/jasonrute/ casesh, refl, lean-proof-recording-public end PACT data pipeline : Before existsi z_snd, the goal state is https://www.github.com/ jesse-michael-han/lean-step-public z_fst z_snd: α h:(z_fst, z_snd).fst=(z_fst, z_snd).snd `∃ (y: α), (y,y) ≈ (z_fst, z_snd)
This goal state never appeared in mathlib. norm le zero iff The following proof is remarkable because it uses fewer tactic steps and takes a different route to the proof than the ground truth, uses a complex idiom simpa [...] using@... , and was predicted in one shot. lemma norm_le_zero_iff{ α : Type u_1} [_inst_1: normed_group α]