Arxiv:1305.7074V1 [Physics.Chem-Ph]

arXiv:1305.7074v1 [physics.chem-ph] 30 May 2013 oetmi h omo h aeil eoeinitia- genome materials the considerable of gained form tive of the has discovery in materials accelerated momentum novel the and serve improved that treatment data evrt aclt eeateetoi rprisfrmil- chromophores for of properties lions en- electronic relevant Project calculate Energy of to Clean deavor members Harvard advertised computing, widely distributed the quan- and using chemistry calcu- example, For tum be accuracy. routinely satisfying can electronic with lated compounds where virtual point of the properties to progressed have hardware tutdfreetoi tutr rpris Examples using potentials exchange-correlation properties. for structure QSPRs con- include be electronic also struc- for can quantitative structed QSPRs chem- the bioinformatics, in and prevalent to informatics (QSPRs) analogy relationships property close ture In cal- structure culations. electronic high-level of compu- demanding outcomes of tationally outcomes only also not but have measurements predicting methods, experimental to (ML) applied learning been machine already or statistical analysis, sophisticated data increasingly resources, tational millions to compounds? thousands chemical different the for of reflect obtained properly already that of information models properties robust structure construct systematically electronic manner. to struc- quantitative we from and relate are predictive distill that How a insights in to property crucial how to the ture obvious data not resulting is the It however: persists, ahn erigo oeua lcrncPoete nChe in Properties Electronic Molecular of Learning Machine h oitlne o oe opttoa ol and tools computational novel for need societal The ihicesn mut fdt n vial compu- available and data of amounts increasing With 1 oeneetoi tutr hoyadcompute and theory structure electronic Modern . Hansen, Grégoire Montavon, 4 1 ron edrhpCmuigFclt,AgneNtoa L National Argonne Facility, Computing Leadership Argonne ahn erigGop ehia nvriyo eln Fr Berlin, of University Technical Group, Learning Machine 5 htaesnhtclyacsil.Hr,w rsn machi a present we of Here, base exponen accessible. an synthetically in are dat that properties intelligent relevant for rel to screening structure-property amenable throughput predictive data and novel, of meaningful, amount unprecedented an n orltosbtenvrosmlclrpoete.Th artificia multi-task properties. deep molecular a various on between based correlations is ing model ML The ioni energies. eigenvalues, orbital frontier polarizability, ergy, utpeeetoi rud n xie-tt properties excited-state and ground- electronic multiple i.e. ayo uha“unu ahn”i iia,adsmtmssu sometimes cost. and computational similar, negligible is Machine” methods—at “Quantum a such of racy eateto hmsr,Phn nvriyo cec n T and Science of University Pohang Chemistry, of Department h obnto fmdr cetfi optn ihelectro with computing scientific modern of combination The .INTRODUCTION I. ula hre n atsa oriae falaos For atoms. all of coordinates Cartesian and charges nuclear 3 lxnr Tkatchenko, Alexandre binitio ab 2 2 nttt fPamcuia cecs T uih 03Zür 8093 Zurich, ETH Sciences, Pharmaceutical of Institute oefnaetlchallenge fundamental more A . 3 rt-ae-ntttdrMxPac-eelcat 1419 Max-Planck-Gesellschaft, der Fritz-Haber-Institut oe nvriy nmdn,Sogu-u eu 136-713, Seoul Seongbuk-gu, Anam-dong, University, Korea aclto eut o huad fogncmlcls th molecules, organic of thousands for results calculation 1 atisRupp, Matthias 6 eateto ri n ontv Engineering, Cognitive and Brain of Department ,5, 3, ∗ lu-oetMüller,Klaus-Robert Dtd a 1 2013) 31, May (Dated: 2 ieaadGobre, Vivekanand ainptnil lcrnant,adexcitation and affinity, electron potential, zation erlntok (NNs) networks neural h aevralsta loetrteeetoi Hamil- electronic the enter using also molecules tonian that query variables organic same for the inferred are (SE) tion nasrc rtpicpe iwo hmclcompound chemical on view principles first space strict a on ffcigcag rnfrrates transfer charge affecting machines vector n htaempe otecrepnigttlpoten- total corresponding the energy, to tial mapped are that and ides nrdcdb oeo sfrtemdln fmolec- of modeling the for energies us atomization of ular some by introduced prop- every for repeated chemicals. be of to class is and and erty QSPR, any as- challenging of and pect crucial a represents descriptors the priate surprisingly, Not interest. heuristic of property opti- electronic variables statistically descriptor and called create so between that relationships mized methods regression using tdrcl assse oeeg bsdo examples on (based energy, energy training), wavefunction its to for to given the system Hamiltonian maps finding system’s directly of the it instead maps i.e. which problem heuris- Ψ learning SE, any supervised by of the free encodes posed exactly is It model ML tics: this QSPRs mentioned omlyeatidcieeuvln oteddcieso- a deductive becomes the to model equivalent ML inductive the exact examples formally training converged of of limit number the in and problem supervised-learning citr ubd“olm”mti,i ietyobtained directly from is “Coulomb”-matrix, dubbed scriptor, emk s fa lentv Lapoc,recently approach, ML alternative an of use make We 9 tosis uhrltosiseal high- enable relationships Such ationships. h rprisicueaoiainen- atomization include properties The . riaiy hs plctosrl nassociation, on rely applications these Ordinarily, . elann M)mdl rie nadata a on trained model, (ML) learning ne 11 { nu sietclto identical is input e ilygoigpo fvrulcompounds virtual of pool growing tially Z pcfial,sltost Schrödinger’s equa- to solutions Specifically, . H nlnt 82,157Bri,Germany Berlin, 10587 28/29, anklinstr brtr,Agne lios649 USA 60439, Illinois Argonne, aboratory, i 3 ,6, 1, , dhoc ad ..ncercharges nuclear i.e. , erlntok xliigunderly- exploiting network, neural l R eir omdr quantum-chemical modern to perior, laoVazquez-Mayagoitia, Alvaro nlssfrteietfiainof identification the for analysis a cnlg,Phn 9–8,Korea 790–784, Pohang echnology, I † H ml rai oeue h accu- the molecules organic small i tutr hoycnla to lead can theory structure nic } n .AaoevnLilienfeld von Anatole O. and ssc hscntttsawl defined well a constitutes this such As . ( eln Germany Berlin, 5 { 5,6 Z dnicto n omtigo appro- of formatting and identification I rmlclrrognzto energies reorganization molecular or , , tsmlaeul predicts simultaneously at c,Switzerland ich, { R Z I 3,4 I ia opudSpace Compound mical } , Korea ) R ai-e ffcsuigsupport using effects basis-set , 10 7−→ binitio ab I Ψ } hsapoc sbased is approach This . 7,8 7−→ ML E rfrsldtrayox- ternary solid for or , Z . 11,12 I E methods, n positions and h mlydde- employed The . 4 nieteafore- the Unlike Katja 4, ‡ R I , 65 2 lution of SE. It is advantageous that the training data amide, amines, alcohol, epoxy, sulfide, ether, ester, chlo- can come from experiment just as well as from numeri- ride, aliphatic, and aromatic groups. For each of the cal evaluation of the corresponding quantum mechanical many possible stoichiometries, many constitutional iso- observable using approximate wave-functions (separated mers are considered, each being represented only by a nuclear and electronic wavefunctions, Slater determinant single conformational isomer. expansions etc.), Hamiltonians (such as Hückel or any Based on the string representation (SMILES17,18) of exchange-correlation potential), and self-consistent field molecules in the data base, we used the universal force- procedures. 19 10 field to generate reasonable Cartesian molecular geome- Building on our previously introduced work , we here tries, as implemented in OpenBabel20. The resulting ge- present a more mature ML model developed to accom- ometries were relaxed using the PBE approximation21 to plish the following two additional tasks, (i) simultane- Kohn-Sham DFT22 in converged numerical basis, as im- ously predict a variety of different electronic properties plemented in the FHI-aims code23 (tight settings/tier2 for a single query, and (ii) reach an accuracy comparable basis set). All geometries are provided in the supplemen- with the employed reference method used for generating tary material. the training set. The presented ML model is based on a multi-task deep artificial NN approach that captures correlations between seemingly related and unrelated properties and levels of theory. Remarkable predictive accuracy for “out-of-sample” molecules (i.e. molecules that B. Molecular representation (descriptor) were not part of the training set) has been obtained through the use of random Coulomb-matrices that intro- One of the most important aspects for creating a func- duce invariance with respect to atom indexing. For train- tional ML model is the choice of an appropriate data ing we generated a quantum chemical data base contain- representation (descriptor) that reflects important con- ing nearly 105 entries for over seven thousand stable or- straints and properties due to the underlying physics, SE ganic molecules, made of up to 7 atoms from main-group in our case. While there is a wide variety of descriptors elements, consisting of C, N, O, S, and Cl, saturated with used in chem- and bio-informatics applications24–28 they hydrogens to satisfy valence rules.13,14 For each molecule conventionally are based on prior knowledge about chem- atomization energy, static polarizabilities, frontier orbital ical binding, electronic configuration, or other quantum eigenvalues, and excitation energies and intensities have mechanical observables. Instead, we derive our repre- been calculated using a variety of widely used electronic sentation without any pre-conceived knowledge, i.e. ex- structure methods, including state-of-the-art first prin- clusively from stoichiometry and configurational infor- ciples methods, such as hybrid density-functional the- mation, from the generated according to the previous ory and the many-body GW approach (see Methods and subsection. As such, the molecular representation is in Ref.15). Fig. 1 illustrates the complete property data complete analogy to the electronic Hamiltonian used in base, and how it has been used for model training and ab initio methods. prediction. For this study we use a randomized variant of the recently introduced “Coulomb matrix”, M.10. The II. METHODS Coulomb matrix is an inverse atom-distance matrix representation that is unique (i.e. no two molecules will A. Molecular structures (input) have the same Coulomb matrix unless they are identical or enantiomers), and retains invariance with respect to molecular translation and rotation by construction. While the present ML model approach is generally applicable, for the purpose of this study we restrict our- 2.4 selves to the chemical space of small organic molecules. 0.5ZI for I = J, MIJ = (1) For all the cross-validated training and out-of-sample ZI ZJ for I = J. ( |RI −RJ | model performance testing, we rely on a controlled 6 test-bed of molecules, namely a subset of the GDB-13 data base13,14 consisting of all the 7211 small organic Off-diagonal elements encode the Coulomb repulsion be- molecules that have up to 7 second and third row atoms tween nuclear charges of atoms I and J, while diagonal consisting of C, N, O, S, or Cl, saturated with hydrogens. elements represent the stoichiometry through an expo- The entire GDB-13 data base represents an exhaustive nential fit in Z to the free atoms’ potential energy. We list of the 0.97B organic molecules that can be con- have enforced invariance with respect to atom indexing structed from∼ up to 13 such “heavy” atoms. All GDB by representing each molecule by a probability distribu- molecules are stable and synthetically accessible accord- tion over Coulomb matrices p(M) generated by different ing to organic chemistry rules16. Molecular features such atoms indexings of the same molecule. Details for pro- as functional groups or signatures include single, dou- ducing such random Coulomb matrices are given in the ble and triple bonds; (hetero-)cycles, carboxy, cyanide, supplementary material. 3

FIG. 1: Overview of calculated data base used for training and testing of the Machine Learning model. Quantum chemistry results for 14 properties of 7211 molecules are displayed. All properties and level theory, GW (G), PBE0 (P), and ZINDO (Z), are deﬁned in Section II C. Cartoons of 10 exemplary molecules from the data base are shown, they are used as input for quantum chemistry, for learning, or for prediction. Relying on input in “Coulomb” matrix form, the concept of a “Quantum Machine” is illustrated for two seemingly uncorrelated properties, atomization energy E and HOMO eigenvalue, which are decoded in terms of the two largest principal components (PCA1,PCA2) of the last neural network layer for 2k molecules, not part of training. The color-coding corresponds to the HOMO eigenvalues.

C. Molecular electronic properties (output) electron addition and removal processes35. The SCS, PBE0, and GW calculations have been carried out using FHI-aims23,36, ZINDO/s calculations are based on the The reference values necessary for learning and test- ORCA code37. ZINDO/s is an extension of the INDO/s ing consist of various electronic ground and excited-state semiempirical method with parameters to accurately re- properties of molecules in their PBE geometry minimum. produce single excitation spectra of organic compounds Specifically, we consider atomization energies E, static and complexes with rare earth elements. The INDO polarizabilities (trace of tensor) α, frontier orbital eigen- Hamiltonian neglects some two-center two-electrons in- values HOMO and LUMO, ionization potential IP, and tegrals in order to simplify the calculation of time- electron affinity EA. Furthermore, from optical spectrum dependent Hartree-Fock equations. While ZINDO re- simulations (10nm-700nm), we consider first excitation ∗ sults are usually not as accurate as highly correlated energy E st , excitation of maximal optimal absorption 1 methodologies the semiempirical Hamiltonian reproduces E∗ , and its corresponding intensity Imax. Data ranges max the most important features of the absorption spectra of of properties for the molecular structures and for various many small molecules and complexes, particularly char- levels of theory are given in footnote15, property mean- acterizing their most intense bands on the UV-Vis spec- values in the data set also feature in Table I. tra. All properties are provided in the supplementary To also gauge the impact of the reference method’s material. level of theory on the ML model, polarizabilities and fron- Similar conclusions hold for the selected levels of the- tier orbital eigenvalues were evaluated with more than ory: The employed methods can be considered to rep- one method. Static polarizability has been calculated resent a reasonable compromise between computational 29 using self-consistent screening (SCS) as well as hybrid cost and predictive accuracy. It should be mentioned, 30,31 density functional theory (PBE0) . PBE0 has also that ML methods can, in principle, be applied to any been used to calculate atomization energies and fron- method or level of approximation. tier orbital eigenvalues. Electron affinity, ionization potential, excitation energies, and maximal absorption intensity have been obtained from Zerner’s intermediate neglect of differential overlap (ZINDO)32–34. Hedin’s D. Training the model GW approximation35 has also been used to evaluate frontier orbital eigenvalues. GW is a quasi-particle ab Our model consists of a deep and multi-task neural initio many-body perturbation theory, known to accu- network38,39 that is trained on molecule-properties pairs. rately account for electronic excitations that describe It learns to map Coulomb matrices to all the 14 proper- 4 ties of the corresponding molecule simultaneously. NNs (i.e. NN based ML model) extracts and exploits hid- are well-established for learning functional relationships den correlations for these properties despite the fact that between input and output. They have successfully been they can not be recognized by visual inspection. Simi- applied to varying tasks such as object recognition40 and lar conclusions hold for atomization energy versus first speech recognition41. Given a sufficiently large NN, its excitation energy, or polarizability versus HOMO. universal approximation capabilities42, and the existence of the underlying noise-free Schrödinger equation, a NN solution can be expected to exist that satisfyingly relates molecules to their properties. Specifically, a deep NN will B. Accuracy vs. training set size properly unfold, layer after layer, a complex input into a simple representation of molecular properties. Finding It is an important feature of any ML model that the er- the true relationship unfolding among those that fit the ror can be controlled systematically as the training size is training data can be challenging because there is typically varied. We have investigated this dependence for our ML a manifold of solutions. The multi-task set up forces the model. Fig. 2 shows a typical decay of the ML model’s NN to predict multiple properties simultaneously. This mean absolute error (MAE) for predicting properties of is conceptually appealing because these additional con- 43 “out-of-sample” molecules as the number of molecules in straints narrow down the search for the “true model” training set increases logarithmically from 500 to 5000, as the set of models that fit all properties simultaneously its maximal value in the data base of 7211. For all in- is smaller. Details about the neural network training pro- vestigated properties, the improvement of error suggests cedure are provided in the supplementary material. that the MAE could still be lowered even further through addition of more molecules. However, since the reference method’s “precision” (i.e. estimated accuracy of the em- III. RESULTS AND DISCUSSION ployed level of theory) is reached for almost all properties already using 5000 examples, adding further examples Before reporting and discussing our results, we note does not make sense. For the atomization energy the de- the long history of statistical learning of the potential en- cay is particularly dramatic: A ten-fold increase in num- ergy hyper surface for molecular dynamics applications. ber of molecules (500 5000) reduces the error by 70%, It includes, for example, the modeling of potential ener- from 0.55 to 0.16 eV.→ But also for the HOMO/LUMO gies surfaces with artificial neural networks starting with eigenvalues, the error reduces substantially. We find that the work of Sumpter and Noid in 199244–49, or Gaus- the expected error decay law of 1/√N is only recov- sian processes50,51. Our work aims to move beyond sin- ered for the atomization energy, for∝ other properties the gle molecular systems and learn to generalize to unseen error decays more slowly. Fig. 2 also features the sta- compounds. This extension is not trivial, as the input tistical error bars for the MAEs—a measure of outliers. representation must deal with molecules of diverse sizes The error bar is only slightly larger than symbol size, and and compositions in the absence of one-to-one mapping hardly varies as the training set increases and the testing between atoms of different molecules. set decreases.

A. Database

Scatter plots among all properties for all molecules are shown in Fig. 1. Visual inspection confirms expected relationships between various properties: Koopman’s the- orem relating ionization potential to the HOMO eigen- value52, hard soft acid base principle linking polarizability to stability53, or electron affinity correlating with first excitation energy. Correlations of identical properties at different levels of theory reveal more subtle differences. Polarizabilities, calculated using PBE0, or with the more approximate self-consistent screening (SCS) model29, are strongly correlated. Also less well known relationships can be extracted from this data. One can obtain to a very decent degree, for example, the GW HOMO eigenval- FIG. 2: Error decay of ML model with increasing number ues by subtracting 1.5 eV from the corresponding PBE0 of molecules in training set (shown on a logarithmic scale). HOMO values. The MAE and its error bar is shown for atomization energy (E), polarizability (α), frontier orbital eigenvalues (HOMO, Some properties, such as atomization and HOMO en- ∗ LUMO), and first excitation energy (E st ). ergies, exhibit very little correlation in their scatter plot. 1 The inset of Fig. 1 illustrates how our Quantum Machine 5

C. Final ML model TABLE I: Mean absolute errors (MAE) and root mean square errors (RMSE) for out-of-sample predictions by ML model, After cross-validated training on the largest training together with typical error estimates of corresponding refer- set with randomly selected 5000 molecules, 2211 predic- ence level of theory. Errors are reported for all 14 molecu- tions have been made for the remaining “out-of-sample” lar properties, and are based on out-of-sample predictions for molecules, yielding at once all the 14 quantum chemi- 2211 molecules using a multi-task multi-layered NN ML model cal properties per molecule. The corresponding true ver- obtained by cross-validated training on 5000 molecules. The sus predicted scatter plots feature in Fig. 3. The corre- corresponding true versus predicted scatter plots feature in Fig. 3. Property labels refer to level of theory and molecular sponding mean absolute and root-mean square errors are property, i.e. atomization energy (Eref ), averaged molecular shown in Table I, together with literature estimates of polarizability (α), HOMO and LUMO eigenvalues, ionization errors typical for the corresponding level of theory. Er- potential (IP), electron affinity (EA), 1st excitation energy rors of all properties range in the single digit % of the ∗ ∗ (E1st ), excitation frequency of maximal absorption (Emax), mean property. Remarkably, when compared to pub- and corresponding maximal absorption intensity (Imax). To lished typical errors for the corresponding level of the- guide the reader, the mean value of the property across all ory, i.e. used as reference method for training, similar the 7211 molecules in the data base is shown in the second accuracy is obtained—the sole exception being the most column. Energies, polarizabilities, and intensity are in eV, 3 intense absorption and its associated excitation energy. A˚ , and arbitrary units, respectively. This, however, is not too surprising: Extracting the in- Property Mean MAE RMSE Reference MAE formation about a particular excitation energy and the E (PBE0) -67.79 0.16 0.36 0.15a, 0.23b, 0.09-0.22c α (PBE0) 11.11 0.11 0.18 0.05-0.27d, 0.04-0.14e associated absorption intensity requires sorting the en- d e tire optical spectrum—thus encoding significant knowl- α (SCS) 11.87 0.08 0.12 0.05-0.27 ,0.04-0.14 HOMO (GW) -9.09 0.16 0.22 - edge that was entirely absent from the information em- HOMO (PBE0) -7.01 0.15 0.21 2.08f ployed for training. For all other properties, however, our HOMO (ZINDO) -9.81 0.15 0.22 0.79h results suggest that the presented ML model makes “out- LUMO (GW) 0.78 0.13 0.21 - of-sample” predictions with an accuracy competitive to LUMO (PBE0) -0.52 0.12 0.20 1.30h the employed reference methods. These methods include LUMO (ZINDO) 1.05 0.11 0.18 0.93h some of the more costly state-of-the-art electronic struc- IP (ZINDO) 9.27 0.17 0.26 0.20g, 0.15d ture calculations, such as GW results for HOMO/LUMO EA (ZINDO) 0.55 0.11 0.18 0.16g ,0.11d eigenvalues and hybrid DFT calculations for atomization ∗ h i E1st (ZINDO) 5.58 0.13 0.31 0.18 , 0.21 ∗ energies and polarizabilities. Work is in progress to ex- Emax (ZINDO) 8.82 1.06 1.76 - tend our ML approach to other properties, such as the Imax (ZINDO) 0.33 0.07 0.12 - prediction of ionic forces or the full optical spectrum. We aPBE0, MAE of formation enthalpy for G3/99 set54,55 note, however, that for the purpose of this study any level bPBE0, MAE of atomization energy for 6 small molecules56,57 of theory and any set of geometries could have been used. cB3LYP, MAE of atomization energy from various studies52 dB3LYP, MAE from various studies52 The remarkable predictive power of the ML model can e 52 be rationalized by (i) the deep layered nature of the MP2, MAE from various studies fMAE from GW values NN model that permits to progressively extract the rel- gPBE0, MAE for G3/99 set54,55 evant problem subspace from the input representation hZINDO, MAE for set of 17 retinal analogs58 and gain predictive accuracy59,60; (ii) inclusion of ran- iTD-DFT(PBE0), MAE for set of 17 retinal analogs58 dom Coulomb matrices for training, effectively imposing invariance of property with respect to atoms indexing, clearly benefits the model’s accuracy: Additional tests they resemble the 2211 test molecules. For compounds suggest that using random, instead of sorted or diagonal- that bear no resemblance to the training set, the ML ized10 Coulomb matrices, also improves the accuracy of model must not be expected to yield accurate predic- Kernel Ridge Regression models to similar degrees; and tions. This limited transferability might one day become (iii) the multi-task nature of the NN accounts for strong moot through more intelligent choice and construction of and weak correlations between seemingly unrelated prop- molecular training sets tailored cover all of a pre-defined erties and different levels of theory. Aspects (i) and (iii) chemical compound space, i.e. all of the relevant geome- are also illustrated in Fig. 4. tries and elemental compositions, up to a certain number We reiterate that evaluation of all the 14 properties at of atoms. said level of accuracy for an out-of-sample molecule requires only milli-seconds using the ML model, as opposed to several CPU hours using the reference methods used IV. CONCLUSION for training. The down-side of such accuracy, of course, are the limits in transferability. All ML model predictions We have introduced a machine learning (ML) model are strictly limited to out-of-sample molecules that infor predicting electronic properties of molecules based on terpolate. More specifically, the 5000 training molecules training deep multi-task artificial NNs in chemical space. must resemble the query molecule in a similar fashion as Advantages of such a “Quantum Machine” (QM) (con- 6

FIG. 3: Scatter plot of true value versus ML model value for all properties. The red line indicates the identity mapping. All units correspond to the entries shown in Table I,

have demonstrated the QM’s performance for a rather small subset of chemical space, namely for small organic molecules with only up to seven atoms (not counting hydrogens) as defined by the GDB. Due to its inherent first principles setup we expect the overall approach to be equally applicable to molecules or materials of arbitrary size, configurations, and composition — without any ma- FIG. 4: Principal component analysis (PCA) on the multiple jor modification. We note, however, that in order to ap- layers of the deep neural network. Each point (molecule) is ply the QM to other regions in chemical space with simi- colored according to the rule: E and HOMO large → red ; E lar accuracy differing amounts of training data might be large and HOMO small → blue ; E small and HOMO large necessary. → green ; E and HOMO small → black. We can observe that We conclude that combining reliable data bases with the neural network extracts, layer after layer, a representation of the chemical space, that better captures the multiple ML promises to be an important step towards the gen- properties of the molecule. eral goal of exploring chemical compound space for the computational bottom up design of novel and improved compounds. ceptually speaking, as illustrated in Fig. 1) are the following, (a) multiple dimensions: A single QM execution simultaneously yields multiple properties at multiple lev- V. ACKNOWLEDGMENTS els of theory; (b) systematic reduction of error: By increasing the training set size the QM’s accuracy can be This research used resources of the Argonne Leader- converged to a degree that outperforms modern quantum ship Computing Facility at Argonne National Labora- chemistry methods, hybrid density-functional theory and tory, which is supported by the Office of Science of the the GW method in particular; (c) dramatic reduction in U.S. DOE under contract DE-AC02-06CH11357. K.- computational cost: The QM makes virtually instanta- R. M. acknowledges partial support by DFG (MU 987/4- neous property predictions; (d) user-friendly character: 2) and EU (PASCAL2). M. Rupp acknowledges support Training and use of the QM does not require knowledge by FP7 programme of the European Community (Marie about electronic structure, or even about the existence Curie IEF 273039). This work was also supported by the of the chemical bond. (e) arbitrary reference: The QM World Class University Program through the National can learn from data corresponding to any level of theory, Research Foundation of Korea funded by the Ministry of and even experimental results. The main limitation of Education, Science, and Technology, under Grant R31- the QM is the empirical nature inherent to any statisti- 10008. cal learning method used for inferring solutions, namely that meaningful predictions for new molecules can only be made if they fall in the regime of interpolation. We believe our results to be encouraging numeri- VI. SUPPLEMENTARY MATERIALS cal evidence that ML models can systematically infer highly predictive structure-property relationships from True properties and geometries for all com- high-quality data bases generated via first principles pounds. They can also be retrieved from atomistic simulations or experiments. In this study we www.quantum-machine.org. 7

Appendix A: Details on Random Coulomb Matrices tinuous quantities such as Coulomb repulsion energies encoded in the Coulomb matrix are best processed when Random Coulomb matrices define a probability dis- their information content is distributed across many di- tribution over the set of Coulomb matrices and account mensions of low information content. Such binary expan- for different atoms indexing of the same molecule. The sion can be obtained by applying the transformation following four-steps procedure randomly draws Coulomb M x θ x x + θ matrices from the distribution p( ): (i) Take an ar- φ(x)= ..., sigm − , sigm , sigm , ... bitrary valid Coulomb matrix M of the molecule (ii) θ θ θ Compute the norm of each row of this Coulomb ma- h i R ∞ trix: n = ( M1 ,..., M23 ), (iii) draw a zero-mean where φ : [0, 1] , the parameter θ controls the gran- → x x unit-variance|| noise|| vector|| ε of|| same size as n, and (iv) ularity of the transformation and sigm(x)= e /(1 + e ) permute the rows and columns of M with the same per- is a sigmoid function. Transforming Coulomb matrices mutation that sorts n + ε. An important feature of ran- M of size 23 23 with a granularity θ = 1 yields 3D × dom Coulomb matrices is that the probability distribu- tensors of size [ 23 23] of quasi-binary values, ap- ∞× × tions over Coulomb matrices of two different molecules proximately 2000 dimensions of which are non-constant. are completely disjoint. This implies that the random- Transforming vectors P of 14 properties with a granular- ized representation is not introducing any noise into the ity 0.25 of same units as in Table I yields matrices of size prediction problem. Invariance to atoms indexing proves [ 14], approximately 1000 components of which are ∞× to be crucial for obtaining models with high predictive ac- non-constant. curacy. The idea of encoding known invariances through We construct a four layer neural network with 2000, such data extension has previously been used to improve 800, 800 and 1000 nodes at each layer. The network im- −1 prediction accuracy on image classification and handwrit- plements the function φ f3 f2 f1 φ(M) where 61 ◦ ◦ ◦ ◦ ten digit recognition data sets . functions f1, f2 and f3 between each layer correspond to a linear transformation learned from data followed by a sigmoid nonlinearity. The neural network is trained to Appendix B: Details on Training the Neural minimize the mean absolute error of each property using Network the stochastic gradient descent algorithm (SGD62). Er- rors are back-propagated63 from the top layer back to the inputs in order to update all parameters of the model. We run 250,000 iterations of the SGD and present at each iteration 25 training samples. During training, each molecule-property pair is presented in total 1250 times to the neural network, but each time with a different atoms indexing. A moving average of the model parameters is maintained throughout training in order to attenuate the noise of the stochastic learning algorithm64. The moving FIG. 5: Predicting properties for a new molecule: (a) En- average is set to remember the last 10% of the train- ter Cartesian coordinates and nuclear charges, (b) form a ing history and is used for prediction of out-of-sample Coulomb matrix, (c) binarize representation, (d) propagate molecules. Once the neural network has been trained, into trained neural network, (e) scale outputs back to prop- the typical CPU time for predicting all 14 properties of a erty units. new out-of-sample molecule is 100 milli seconds. Train- ing the neural network on a CPU∼ takes 24 hours. Pre- The ML model and the neural network perform a se- diction of an out-of-sample molecule is obtained∼ by prop- quence of transformation to the input that are illustrated agating 10 different realizations of p(M) and averaging in Fig. 5. The Coulomb matrix is first converted to a bi- outputs. Prediction of multiple molecules can be easily nary representation before being processed by the neural parallelized by replicating the trained neural network on network. The rationale for this binarization is that con- multiple machines.

∗ Electronic address: [email protected] Oﬃce of Naval Research) recently published their support: † Electronic address: [email protected] www.whitehouse.gov/blog/2011/10/26/four-new-federal ‡ Electronic address: [email protected] -programs-support-materials-genome-initiative. 1 Based on the Materials Genome Initiative 2 J. Hachmann et al. The harvard clean energy project: www.whitehouse.gov/mgi announced by US Presi- Large-scale computational screening and design of organic dent Obama in June 2011, four federal science and photovoltaics on the world community grid. J. Phys. research agencies (National Science Foundation, De- Chem. Lett., 2:2241–2251, 2011. partment of Energy, Air Force Research Laborator, and 3 L. Hu, X. Wang, L. Wong, and G. Chen. Combined ﬁrst- 8

principles calculation and neural-network correction ap- 2009. proach for heat of formation. J. Chem. Phys., 119:11501– 17 David Weininger. SMILES, a chemical language and in- 11507, 2003. formation system. 1. Introduction to methodology and en- 4 X. Zheng, L. Hu, X. Wang, and G. Chen. A generalized coding rules. J. Chem. Inform. Comput. Sci., 28(1):31–36, exchange-correlation functional: the Neural-Networks ap- 1988. proach. Chem. Phys. Lett., 390:186–192, 2004. 18 David Weininger, Arthur Weininger, and Joseph 5 Roman M. Balabin and Ekaterina I. Lomakina. Neural Weininger. SMILES. 2. Algorithm for generation of network approach to quantum-chemistry data: Accurate unique SMILES notation. J. Chem. Inf. Model., 29(2):97– prediction of density functional theory energies. The Jour- 101, 1989. nal of Chemical Physics, 131(7):074104, 2009. 19 A. K. Rappé, C. J. Casewit, K. S. Colwell, W. A. Goddard 6 R. M. Balabin and E. I. Lomakina. Support vector machine III, and W. M. Skid. Uff, a full periodic table force field for regression (ls-svm)—an alternative to artifical neural net- molecular mechanics and molecular dynamics simulations. works (anns) for the analysis of quantum chemistry data. J. Am. Chem. Soc., 114:10024–10035, 1992. Phys. Chem. Chem. Phys., 13:11710–11718, 2011. 20 R. Guha, M. T. Howard, G. R. Hutchison, P. Murray-Rust, 7 G. R. Hutchison, M. A. Ratner, and T. J. Marks. Hopping H. Rzepa, C. Steinbeck, J. K. Wegner, and E. Willighagen. transport in conductive heterocyclic oligomers: Reorgani- The blue obelisk – interoperability in chemical informatics. zation energies and substituent effects. J. Am. Chem. Soc., J. Chem. Inf. Model., 46:991–998, 2006. 127:2339–2350, 2005. 21 J. P. Perdew, K. Burke, and M. Ernzerhof. Generalized 8 M. Misra, D. Andrienko, B. Baumeier, J.-L. Faulon, and gradient approximation made simple. Phys. Rev. Lett., O. A. von Lilienfeld. Toward quantitative structure- 77:3865–3868, 1996. property relationships for charge transfer rates of poly- 22 W. Kohn and L. J. Sham. Self-consistent equations in- cyclic aromatic hydrocarbons. J. Chem. Theory Comput., cluding exchange and correlation effects. Phys. Rev., 7:2549–2555, 2011. 140:A1133, 1965. 9 G. Hautier, C. C. Fischer, A. Jain, T. Mueller, and 23 V. Blum, R. Gehrke, F. Hanke, P. Havu, V. Havu, X. Ren, G. Ceder. Finding nature’s missing ternary oxide com- K. Reuter, and M. Scheffler. Ab initio molecular simula- pounds using machine learning and density functional the- tions with numeric atom-centered orbitals. Comput. Phys. ory. Chem. Mater., 22:3762–3767, 2010. Comm., 180:2175–2196, 2009. 10 M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. von 24 G. Schneider. Virtual screening: an endless staircase? Na- Lilienfeld. Fast and accurate modeling of molecular atom- ture Reviews, 9:273–276, 2010. ization energies with machine learning. Phys. Rev. Lett., 25 J.-L. Faulon, D. P. Visco, Jr., and R. S. Pophale. The sig- 108:058301–058306, 2012. nature molecular descriptor. 1. Using extended valence se- 11 O. Anatole von Lilienfeld. First principles view on chemi- quences in QSAR and QSPR studies. J. Chem. Inf. Comp. cal compound space: Gaining rigorous atomistic control of Sci., 43:707–720, 2003. molecular properties. Int. J. Quantum Chem., 2013. DOI: 26 O. Ivanciuc. QSAR Comparative Study of Wiener Descrip- 10.1002/qua.24375. tors for Weighted Molecular Graphs. J. Chem. Inf. Comp. 12 P. Hohenberg and W. Kohn. Inhomogeneous electron gas. Sci., 40:1412–1422, 2000. Phys. Rev., 136:B864, 1964. 27 R. Todeschini and V. Consonni. Handbook of Molecular 13 T. Fink, H. Bruggesser, and J.-L. Reymond. Virtual explo- Descriptors. Wiley-VCH, Weinheim, 2009. ration of the small-molecule chemical universe below 160 28 J. Braun, A. Kerber, M. Meringer, and C. Rücker. Simi- Daltons. Angew. Chem. Int. Ed., 44:1504–1508, 2005. larity of molecular descriptors: The equivalence of Zagreb 14 T. Fink and J.-L. Reymond. Virtual exploration of the indices and walk counts. MATCH, 54:163–176, 2005. chemical universe up to 11 atoms of C, N, O, F: Assem- 29 A. Tkatchenko, R. A. DiStasio Jr., R. Car, and M. Schef- bly of 26.4 million structures (110.9 million stereoisomers) fler. Accurate and efficient method for many-body van der and analysis for new ring systems, stereochemistry, physic- waals interactions. Phys. Rev. Lett., 108:236402–236406, ochemical properties, compound classes, and drug discov- 2012. ery. J. Chem. Inf. Model., 47:342–353, 2007. 30 J. P. Perdew, M. Ernzerhof, and K. Burke. Rationale for 15 Electronic properties considered include PBE0 atomiza- mixing exact exchange with density functional approximation energies ranging from -800 to -2000 kcal/mol; first tions. J. Chem. Phys., 105:9982–9985, 1996. excitation energies (1.52 - 36.77 eV), as well as maximal 31 M. Ernzerhof and G. E. Scuseria. Assessment of the absorption intensities (oscillator strengths (hj|r|0i) rang- Perdew-Burke-Ernzerhof exchange-correlation functional. ing from 0.05 to 3.35 arbitrary units) and corresponding J. Chem. Phys., 110:5029–5037, 1999. excitation energies (3.37 - 39.69 eV) at the ZINDO level 32 J. Ridley and M. C. Zerner. An intermediate neglect of dif- of theory; HOMO and LUMO values calculated at the ferential overlap technique for spectroscopy: Pyrrole and ZINDO/s, PBE0, and GW level of theory (HOMOPBE0: the azines. Theor. Chim. Acta, 32:111–134, 1973. 33 -10.95 - -5.12; HOMOGW: -14.13 - -6.98 eV; LUMOPBE0: A. D. Bacon and M. C. Zerner. An intermediate neglect of -3.81 - 0.41 eV; LUMOGW: -1.84 - 1.96 eV) (the corre- differential overlap theory for transition metal complexes: sponding gap ranging from 6.9 to 20.2 eV and 7.3 to 15.2 Fe, Co and Cu chlorides. Theor. Chim. Acta, 53:21–54, eV, respectively); PBE0 and SCS29 molecular polarizabil- 1979. ities (2.5 - 10 A˚3); electron affinity (-3.99 to 2.91 eV) and 34 M. Zerner. Semiempirical Molecular Orbital Methods. VCH ionization potentials (6.93 to 15.73 eV) at the ZINDO/s New York, 1991. Reviews in Computational Chemistry, level of theory. Volume 2, Editors K. B. Lipkowitz and D. B. Boyd. 16 L. C. Blum and J.-L. Reymond. 970 million druglike small 35 L. Hedin. New method for calculating the one-particle molecules for virtual screening in the chemical universe green’s function with application to the electron-gas prob- database GDB-13. J. Am. Chem. Soc., 131:8732–8733, lem. Phys. Rev., 139:A796–A823, 1965. 9

36 X. Ren, P. Rinke, V. Blum, J. Wieferink, A. Tkatchenko, Lett., 104(13):136403–136406, 2010. A. Sanfilippo, K. Reuter, and M. Scheffler. Resolution- 51 M. J. L. Mills and P. L. A. Popelier. Intramolecular po- of-identity approach to HartreeFock, hybrid density func- larisable multipolar electrostatics from the machine learn- tionals, RPA, MP2 and GW with numeric atom-centered ing method kriging. Computational and Theoretical Chem- orbital basis functions. New J. Phys., 14:053020–053074, istry, 975(13):42 – 51, 2011. 2012. 52 W. Koch and M. C. Holthausen. A Chemist’s Guide to 37 F. Neese. ORCA 2.8, An ab initio, density functional and Density Functional Theory. Wiley-VCH, 2002. semiempirical program package, University of Bonn, Ger- 53 Ralph G. Pearson. Hard and Soft Acids and Bases. J. Am. many. 2006. Chem. Soc., 85:3533–3539, 1963. 38 Rich Caruana. Multitask learning. Mach. Learn., 28(1):41– 54 V. N. Staroverov, G. E. Scuseria, J. Tao, and J. P. Perdew. 75, 1997. Comparative assessment of a new nonempirical density 39 Yoshua Bengio and Yann LeCun. Scaling learning algo- functional: Molecules and hydrogen-bonded complexes. J. rithms towards AI. In Léon Bottou, Olivier Chapelle, Chem. Phys., 119(23):12129–12137, 2003. D. DeCoste, and J. Weston, editors, Large Scale Kernel 55 L. A. Curtiss, K. Raghavachari, G. W. Trucks, and J. A. Machines. MIT Press, 2007. Pople. Assessment of gaussian-3 and density functional 40 Y LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- theories for a larger experimental test set. J. Chem. Phys., based learning applied to document recognition. Proceed- 112:7374–7383, 2000. ings of the IEEE, 86(11):2278–2324, 1998. 56 Y. Zhao, J. Pu, B. J. Lynch, and D. G. Truhlar. Tests of 41 Alex Waibel, T. Hanazawa, Geoffrey E. Hinton, second-generation and third-generation density functionals K. Shikano, and K. Lang. Phoneme recognition using time- for thermochemical kinetics. Phys. Chem. Chem. Phys., delay neural networks. IEEE Transactions on Acoustics, 6:673–676, 2004. Speech, and Signal Processing, 37:328–339, 1989. 57 B. J. Lynch and D. G. Truhlar. Small representative bench- 42 G. Cybenko. Approximation by superpositions of a sig- marks for thermochemical calculations. J. Phys. Chem. A, moidal function. Mathematics of Control, Signals, and 107:8996–8999, 2003. Systems (MCSS), 2(4):303–314, 1989. 58 C. S. López, O. N. Faza, S. L. Estévez, and A. R. de Lera. 43 Jonathan Baxter. A model of inductive bias learning. J. Computation of vertical excitation energies of retinal and Artif. Int. Res., 12(1):149–198, 2000. analogs: Scope and limitations. J. Comp. Chem., 27:15– 44 Bobby G. Sumpter and Donald W. Noid. Potential energy 123, 2006. surfaces for macromolecules. a neural network technique. 59 M. L. Braun, J. M. Buhmann, and K. R. Müller. On rele- Chemical Physics Letters, 192(56):455 – 462, 1992. vant dimensions in kernel feature spaces. J. Mach. Learn. 45 S. Lorenz, A. Gross, and M. Scheffler. Representing high- Res., 9:1875–1906, 2008. dimensional potential-energy surfaces for reactions at sur- 60 G. Montavon, M. L. Braun, and K. R. Müller. Kernel faces by neural networks. Chem. Phys. Lett., 395:210–215, analysis of deep networks. J. Mach. Learn. Res., 12:2563– 2004. 2581, 2011. 46 J. Behler and M. Parrinello. Generalized neural-network 61 Dan Claudiu Ciresan, Ueli Meier, Luca Maria Gam- representation of high-dimensional potential-energy sur- bardella, and Jürgen Schmidhuber. Deep, big, simple neu- faces. Phys. Rev. Lett., 98:146401–146404, 2007. ral nets for handwritten digit recognition. Neural Compu- 47 C. M. Handley and P. L. A. Popelier. Dynamically polar- tation, 22(12):3207–3220, 2010. izable water potential based on multipole moments trained 62 Léon Bottou. Stochastic gradient learning in neural net- by machine learning. J. Chem. Theory Comput., 5:1474, works. In Proceedings of Neuro-Nˆımes 91, Nimes, France, 2009. 1991. EC2. 48 C. M. Handley and P. L. A. Popelier. Potential energy 63 David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. surfaces fitted by artificial neural networks. The Journal Williams. Learning representations by back-propagating of Physical Chemistry A, 114(10):3371–3383, 2010. errors. Nature, 323(6088):533–536, 1986. 49 J. Behler. Atom-centered symmetry functions for con- 64 B. T. Polyak and A. B. Juditsky. Acceleration of stochas- structing high-dimensional neural networks potentials. J. tic approximation by averaging. SIAM J. Control Optim., Chem. Phys., 134:074106, 2011. 30(4):838–855, 1992. 50 Albert P. Bartók, Mike C. Payne, Risi Kondor, and Gábor 65 The number of electrons is implicitly encoded by imposing Csányi. Gaussian approximation potentials: The accuracy charge neutrality. of quantum mechanics, without the electrons. Phys. Rev.