Advances in Deep via Simulators & Deep Learning

James Bird1,∗, Linda Petzold1, Philip Lubin2, Julia Deacon3

Abstract The NASA Starlight and Breakthrough Starshot programs conceptualizes fast via small relativistic that are propelled by directed energy. This process is radically different from traditional space travel and trades large and slow spacecraft for small, fast, inexpensive, and fragile ones. The main goal of these wafer is to gather useful images during their deep space journey. We introduce and solve some of the main problems that accompany this concept. First, we need an object detection system that can detect that we have never seen before, some containing features that we may not even know exist in the . Second, once we have images of , we need a way to take these images and rank them by importance. Equipment fails and data rates are slow, thus we need a method to ensure that the most important images to humankind are the ones that are prioritized for data transfer. Finally, the energy on board is minimal and must be conserved and used sparingly. No images should be missed, but using energy erroneously would be detrimental. We introduce simulator-based methods that leverage artificial intelligence, mostly in the form of computer vision, in order to solve all three of these issues. Our results confirm that simulators provide an extremely rich training environment that surpasses that of real images, and can be used to train models on features that have yet to be observed by humans. We also show that the immersive and adaptable environment provided by the simulator, combined with deep learning, lets us navigate and save energy in an otherwise implausible way. Keywords: computer vision, simulator, deep learning, space, universe, exoplanet, object detection, novelty detection

1. Introduction

∗ arXiv:2002.04051v2 [astro-ph.IM] 6 Jun 2020 Corresponding author Space travel, up until recently, was Email addresses: [email protected] constrained by chemical propulsion, large (James Bird), [email protected] (Linda Petzold), [email protected] (Philip Lubin), of California Santa Barbara, Santa Barbara, CA [email protected] (Julia Deacon) 93106-9530 1Department of Computer Science, University of 3Department of Computing and Mathematical California Santa Barbara, 2104 Harold Frank Hall, Sciences, California Institute of Technology, 1200 E. Santa Barbara, CA 93106-9530 California Blvd., MC 305-16, Pasadena, California 2Department of Physics, Broida Hall, University 91125-2100 Preprint submitted to Advances in June 9, 2020 spacecraft, and therefore, relatively slow 1.1. Previous Work speeds. Since the main objective has been The effectiveness of machine learning, exploration of our , these meth- specifically deep learning via TensorFlow ods were sufficient. In contrast, the recent and cuDNN, has been indisputably demon- Starlight program (Kulkarni et al., 2017) strated in the last decade (Abadi et al. has introduced methods for deep space (2016), Chetlur et al. (2014), Canziani travel that utilize small discs, which travel et al. (2016)). The fight over the best at approximately one-fourth of the speed of model and the most accurate results, espe- light via directed energy. cially between the most popular models like ResNets, DenseNets (Huang et al. (2018)), Alongside the prospect of fast deep Inception(Szegedy et al. (2015)), Masks space travel comes many new challenges. (He et al. (2018)), and models that com- The normal model for space travel in- bine some of these together(Szegedy et al. cludes spacecraft capable of housing instru- (2016)), is one that has produced a plethora ments, propulsion and navigational equip- of potent options to choose from. Models ment, telescopes, energy banks, and much that are more accurate than human beings more. Since the Starlight program will be at doing extremely difficult tasks are still utilizing small wafersats that are approxi- being discovered (Rajpurkar et al. (2017)). mately the size of a coffee can lid, all of The areas of deep learning and astron- these features need to be reworked or dis- omy have come together in recent years carded. (Ruffio et al. (2018), Morad et al. (2018), Schaefer et al. (2018), Pearson et al. Besides physical constraints, this new (2017)), mostly in the form of light curves model of space travel introduces feasibility (Shallue & Vanderburg (2017), Zucker et constraints as well. The of interest is al. (2018), Carrasco-Davis et al. (2018)). beyond four light-years away, meaning that The results and general concepts promote transmission of data and response command a healthy symbiosis between deep learn- transmissions are a combined eight years or ing and the problems that arise in astron- more. Thus, the wafersats need to be able omy. Yet, the processes are carried out from to make decisions without human interven- , not space, and do not address real tion, and for that, artificial intelligence (AI) images, two big issues that create a gap in is paramount. comparability. Outside of , simulators have The major hurdles that we will discuss been used to train data in specific instances are those concerning computer vision via where the benefits outweigh the drawbacks. planetary detection, data and storage block- Smyth et al. (2018) outlines some major ages via novelty detection and ranking, and drawbacks, namely that the process takes a energy management via combining simula- lot of time and knowledge, as well as a note tor features with subtraction-algorithm-fed that simulator-based training may not gen- computer vision. For all of these issues, tak- eralize well to real images. Alongside those ing advantage of a universe simulator will concerns, McDuff et al. (2018) begins with a introduce solutions that were otherwise in- common issue in machine learning models, effective or impossible to find. which is that training data sets are often 2 biased. This bias arises when there are mi- decomposed into red, green, and blue chan- norities in the training set, which in turn nels and vectors were constructed of length produces poor results when the model is 320 × 180 × 3 = 172, 800. asked to evaluate a similar entity in the pop- Our challenge is to label each image as ulation. These issues are handled through- novel or regular. That is, we wish to gener- out this paper and are shown to not be an ate a classification n s.t. for each input x, issue with the specific problem at hand. n(x) ∈ {0, 1}, where a 0 indicates regular- Simulators also introduce a lot of ben- ity and a 1 indicates novelty. The algorithm efits. One large one, also seen in Con- should hopefully yield a large number of 1’s nor & Leeuwen (2018), is that ”the small when a or the is clearly in view catalogue of real events is probably not and should yield very few 1’s when the im- yet a representative sample of the under- age is mostly distant . lying .. population, nor is it big enough to build a meaningful training set for machine 1.2.2. GWR Algorithm learning, deep or otherwise.” An important Define A as the set of nodes in our net- theme throughout this paper, and an ex- work and C as the set of edges between tremely useful aspect of simulators, is that these nodes. We denote our inputs as ξ and they provide an untold amount of training the weight vector for any node n as wn. data, assuming that one can create realistic Each node n has a habituation hn which simulations. represents how familiar that node is to the system. 1.2. Unsupervised Learning for Planetary Detection 1. Initialize two nodes that represent two random samples from the dataset. We The intuition behind object detection, in set their habituations each to 1. The particular planetary detection, might point set of edges between nodes begins toward an unsupervised learning technique. empty. After all, one might reasonably think that detecting a nearby planet after months of 2. Iterate through the dataset. For each traveling through deep space would be easy. data sample ξ: We test this idea using an unsupervised (a) For each node i in the net- technique called a Grow When Required work, calculate its distance from (GWR) Network (Marsland et al., 2002). ξ, which is ||ξ − wi||. (b) Find the node s with the smallest 1.2.1. GWR Setup distance and the node t with the Using the worldwide telescope, we gen- second-smallest distance. erated a 9,000 frame series of solar system (c) Add an edge between s and t if it images. It begins with , then it ex- does not already exist. plores , the Sun, and finally . (d) Calculate the activity P 2 The majority of images contain only back- a = exp(− j(ξ[j] − ws[j]) /C), ground stars. where C = 29,203,200 was The images were down-scaled to a chosen to prevent an integer 320x180 resolution in order to improve com- overflow. There are 172,800 fields putational speed. For learning, they were in each data vector, and since 3 the average of the quantities in each vector is close to 13, and 132 = 169, we divide by 172, 800 × 169 = 29, 203, 200.

(e) If a < aT and s’s habituation hs < hT (where aT is some inser- tion threshold and hT is some ha- bituation threshold), then add a ws+ξ new node r. Set wr = 2 . Insert edges between s and r and r and t and remove the edge between s and t. (f) Otherwise, update the weight and Figure 1: A scatter plot of the detected novelty of habituation of s as follows: δws = the data. b × (ξ − ws) and δhs = τb × 1.05 × (1 − hs) − τb, where b and τb are parameters. Next, update bright stars. No novelty is detected dur- the weight and habituation of s’s ing the long period of only stars. Next we neighbors i as follows: δwi = n × see increased novelty detection when Mer- (ξ −wi) and δhi = τn ×1.05×(1− cury is plainly in view, and then when the hi)−τn, where n and τn are other sun appears, and finally when we zoom into parameters. Mars. (g) Remove any nodes without any neighbors.

Our chosen values are: aT = 0.7, hT = 0.1, τb = 0.3, τn = 0.1, b = 0.1, and n = 0.01.

1.2.3. Results Figure 1 is a scatter plot that was gener- ated to visualize the novelty detected from the data. The x-axis is the id of each pic- ture, and the y-axis is the number of novel images that were detected in each bin of 100 images. Figure 2 is a continuous represen- tation of the same concept. Figure 2: A binned scatter plot of the novelty of the Moving along the Image ID axis, we data. Image ranges that are salient to the human see that novelty was detected in clumps eye are labeled on the plot. around 0-500, 900, 4000-4500, 6000-6500, 7000-7500, 7700-8000, and 8200-8500. We We notice that Neptune’s collection of observed that novelty was detected first on novelty is roughly one quarter the size of the Neptune, then again on some particularly other three celestial objects that come into 4 view. We also notice a huge spike around walk the streets of New York and see if our image 8300. This is very interesting because algorithm can identify human beings. In there are no large celestial objects in view this setting, identifying a human being is a at this time. success, and not identifying a car or stop sign as a human being would also be a suc- 1.2.4. GWR Discussion cess. Yet, identifying anything non-human A deep space exploration mission would as a human being would be a failure. The come with many challenging objectives. A accuracy of a model, which is mathemat- small but connected subset of those would ically computed per identification, can be involve detecting objects, deciding whether used as a measure of how sure the algorithm they are important, extracting key features is that the object being identified is the cor- that we would want to study or observe, and rect type. In this paper, we will delve into prioritizing their information retrieval. why this is difficult for our specific scenario, GWR wouldn’t be able to decide impor- and we will test whether this can benefit tance, extract features, or prioritize infor- severely from the use of simulators. mation retrieval, yet if it could detect novel On the other hand, novelty detection is objects in deep space, this would be useful. used to attempt to identify something that We can see from Figure 1 and Figure 2 that has never been seen before. One powerful the detection is inconsistent and unreliable. example is self-driving cars being able to Neptune is almost completely missed and see traffic signs that are unique to a certain the three smaller peaks at the Sun, as seen country, and therefore have never been seen in Figure 2, are larger than Neptune. The or used during the training process (Kim et largest peak of all happens while Mars is al., 2017). In this example, the self-driving minuscule and essentially not in view. car algorithm has never seen this specific Although GWR had high novelty de- sign before, and so identifying it without tection peaks while passing by Mercury any training data is very difficult. In our and the Sun, it failed to correctly activate paper, unseen planetary features are analo- at Mars or Neptune. These observations, gous to the unseen traffic sign in the exam- paired with its inability to do anything fur- ple, and we delve into methods of solving ther with the data, introduces a need for a this via simulators. more advanced model that can achieve all of the above objectives. 2. The Simulator

1.3. Object Detection vs. Novelty Detection Although there are quite a few uni- verse simulators available today. Here, we Throughout this paper, our main goals utilized SpaceEngine (SpaceEngine.org) for will constantly be alluding to object detec- its realism, expansive set of options and tion and novelty detection. In a general customizations, and unique informational computer science setting, object detection tools. is used to identify something in an image that has already been trained via some algo- 2.1. Simulator Features rithm. For example, we may feed thousands One of the best features of the simula- of images of human beings into a YOLO al- tor is its extremely realistic rendering capa- gorithm, and then once it is trained, we can bilities. In combination with a 3840x2160 5 4K monitor and GTX 1080 Ti, the simula- and lack lens flares. Second, a feature called tor produces extremely detailed and realis- overbright can drastically adjust how bright tic images. the background stars and nebula appear. Training on images that embrace the en- tire spectrum of overbright will allow this machine to deal with novelty detection in a very advanced manner. Having a plethora of options to enable and tweak can intro- duce a much larger set of training images and will let the AI absorb more information before embarking into deep space.

Figure 3: Examples of 3D-rendered randomly gen- erated exoplanets

The simulator also includes the ability to edit any planet, so that instantly ren- dering an exoplanet with a very particular feature set is simple. Alongside image fea- tures are astronomical features, which are tracked and shown for every body in the universe. Some of these features are type, class, orbital period and mass, but most im- portantly, distance.

2.2. Simulator Options Graphical options in the simulator are abundant, which allow for complete con- Figure 4: Eight lens options applied to the same trol of the simulated universe. In general, star the feature set should be optimally set for realism while traveling through space, but Some other important options, besides the ability to tweak these options speaks those that deal with graphics and render- to greater breadth for learning and adapt- ing, are diffraction spike intricacy and size, ing to unique situations that may arise in lens effect on stars, and planetary shine. Al- space. For example, the image of an exo- tering all of these settings and training on planet while traveling at one-fourth of the the resulting images enables the capture of speed of light with a nebula in the back- more information. ground is a completely new concept. Yet, two features in the simulator may be able 2.3. Overall Simulator Importance to deal with that combination. First, the In this paper, We consider three main ar- ability to toggle lens flares will provide the eas of deep space travel that can be drasti- AI with training images that both contain cally improved with the use of a simulator. 6 First, computer vision is an extremely elty ranking. A major hurdle in deep space useful tool for detecting objects and mak- wafersat travel is data storage and transmis- ing decisions based on what is seen. The sion. On-board memory is limited by phys- training process consists of tagging images ical constraints of the wafersat and astro- and providing a label for each tag, feeding physical exposure, while transmitting data those images and tags into a model, and from a wafersat to a communications hub having that model learn the associations. would be slow and dependent on energy re- The model can then be given images, and serves. based on how successfully it was trained, it A system that can deal with this issue may be able to identify parts of the image. is one that prioritizes the most important Since we have never photographed exo- on-board data and sends that first. This planets in detail, training a model using real not only ensures that the critical images are images is not feasible. Therefore, we rely sent in descending order of importance in on training using images of planets that we case of some malfunction, but that the most have photographed, which would be those relevant data is quickly known for the next in our own solar system. Yet, detailed pho- wafersats in line. tographs of planets are not very abundant With the overarching goal being the iden- and would only teach the model to look for tification and transmission of the most im- those specific features. In realizing that this portant data, novelty ranking will quantify would not be sufficient, we may move to- the on-board images based on importance. ward novelty detection, a branch of com- Simulators will provide the breadth of plan- puter vision that tries to classify data that etary features that are needed to find out deviates from the data used during training. what importance means, as perceived by hu- Co-domain embedding (Kim et al., 2017) mans, and then this information can be ap- has proven useful in some situations, such as plied to software. those where a template design would resem- Third and last, sending a small disc into ble a real image almost exactly, but plane- deep space means that on-board energy re- tary features do not translate well to use in serves will be very small. Yet, the objective novelty detection. This is because planetary of detecting and imaging astronomical bod- features, such as atmospheric patterns, are ies while traveling must still be met. extremely unique. Simulators can provide very detailed and 3. Simulator for Planetary Detection randomly generated images of planets that obey universal physical laws. Therefore, we Our main objective here is to identify will be able to generate countless images of novel planets while traveling through deep planets that resemble real images of possi- space. In order to do so, and for subse- ble exoplanets. Training on these images quent sections, we will require a basic con- and features, the model will learn an exor- ceptual understanding of object detection bitant amount of information. While travel- in order to logically progress. We should ing through space and faced with an image point out that the main backbone of object of a real exoplanet, the model will now have detection, through a few core processes, is a much broader knowledge base. the same as that used by humans when they Second, we introduce the notion of nov- naturally process information and identify 7 objects. We will discuss these fundamental planetary detection and planetary features. core processes. These two processes, the detection of plan- First, the object that we will try to have ets while traveling through space, as well the machine identify should be seen before- as the detection and recognition of features hand in order to train the model. Unsu- on those planets, are the inspiration for the pervised techniques have their uses and do two main experiments that are set up. Cur- not require this training process, but we rently, our collection of useful astronomi- will only deal with supervised learning mod- cal images is very limited. Therefore, us- els from here on out. Mainly, this is done ing only real images of planets would limit because planetary detection is the simplest us to those found in our solar system. Also, task, so we need a model that can adapt planetary features would suffer since our so- afterwards in order to successfully identify lar system contains very few features out of planetary features and rank novelty. the set of total planetary feature combina- Second, the machine will learn these tions. trained models better with more images. Of The first experiment will test the validity course there are exceptions here, such as of simulator images in general. It is set up in feeding poor images or images that do not three different stages using the same object match the objects category. We will test detection framework and always testing on this concept thoroughly while we also test the set of real images of . First, we the importance of simulated images. will train on real images of every planet in our solar system except for Jupiter. These 3.1. Setup images will be collected from NASA image Here we will discuss the details of our repositories and will not include composite model, our hypothesis, and how we will go images, artist renditions, or any other varia- about testing the importance of simulator tions except for true unaltered images. Sec- images. Our main goals when choosing a ond, we will train on only simulator images model are finding one that has high accu- with the goal of solidifying whether simula- racy, low to medium computation time, and tor images alone are useful in detecting real has been tested to be a reliable model. Be- planets. Third, we will combine the first cause of this, no new models that haven’t and second training sets, comprised of sim- had time to be tested thoroughly through- ulator images and all real images (exclud- out the computer vision community will be ing Jupiter),to determine whether simulator used. Also, the ideal model will sacrifice images and real images together provide the computation time for accuracy, if needed. best of both worlds. Since we will be testing different combina- The second experiment will introduce tions of image sets with the same model, our and test an extremely important feature model choice will be a control and therefore of using simulator images - the ability to we will again stress the importance of relia- detect novel planetary features, i.e. those bility and accuracy over computation time which have never been seen in any real im- or creation date. ages. Since simulators can be programmed Our hypothesis stems from our sec- to emulate real physics, the outcome can ond core process and states that simula- give us an extremely large number of realis- tor images will not decrease accuracy for tic looking planets with features that have 8 never been observed. In order to proceed, we need to use a planetary feature that exists in our solar system so that we can train using simulator images and test using real images. Planetary rings have a solid theoretical foundation and would easily ap- pear in any physics-based simulator, while also being present around . Rings are also fairly complex, as they contain ex- tremely unique striation patterns, can look wildly different depending on the viewing angle, and can even co-exist with other rings around the same host planet. Figure 5: Results shown in Canziani et al.(2016) that compare model accuracy vs. operation count 3.2. The Model during the ImageNet challenge. When using deep learning models for a specific purpose, it is imperative that a training phase introduces the vanishing gra- model is chosen that optimizes what it can dient problem (He et al., 2015). Along with while prioritizing what it must. For exam- this feature, ResNet is a very established ple, an object detection model that may be model in many domains, and for these rea- implemented on a smart phone for real-time sons, will be our model of choice going for- detection of human faces might prioritize ward. speed and give up a small amount of ac- 3.3. Experiment #1 - Planetary Detection curacy. In this experiment, we tested the theory For our purposes, accuracy is of the ut- that simulator images could be used to train most importance, while operation count is a model that could then detect real objects, also of some importance. In deep space, we and in particular, planets. Our hypothesis have plenty of time to do calculations, but is that simulator images are at least as good we also have very little energy. Therefore, as real images in terms of information gain reaching maximum accuracy with a small during training. Although simulator images amount of operations is the ideal scenario. can be produced in bulk, the idea was to As we can see from Figure 5, origi- test the theory using similar image count in nally presented in Canziani et al. (2016) order to avoid any bias. The table below whereas the authors compared many models shows the number of images used for each for practical applications like this one, there model and for their testing phase. are few models that fit into the optimal space of accuracy and operations. The main Real Sim Real+Sim choice was ResNet architecture vs. Incep- tion architecture. The accuracy and opera- 120 122 242 tions for both models are almost identical, yet the residual neural network (ResNet) ar- For each of the three separate runs, the chitecture provides a shortcut in case the models reached a minimum of 60,000 itera- 9 tions in order to ensure ample training time 3.4. Experiment #2 - Novel Features and accuracy. Concerning the testing im- ages, the images were broken down into 2 In order to be able to show the impor- sections. The first section was comprised of tance of using a simulator for novel plane- independent images taken at differing an- tary features, we use the results from Exper- gles. The second section was the exact same iment #1 as proof of concept. Those show frame of reference, including angle and dis- that Sim images do not decrease accuracy tance, but included a time-lapsed series of on real testing images, with the added ben- images. efit of being able to mass produce them and The results of the testing phase produced customize feature information in each im- a model accuracy, which is a mathematical age. score given by the model as to how surely With this in mind, Experiment #2 will it has identified an object that it has previ- gather simulator images of ringed planets ously seen during the training phase. The and train a new model with the same frame- table below shows the final accuracy results. work as Experiment #1. We will then test novel feature detection on real images of Saturn. The machine will have never seen Section Real Sim Real+Sim any real image and will have never been exposed to prior knowledge of Saturn or 1 99.875 99.375 99.875 our solar system at all. This experiment 2 98.889 100 100 is, in theory, identical to training a model Total 99.353 99.706 99.941 with simulator images on Earth and send- ing it out into deep space in order to iden- tify novel, never-before-seen features found The results show quite a few compelling on real exoplanets and in real images. results right off the bat. The most direct One of the main benefits of simulator im- one being that Real+Sim has achieved equal ages can be observed here. Even a planetary or better results than Real or Sim alone did feature that we can observe will be found in all categories. Besides Real+Sim, we can once, or perhaps a few times at best. There- also say something about Real vs. Sim. Al- fore, we have limited variability to work though Real achieved slightly better accu- with in terms of ring structure, width, pat- racy in Section 1, Sim not only achieved tern, count, etc.. Yet, if this experiment better accuracy in Section 2, the total ac- produces promising results, we can simply curacy of Sim was also higher and Sim con- build a physics-based simulator that gener- tained at least one section that had perfect ates planets, filter by the presence of rings, accuracy. capture an image, and repeat the process Our initial hypothesis was that using any amount of times. From Experiment #1, simulator images would be as good as real we know that training on these simulator images. Different models and training im- images will net us generally equivalent in- age sets will always produce different re- formation gain when compared to real im- sults, but considering total accuracy with ages of ringed planets. Since our simula- our two sections, Sim produced equal or tor is physics-based, it should produce many better results when compared to Real. features that we have not even seen before, 10 transforming this problem from novelty de- having to detect unknown features, we can tection into object detection. simply construct planets randomly based on The experiment was set up in parallel to physical laws and train a model using those what would hypothetically happen during simulator images. deep space exploration. The training was done on a small batch of simulated images 4. Simulator for Novelty Ranking of ringed planets. The idea in the experi- ment is that, in theory, we have never seen We have shown previously that simula- a ringed planet before. Yet, our physics- tor images can be used with astounding based models of planet formation dictate accuracy, and with mass production, can that they would naturally occur. So we col- make training via real images unnecessary. lect simulated images, train on that, and Therefore, we can train using hundreds or then send it deep into space. Upon finding thousands of simulated images and when we a ringed planet for the first time, it would encounter a planet, we can detect it, image need to recognize those planets. Normally, it, and send those images back to Earth. we wouldn’t be able to do this since we have Say, for instance, that the wafer passes no ringed planets to train on (in this hypo- and images the five planets in a hypothetical thetical experiment), but since we used sim- solar system. Soon after that, it may be on ulated images, we now have a model to deal an inevitable course toward that solar sys- with this. The experiment goes through tem’s star, which will destroy the wafer and this entire process, and even tests the model all of the images. One downside to small on real images of Saturn. Again, the ma- wafers is that they are easily destroyed or chine has only seen a small batch of ran- corrupted. This makes a priority system vi- domly generated simulated images of hypo- tally important, as it would allow the wafer thetical ringed planets, never a real image to possibly send back one or two images of a planet. The results of the experiment from the five that it collected before it is are extremely positive and can be seen in destroyed. This section is dedicated to fig- the table below. uring out which images should be sent back, and discussing the approach in doing so.

Type Accuracy Detection Errors 4.1. The Concept Planet 99.22 None We will assume that we have a small stor- Rings 99.11 None age of images that we need to send back to Earth in an order that is based on impor- tance. Wafers could be destroyed relatively As we can see, the model is dependably easily and data transmission rates in space accurate based on solely simulated images. are very slow, so sending data based on a This experiment shows a key point of us- notion of importance is paramount. ing simulators - by combining planet gener- Figure 6 helps show us the extremely ab- ation theory and realistic rendering, we have stract definition of importance that we, as turned a novelty detection problem into an humans, may place on new planets. The object detection problem, which is signifi- top planet is colorful and full of land and cantly easier to deal with. Now, instead of different bodies of liquid, while the bottom 11 4.2. The Human Experiment The implausibility of teaching vast con- ceptual knowledge to a machine in hopes of it gaining context made us seek out a dif- ferent approach:

1. Generate simulated images of planets that range in features. This will re- move the bias that some people may have about our own solar system, since we are not using any real images of our own planets. It will also allow data to be gathered about features that do not currently exist in our solar system, but based on astrophysical theory, could exist in the universe. Figure 6: Hypothetical storage of two images that need to be ranked based on importance. 2. Ask experimental subjects to rate each planet by Importance. This is posed via the question: ”On a scale from 1-7, how important would it be for humankind to see this image if it were gathered by planet has a unique double ring, an atmo- a spacecraft during deep space explo- sphere, and a single large ocean, one that ration?” may be assumed to be water by visual in- spection alone. The main question we want 3. Ask experimental subjects to rate each to ask here is: If you could only send one of planetary feature. This will comprise these images back to humanity, which would our total planetary feature set. For you send? instance: On a scale from 1-7, how much does this planet exhibit the pres- An astrobiologist might choose the top ence.. of rings? ..of an atmosphere? planet since the presence of land and many ..of ? ..of a livable environment differently-composed bodies of liquid exist, for humans? giving multiple opportunities for life to pos- 4. Using the data gathered from human sibly flourish. Yet, someone interested in thought processes and individual anal- another planet that may be able to accom- ysis of importance and interestingness modate human existence might choose the of a planet, train a model to predict bottom planet since it seems to offer two importance given a feature set. important features for us, water and an at- 5. Rank all planets in storage based on mosphere. The question of importance to importance and send them back to humans is very subjective, yet we need a Earth via this priority system. solution that would be able to rank these two planets, and many more, in order of This process takes in human definitions importance. and thought processes in order to break 12 down the concept of what we find inter- 5.1.1. Phase One esting in planets that we have not even Phase One is essentially comprised of seen before. Using this method, we can by- time spent in open-space travel. This would a problem of novelty detection, which mean that the probe is beyond a ’fair’ dis- is difficult, and machine contextual learn- tance away from any nearby star and that ing, which is extremely difficult, and turn it planetary detection would be a fruitless en- into a problem of human information gain, deavor. Yet, during this phase, the main which is easy, and object detection, which objectives would be nearby star detection is also easy. and star distance predictions. Future work will show experimental re- sults beyond this proof of concept solution. 5.1.2. Phase Two Phase Two would be a rare occurrence 5. Simulator for Energy Management whereas the probe has traveled within a ’fair’ distance of a star and we no longer During a deep space voyage, the wafer- need to deal with nearby stars until we have sat will need to be supplied with enough en- left that star’s system. Instead, this phase ergy to perform necessary functions, such as would prioritize planetary detection, imag- imaging, analyzing the images, and trans- ing, and ranking. mitting data. We don’t assume that the system is perfect, nor do we need certain re- 5.2. The Process strictions on the amount of energy available. We have one goal: minimizing the amount The trip from Earth to Alpha Centauri of energy needed while ensuring planetary can be done in approximately 20 years. But, detection. At one end of the spectrum is in the simulator, one can travel at any speed full energy conservation, which would mean and cut out the majority of the time spent that the camera never turns on and there- in an uneventful space. This makes it pos- fore we never collect any data. On the other sible to simulate a 20 year journey in a few end of the spectrum is full energy use, mean- hours, or many journeys in just a single day. ing that the camera never turns off until the Once these are done, we can train a ma- energy runs out, which would yield us many chine learning model using star type, the images but most likely none with important section of the image containing the star, and findings. Somewhere in between is optimal, the distance from the probe to the star (a but how do we find it? simulator feature). Combined with a sub- traction algorithm, and only using enough 5.1. The Two Phases energy to take two images, the machine will Simulators open a whole new universe be able to identify stars and predict their that can be utilized in order to make a vir- distance from the probe. tual interstellar journey to Alpha Centauri Using this information, the probe will hundreds of times in the span of a day. By know the approximate distance to the near- doing this process, we can train our models est star in its forward path. A simple cal- to identify stars, predict distances, swap be- culation can tell it a safe amount of time to tween the two possible phases, and in doing wait until it should take two more images, so, save energy while capturing meaningful confident that the time it has waited has images. been uneventful. 13 Repeating this process is extremely en- black. If you wait too long, even things ergy efficient, and should eventually lead to that are very far away will begin to shift coming within a reasonably ’fair’ distance and you will be left with a large amount of from a star. When this occurs, we would stars, still unsure about which of those are change into Phase Two. actually close. This difficult part becomes Phase Two would use the same intuition approachable with the use of simulators. except that instead of stars, we substitute The example deals with a simulated star in planets. Once identified, instead of be- that exists 0.08 light-years away from the ing interested in distances, we would prior- probe. We travel at 500c and perform a sub- itize imaging. Details on planetary detec- traction algorithm in 10 second intervals, re- tion, imaging, feature extraction, and rank- setting after each one. This equates to trav- ing have been detailed in earlier sections. eling at 0.25c and performing a subtraction algorithm every 20,000 seconds, or approxi- 5.3. An example of Phase One mately every 5.55 hours. So, the first image One extremely difficult concept in this is 5.55 hours in real-time, the second im- entire process is making sure that the probe age is 11.11 hours in real-time, then 16.66 can successfully understand what is close to hours, and so on. The goal is to see if simu- it versus what is very far away. The con- lators can be useful, and if so, at what point cept used is straight-forward: bodies that we would want to optimally take images in are closer will tend to shift more while the order to ensure we capture bodies that are probe travels in a straight path. As an ex- nearby while also saving energy. treme example, a body that is 1 AU away As we can see in Figure 7, the first few from the probe will shift from center screen bins produce a hazy image of the target star. to completely off screen in approximately At the fifth image, which would have the 33 seconds. Yet, a very distant star could probe waiting approximately 28 hours be- go without changing position for months or tween images, we can see a full image of years. the star. By the last image, which is rep- In order to deal with this, a subtraction resented by approximately 50 hours of real algorithm is implemented. The probe will time, other nearby stars were showing very take a photo, wait a certain amount of time, hazy signs of recognition from the subtrac- and take another photo. Then, the first will tion algorithm. be subtracted from the second and the re- This proof of concept is extremely vital sulting image will show any pixels that have to star recognition and energy management. shifted state during the elapsed time. If Depending how far away from a star we enough of these pixels shift, we will get a want the probe to be when it is able to rec- clear image of something that is relatively ognize it, this process can be altered and close. honed easily. The main problem here, again, is that no- body has any concept of the ”wait a certain 6. Conclusion amount of time” part of the process. How much time is the right amount of time? If We began with a set of new challenges you do not wait long enough, nothing will that arise from the Starlight program and move and your subtracted image will be all the ability to perform fast interstellar travel. 14 not suffer, which is an astounding concept for such an application. Along with this, we provide results and many reasons why simu- lators enable us to identify features that we have yet to observe in actual images. With the use of simulators, we can run experi- ments on humans in order to extract the features of an important planet and use that knowledge to dictate decisions for planetary importance rankings. Lastly, we demonstrate how simulators can be utilized to save energy while ensuring that all necessary functions are completed. There is much planned future work on this topic. This includes optimizing simula- tors for this specific task, choosing the best hardware given restraints such as size and the harsh environment of space, and incor- porating more human knowledge gain into the AI process.

7. Acknowledgements

PML gratefully acknowledges funding Figure 7: Subtraction algorithm performed in 20,000 second intervals from NASA NIAC NNX15AL91G and NASA NIAC NNX16AL32G for the NASA Starlight program and the NASA California These include identifying stars, identify- Space Grant NASA NNX10AT93H, a gen- ing new planets, extracting never-before- erous gift from the Emmett and Gladys W. seen features, conceptually ranking these Technology Fund, as well as support from new planets against each other in terms the Breakthrough Foundation for its Break- of importance, understanding what impor- through StarShot program. More details on tance means in the context of planets, and the NASA Starlight program can be found conserving energy while performing needed at www.deepspace.ucsb.edu/Starlight. tasks. We started off by showing that a simple 8. Citations classification model would not suffice. Not only does it perform poorly, but it does not References come with the range of tools that are needed for further processes down the line. Abadi, M., Agarwal, A., Barham, P., et al. 2016, TensorFlow:large-scale machine learn- We show that while training on simulated ing on heterogeneous distributed systems, images, our accuracy on real images does arXiv:1603.04467v2 [cs.DC]. 15 Canziani, A., Paszke, A., & Culurciello, E. 2016, An strong gravitational lens detectors, A&A, 611, analysis of deep neural network models for prac- A2 (2018). tical applications, arXiv:1605.07678v4 [cs.CV]. Shallue, C.J., & Vanderburg, A. 2017, Identifying Carrasco-Davis, R., Cabrera-Vives, G., Frster, exoplanets with deep learning: a five planet reso- F., et al. 2018, Deep learning for image se- nant chain around Kepler-80 and an eight planet quence classification of astronomical events, around Kepler-90, AJ, 155, 94. arXiv:1807.03869v3 [astro-ph.IM]. Smyth, D.L., Glavin, F.G., & Madden, M.G. 2018, Chetlur, S., Woolley, C., Vandermersch, P., et al. Using a game engine to simulate critical inci- 2014, cuDNN: efficient primitives for deep learn- dents and data collection by autonomous drones, ing, arXiv:1410.0759v3 [cs.NE]. IEEE Games, Entertainment, Media Conference Connor, L. & Leeuwen, J. van 2018, Applying deep (GEM), Galway, 2018, pp. 1-9. learning to fast radio burst classification, AJ, Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, 156, 256. A. 2016, Inception-v4, Inception-ResNet and He, K., Gkioxari, G., Dollr,P., Girshick, R. 2018, the impact of residual connections on learning, Mask R-CNN, arXiv:1703.06870v3 [cs.CV]. arXiv:1602.07261v2 [cs.CV]. He, K., Zhang, X., Ren, S., Sun, J. 2015, Szegedy, C., Vanhoucke, V., Ioffe, S., et al. 2015, Deep Residual Learning for Image Recognition, Rethinking the inception architecture for com- arXiv:1512.03385v1 [cs.CV]. puter vision, arXiv:1512.00567v3 [cs.CV]. Huang, G., Liu, Z., Maaten, L.van der, Weinberger, Zucker, S., Giryes, R. 2018, Shallow transits - deep K.Q. 2018, Densely connected convolutional net- learning I: feasibility study of deep learning to works, arXiv:1608.06993v5 [cs.CV]. detect periodic transits of exoplanets, AJ, 155, Kim, J., Lee, S., Oh, T.-H., & Kweon, I.S. 147. 2017, Co-domain embedding using deep quadru- plet networks for unseen traffic sign recognition, arXiv:1712.01907 [cs.CV]. Kulkarni, N., Lubin, P., & Zhang, Q. 2017, Rela- tivistic spacecraft propelled by directed energy, arXiv:1710.10732 [astro-ph.IM]. Marsland, S., Shapiro, J. & Nehmzow, U. 2002, A self-organising network that grows when re- quired, Neural Networks, 15, Issue 8-9, 1041- 1058. McDuff, D., Cheng, R., & Kapoor, A. 2018, Identifying bias in AI using simulation, arXiv:1810.004171 [cs.LG]. Morad, S., Nallapu, R.T., Kalita, H., et al. 2018, On-Orbit smart camera system to observe illu- minated and unilluminated space objects, arXiv: 1809.02042 [cs.CV]. Pearson, K.A., Palafox, L., & Griffith, C.A. 2017, Searching for exoplanets using artificial intelli- gence, arXiv:1706.04319v2 [astro-ph.IM]. Rajpurkar, R., Irvin, J., Zhu, K., et al. 2017, CheXNet: radiologist-level pneumonia detec- tion on chest x-rays with deep learning, arXiv:1711.05225v3 [cs.CV]. Ruffio, J.-B., Mawet, D., Czekala, I., et al. 2018, A bayesian framework for exoplanet direct detec- tion and non-detection, AJ, 156, 196. Schaefer, C., Geiger, M., Kuntzer, T., & Kneib, J.- P. 2018, Deep convolutional neural networks as

16