Designing Low-Correlation GPS Spreading Codes with a Natural Evolution Strategy Machine Learning Algorithm

Tara Yasmin Mina and Grace Xingxin Gao, Stanford University

ABSTRACT With the birth of the next-generation GPS III constellation and the upcoming launch of the Navigation Technology Satellite-3 (NTS-3) testing platform to explore future technologies for GPS, we are indeed entering a new era of satellite navigation. Correspondingly, it is time to revisit the design methods of the GPS spreading code families. In this work, we develop a natural evolution strategy (NES) machine learning algorithm with a Gaussian proposal distribution which constructs high-quality families of spreading code sequences. We demonstrate the ability of our algorithm to achieve better mean-squared auto- and cross-correlation than well-chosen families of equal-length Gold codes and Weil codes, for sequences of up to length-1023 and length-1031 bits and family sizes of up to 31 codes. Furthermore, we compare our algorithm with an analogous genetic algorithm implementation assigned the same code evaluation metric. To the best of the authors’ knowledge, this is the ﬁrst work to explore using a machine learning approach for designing navigation spreading code sequences.

I. INTRODUCTION On January 13th of 2020, the U.S. Air Force 2nd Space Operations Squadron (2 SOPS) issued a statement that the first GPS III satellite was marked healthy and available for use [1]. This announcement officially marked the birth of the next-generation GPS constellation. In addition to broadcasting the new L1C signal, the modernized constellation is distinguished by its reprogrammable payload, which allows it to evolve with new technologies and changing mission needs. Furthermore, with the upcoming launch of the Navigation Technology Satellite-3 (NTS-3) testing platform in 2022 [2], the United States Air Force (USAF) seeks to explore technologies which will help shape future GPS constellations [3]. NTS-3 will demonstrate the agility of the next-generation satellite-based navigation architecture and the ability to rapidly deploy new technological advancements and capabilities via the reprogrammable nature of the upcoming GPS system. Indeed, this is the third Navigation Technology Satellite (NTS) mission, with the previous two, NTS-1 and NTS-2, developed in the 1970s in order to validate technologies, including the rubidium and cesium atomic clocks [4], that were integrated into the first generation of GPS satellites, which were launched later that decade [5]. Illustrations of the three NTS testing platforms are shown in Fig. 1. arXiv:2101.02850v2 [eess.SP] 23 Apr 2021

Figure 1: Illustrations of the satellite testing platforms used in each of three Navigation Technology Satellite (NTS) initiatives. Illustration credit: Lt. Jacob Lutz, AFRL Space Vehicles Directorate. The NTS-3 program will further test several new technologies, including new signal designs for improved GPS security and interference mitigation [3]. According to a Request for Information announcement [6], the Air Force Research Lab (AFRL) has expressed interest in exploring modiﬁcations to all layers of the GPS signal in order to enhance PNT resiliency and performance. As we enter a new era of satellite navigation, we believe it is time to revisit the design methods of the GPS spreading code families. Correspondingly, in this work, we leverage recent advances in machine learning techniques to explore a new platform for learning high-quality spreading codes via a natural evolution strategy (NES) machine learning framework.

1. Advantages of Using Memory Codes for GPS Currently, all broadcast GPS signals use algorithmically generable spreading code sequences. The legacy L1 C/A signal uses Gold codes [7], while the modernized L1C signal, to be broadcast on the GPS III satellites, uses Weil codes [8] [9], along with a 7-bit extension in order to satisfy the desired spreading code length of 10,230 bits. Having algorithmically generable sequences was an important design consideration during the initial development phase of GPS in the 1970s due to memory storage constraints, as well as computation limitations which restricted the ability to search for better sequences. However, memory has become an increasingly inexpensive resource, allowing receivers to store complete families of spreading code sequences – though it should be acknowledged that currently, memory is still one of the main influencers of receiver chip size [10] [11]. Furthermore, over the past several decades, processing power has also significantly increased, which correspondingly has allowed the field of machine learning and related research advancements to flourish [12]. Lower bounds on the maximum periodic non-central auto- and cross-correlation value for spreading code families have been analytically derived in prior work [13], including the Welch bound [14], the Sarwate bound [15], and the Sidelnikov bound [16] [17]. The Welch bound is valid for any sequence with symbols of a fixed norm, including potentially complex spreading codes, while the Sidelnikov bound is a tighter lower bound for sequences with symbols that are roots-of-unity. As a result, for binary spreading codes, which fall into this roots-of-unity classification with symbols in the set {+1, −1}, the Sidelnikov bound provides a tighter lower bound than the Welch bound on the maximum periodic correlation performance. Gold code families are known to be asymptotically optimal with respect to the Sidelnikov bound, as the code length increases [18]. However, this optimality characteristic only applies for the specific sequence length and the complete set of code sequences provided by the Gold code family definition. For the length-1023 bit Gold codes, for example, this optimality characteristic only applies for the complete Gold family size, made up of over 1000 spreading codes in total [7]. Indeed, the lower bounds on the maximum correlation sidepeak increase with the size of the spreading code family [14] [15] [16] [17], suggesting that for the same sequence length, better performance could be found for a smaller code family size. Furthermore, there is debate as to whether the maximum correlation sidepeak is an ideal metric for spread spectrum system applications [19] [20] [21] [22]. Indeed, the multiple access interference (MAI) of a spread spectrum system, i.e. the signal noise caused by inter-signal interference within the shared channel, relates to the average correlation performance rather than the worst-case. Spreading codes which are not algorithmically generable are commonly called memory codes, since these codes must be stored in memory. Memory codes are not limited to specific lengths of code sequences or specific code family sizes. Indeed, designing a nonconforming spreading code length, as was done for the GPS L2C, L5, and L1C signals, would require truncation or extension of the algorithmically generable sequences, which disturb the coveted correlation properties of these codes [23]. As a result, by expanding the design space to the set of all binary sequences, we have a greater range of possible code families and a better opportunity to find a superior set of codes; although, the exponentially larger design space also greatly complicates the code design method.

2. Related Prior Work Several past works have designed memory codes using genetic algorithms (GAs), including for the E1 Open Service (OS) and the E6 Commercial Service (CS) signals of the Galileo satellite constellation [23] [24]. Genetic algorithms [25] are optimization algorithms which mimic the process of biological evolution. In particular, GAs maintain a population of design points, while selecting and recombining high-performing candidate solutions at each iteration. Genetic algorithms were utilized to design spreading codes for Galileo which exhibit the Autocorrelation Sidelobe Zero property, where the auto-correlation is 0 at a relative delay of ±1 chip [23]. GAs were also utilized to design spreading code families for improved indoor positioning applications [26]. Work has also been done to deﬁne several cost parameters for selecting high quality GNSS spreading codes [27] [28], including evaluating the mean squared auto- and cross-correlation components above the Welch bound [14]. Additionally, in our prior work, we designed a multi-objective GA platform for designing navigation spreading codes which improved on two objectives simultaneously: the mean absolute auto-correlation and cross-correlation of the code family [29]. Learning-based techniques have also been explored to design Error Correcting Codes (ECCs) [30], which are binary sequences that encode messages to improve the detection of communication transmission errors. In their work, Huang et al. [30] explore policy gradient and advantage actor critic methods to design ECCs by optimizing the code set performance with regards to the block error rate. 3. Objective and Key Contributions In this work, we seek to explore using a machine learning technique for the application of navigation spreading code design. This work is based on our recent ION GNSS+ 2020 conference paper [31], and to the best of our knowledge, this is the ﬁrst work which explores using a machine learning approach to design navigation spreading codes. In particular, the key contributions of our work are the following: 1. We develop a natural evolution strategy (NES) machine learning algorithm which constructs high-quality families of spreading code sequences. 2. We utilize a Gaussian proposal distribution parameterized by a generative neural network (GNN) in order to model a search distribution over the design space of possible binary spreading code families. 3. We incorporate a baseline to reduce the variance of the NES gradient estimate and to improve the rate of learning. 4. We use a maximization evaluation metric to ensure the algorithm minimizes both the auto-correlation and cross- correlation characteristics of the spreading codes simultaneously. With our algorithm, we demonstrate the ability to achieve low auto- and cross-correlation side peaks within the family of spreading codes. We further compare the correlation performance of the learned spreading codes with those of well-chosen families of equal-length Gold codes and Weil codes as well as with an analogous genetic algorithm implementation assigned the same code evaluation metric as our proposed algorithm.

4. Paper organization The remainder of the paper is organized as follows: Section II. provides relevant technical background for the paper; Section III. describes our NES machine learning algorithm for the design of spreading code families; Section IV. presents our experimental validation setup and results; Section V. concludes this paper; and Section VI. provides additional background on artiﬁcial neural networks.

II. BACKGROUND 1. Binary Sequence Representation For mathematical convenience, we represent a binary sequence x¯(0,1) with elements of the set {0, 1} as a sequence x¯ with elements of the set {+1, −1} by using the following conversion ∀i :x ¯(0,1)(i) = 0, x¯(i) = +1 ∀i :x ¯(0,1)(i) = 1, x¯(i) = −1 , (1) where x¯(0,1)(i) and x¯(i) represent the ith element of the two binary sequence representations. In this paper, unless otherwise speciﬁed, we represent sequences using the (+1, −1) binary representation. To denote sequences with the (0, 1) binary representation, we utilize an additional superscript of (0, 1), i.e. x¯(0,1).

Figure 2: Illustration of the conversion from the (0, 1) binary sequence representation to the (+1, −1) representation used for mathematical convenience. Consequently, the exclusive-OR operation between elements of the (0, 1) representation becomes a multiplication operation.

The (+1, −1) representation of the binary sequences simpliﬁes the auto-correlation and cross-correlation computations performed on one or more binary sequences. In particular, an exclusive-OR operation, i.e. b1 ⊕ b2, between two elements of the (0, 1) binary representation becomes a simple multiplication operation, i.e. b1 ·b2, between two elements of the (+1, −1) binary representation, as illustrated in Fig. 2.

2. Periodic Auto- and Cross-Correlation th Using the (+1, −1) binary sequence representation deﬁned in Eq. (1), let xk represent the k binary sequence in a family of th length-` spreading code sequences. Let xk(i) ∈ {+1, −1} represent the i element of the sequence, where i ∈ Z∩[0, `−1] and Z represents the set of all integer values. Because xk is a periodic sequence, we have that xk(i) = xk(i + `). The normalized, periodic even auto-correlation of sequence k at a relative delay of δ bits is deﬁned as

`−1 1 X R (δ) := x (i)x (i − δ) . (2) k ` k k i=0

Similarly, in Eq. (3) we deﬁne the normalized, periodic even cross-correlation between sequences k and m at a relative delay of δ bits as

`−1 1 X R (δ) := x (i)x (i − δ) . (3) k,m ` k m i=0

The even correlation is distinguished from the odd correlation, which defines the correlation in the case that a data flip occurs within the integration period of the correlators [28] [32]. Correspondingly, the odd correlation for sequence k would be computed by flipping the sign of the second multiplicative terms of Eq. (2) and (3), i.e. xk(i − δ) and xm(i − δ) respectively, when i < δ.

3. Algorithmically Generable GPS Spreading Codes Currently, all broadcast GPS signals use algorithmically generable spreading code sequences. The legacy L1 C/A GPS signal, for example, uses length-1023 Gold codes, which can be generated with two 10-bit linear feedback shift registers (LFSRs), as shown in Fig. 3. Gold codes [7] are a class of binary spreading code families which can be generated from two maximum-length sequences, or m-sequences. Maximum-length sequences are pseudorandom sequences of length ` = 2n −1, which are generated using an n-bit LFSR [33].

Figure 3: Diagram of the linear feedback shift register (LFSR) pair with their corresponding characteristic equations (G1 and G2) utilized to generate length-1023 bit Gold codes used in GPS L1 C/A [34]. Only two 10-bit LFSRs are required for the generation of the GPS L1 C/A spreading codes.

Recently developed for the modernized GPS L1C signal, Weil codes are families of spreading codes which were initially proposed by Guohua and Quan [8] and further analyzed and proposed for the L1C signal by Rushanan [9] [35]. A Weil code family exists for any prime number sequence length p and is generated from the corresponding length-p Legendre sequence, constructed from the Legendre symbol [36] [37]. In this work, we construct both Gold codes and Weil codes in order to compare the performance of these spreading code families utilized in GPS with the performance of the spreading codes generated from our proposed NES machine learning algorithm.

4. Natural Evolution Strategy Algorithms Natural Evolution Strategies (NES) [38] [39] [40] [41] are a subset of stochastic optimization methods which model a probability distribution, or a proposal distribution, over the optimization design space. Each sample from this proposal distribution outputs a candidate design point. In the context of designing navigation spreading codes, for example, each sample of the proposal distribution would output a set of binary sequences. NES requires choosing a family of parameterized proposal distributions PΘ, explicitly deﬁned as

PΘ = {pθ(x): x ∈ X , θ ∈ Θ} where Θ is the set of possible vector-valued parameters θ deﬁning each distribution in the family and where X represents the design space over which we are optimizing.

Within the specified family of probability distributions, NES methods directly optimize the proposal distribution pθ with respect to its parameterization θ, in order to guide the search process towards higher performing regions of the design space. NES algorithms are especially useful for non-convex optimization problems with large design spaces, including combinatorial design spaces, which applies to our application of interest with the design of navigation spreading codes. For high-dimensional design spaces, often diagonal covariance matrices are considered for the proposal distribution in order to improve computational efficiency with regards to the number of NES parameters to optimize as well as sample efficiency with regards to the number of required Monte Carlo samples to effectively estimate the search gradient during the optimization step [40] [42]. Further variations of traditional NES methods have also been explored to handle highly multi-modal optimization landscapes by leveraging heavy-tailed distributions, including those with undefined variances, such as the Cauchy distribution [42].

NES methods optimize the proposal distribution within the deﬁned family PΘ, by seeking to minimize the expected objective function J, deﬁned as

J(θ) := Ex∼pθ [f(x)] Z = pθ(x)f(x)dx , (4)

where x represents a particular point in the design space X , pθ(x) represents the probability of selecting x given the parameter θ, and f(x) represents the objective function value of the design point x, which we seek to minimize. Thus, NES methods seek to ﬁnd the best proposal distribution by solving the optimization problem θ∗ = arg min J(θ) . (5) θ

To optimize the proposal distribution, the gradient of the expected objective function, i.e. ∇θJ(θ), can be estimated via Monte Carlo sampling of the objective function using the current proposal distribution [43] [41] [44]:

∇θJ(θ) = ∇θEx∼pθ [f(x)] Z = f(x)∇θpθ(x)dx Z ∇θpθ(x) = f(x) pθ(x)dx (6) pθ(x) Z = [f(x)∇θ log pθ(x)] pθ(x)dx (7)

= Ex∼pθ [f(x)∇θ log pθ(x)] (8) N 1 X ≈ f(xi)∇ log p (xi) (9) N θ θ i=1 where N represents the number of samples from the proposal distribution and xi represents the ith sample. Eqs. (6) and (7) demonstrate the so-called “log-derivative trick,” which is observed in a variety of machine learning contexts, including reinforcement learning and variational inference [44]. Figure 4: Illustration depicting how the NES algorithm optimizes the expected objective J(θ). From instance A to instance B, the NES optimizer updates the parameters of the proposal distribution in order to increase the likelihood of design points which lead to lower values of the objective function (indicated by the green sampled points), while decreasing the likelihood of design points which have higher objective function values (indicated by the red sampled points). This update then guides the proposal distribution down the objective function landscape in instance B, toward regions where the objective is lower. The optimizer again repeats this process from instance B to instance C, thereby further guiding the proposal distribution down the objective function landscape. Indeed, the stochasticity of the proposal distribution induces the NES algorithm to explore the design space and the optimization step iteratively steers the search process towards better performing regions of the design space, where the objective function value is lower.

From this gradient estimate, using any ﬁrst-order optimizer (e.g. stochastic gradient descent, Adam [45]), NES improves its proposal distribution vector-valued parameter θ in order to minimize the expected objective function in Eq. (4)

θ ← θ − δθ

δθ := gopt (∇θJ(θ)) , (10) where gopt is the generic ﬁrst-order optimizer utilized to iteratively improve the proposal distribution. As illustrated in Fig. 4, this optimization process increases the probability of design points which lead to lower values of the objective function f to be minimized and decreases the probability of those which result in higher values of the objective function. Indeed, the stochasticity of the proposal distribution induces the NES algorithm to explore the high-dimensional design space and further improve its proposal distribution to output better performing design points.

III. PROPOSED NATURAL EVOLUTION STRATEGY ALGORITHM FOR SPREADING CODE DESIGN 1. High-Level Learning Framework For the spreading code design application, the machine learning framework utilized is represented at a high-level in Fig. 5. A code generator maintains a proposal distribution over the spreading code design space. Then, an evaluator assesses the performance of the code generator with respect to the objective function f by sampling from its proposal distribution. The objective function represents an evaluation metric for the spreading code family, based on the auto- and cross-correlation properties of the code set. Finally, as in Eq. (9), the optimizer estimates the gradient of the expected objective function with respect to the parameters of the proposal distribution, as represented by a generative neural network (GNN). The optimizer then updates the network parameters in order to improve the proposal distribution for the code generator.

Figure 5: High-level learning framework utilized for spreading code generation. The code generator outputs a set of sampled candidate code sets which are assessed by the evaluator. From the sampled design points and corresponding objective function values, optimizer improves the proposal distribution by updating its parameters as represented by a generative nerual network. 2. Gaussian Proposal Distribution Representation Using a neural network architecture, we represent a probability distribution over the design space of all possible binary spreading code families. Thus, the parameters of the proposal distribution θ correspond to the hidden layers of the network. As depicted in Fig. 6, for the code generator to output a set of binary codes, it must sample from this proposal distribution provided by the GNN, with the currently learned network parameters θ. For a concise introduction to artiﬁcial neural networks, we would suggest the reader to reference Sec. VI.

Figure 6: Illustration of generative neural network (GNN) and proposal distribution representation. The GNN outputs the mean vector of the multivariate Gaussian proposal distribution, from which a sampled output code set for the complete spreading code family is generated. In particular, the sampled output code set is a binary vector of length-K` bits, where K denotes the number of codes in the family and ` denotes the length of each spreading code sequence. With this representation of the proposal distribution, the network parameters correspond to the NES parameters θ, which are optimized during training.

We model the code design proposal distribution as an uncorrelated multivariate Gaussian family as depicted in Fig. 7, where each component corresponds to an index in the complete family of binary spreading codes. Our neural network model represents this proposal distribution by parameterizing the mean vector of the Gaussian distribution via a bounded output activation function, such as the logistic or hyperbolic tangent output activations. The variance σ2 of each component is maintained as a constant value. We can represent this Gaussian proposal distribution in terms of the network parameters θ as

2 (K`) pθ(x) = N µ(θ), σ I , (11) where K denotes the number of codes in the family, ` denotes the length of each sequence in the code family, I(m) represents the identity matrix of size (m × m) where m is a positive integer, and µ(θ) represents the mean of the Gaussian proposal K` distribution, as output by the GNN with parameters θ. Note that the mean of pθ is bounded, i.e. µ(θ) ∈ [µL, µU ] , due to the bounded nature of the output activation function of the network. Due to the uncorrelated model of the multivariate Gaussian proposal distribution, each component represents a univariate Gaussian distribution over each binary value in the spreading code family. As depicted in Fig. 7, to obtain a random output family of spreading codes, the code generator first samples directly from the multi-variate Gaussian proposal distribution defined in Eq. (11), i.e. x ∼ pθ. From this sampled vector x, the output code set is deterministically obtained by discretizing each component according to a threshold ζ defined in Eq. (13), which we choose to be the midpoint of the bounded output range of Figure 7: Illustration of the proposal distribution utilized for spreading code design. We model the code design proposal distribution as an uncorrelated multivariate Gaussian family, where each component corresponds to an index in the family of binary spreading codes. Correspondingly, the total length of the spreading code set is K`, where K denotes the number of codes in the family and ` denotes the length of each binary spreading code sequence. the GNN. Correspondingly, the output binary code family x can be defined in an element-wise manner as in Eq. (14):

2 xi ∼ N µi(θ), σ . (12) µ + µ ζ := L U (13) 2 xi = 1{xi ≥ ζ} . (14)

Thus, after sampling the design point x from the proposal distribution pθ, if a component of x is above the threshold ζ, the corresponding sampled bit in the output binary code set x is a 1; otherwise, the corresponding bit is a 0. Note that the design space X , where x ∈ X , is in fact a continuous design space. Indeed, we map each continuous design point x to a binary vector x according to the threshold ζ as in Eq. (14). Correspondingly, we evaluate the performance of the design point x by utilizing its discretized binary vector x with the spreading code evaluation metric, as deﬁned in Sec. 3.

3. Spreading Code Evaluation Metric In the NES framework, the evaluation metric provides feedback to the code generator in order to improve its proposal distribution. In this work, we consider the mean-square non-central even auto-correlation and the mean-square even cross-correlation as the two objectives to minimize, deﬁned respectively as

K `−1 ! 1 X X f := |R (δ)|2 (15) AC K` k k=1 δ=1  K K `−1  1 X X X 2 fCC :=  |Rk,j(δ)|  , (16) Kp` k=1 j=k+1 δ=0

K K x ` K := where represents the number of sequences in the output code family , represents the length of each sequence, p 2 th represents the number of pairs of sequences in the code family of size K, Rk represents the auto-correlation of the k sequence in the output code family x as defined in Eq. (2), and Rk,j represents the cross-correlation of sequences k and j of the output code family x as defined in Eq. (3). In order to ensure the code generator reduces both objectives, we define the comprehensive objective function as the maximum of the two components from Eqs. (15) and (16)

f(x) = max (fAC , fCC ) . (17) Thus, to minimize the objective, the NES algorithm will seek to reduce both components of the objectives simultaneously, without compromising one component over the other. Indeed, modiﬁcations of the objective function in Eq. (17) could be readily incorporated within this machine learning framework. In particular, if seeking to design spreading codes with smaller auto-correlation sidelobes than cross-correlation sidelobes, one could scale fAC by a corresponding factor that is greater than 1 in Eq. (17), which would encourage the algorithm to generate codes that reduce the auto-correlation metric in comparison to the cross-correlation metric by the corresponding user-deﬁned factor. Furthermore, additional components of objectives could be incorporated into the comprehensive objective function, including, for example, odd correlation performance metrics and relative Doppler frequency correlation metrics. These additional objectives could be incorporated as new arguments within the maximization function of Eq. (17) with relative scaling factors if desired, or these objective could be combined via a weighted summation, as proposed in [27].

4. NES Optimization

We optimize the network parameters θ, which deﬁne the proposal distribution pθ, by following the NES optimization process described in Section 4. To reduce the variance in the gradient estimate, we incorporate a constant baseline b, which is subtracted from the objective function in the gradient estimate expression of Eq. (9) [44]. Thus, with the baseline subtraction, the gradient estimate from Eq. (9) becomes

N 1 X ∇ J(θ) ≈ f(xi) − b ∇ log p (xi) , (18) θ N θ θ i=1 where N represents the cardinality of the batch of samples {x1, x2, . . . , xN }. For our constant baseline b, we utilize the mean return in the current batch of samples, deﬁned as

N 1 X b := f(xi) . (19) N i=1

Subtracting a constant value does not introduce any bias in the gradient estimate, but it can signiﬁcantly reduce its variance [39] [44]. Indeed, by reducing the variance of the gradient estimate, we observe improved convergence and overall learning performance during training. Section IV. 2. demonstrates the learning rate of the NES algorithm with and without the incorporation of this baseline from Eq. (19).

IV. EXPERIMENTAL VALIDATION 1. Details of Experimental Setup We validate the ability of our NES machine learning algorithm to devise low-correlation spreading codes and further compare its performance with that of well-chosen families of equal-length Gold codes and Weil codes. We compare the performance of our algorithm with Gold codes of length-63, 127, 511, and 1023 bits. Similarly, since Weil codes only exist for sequence lengths that correspond to a prime number length, we compare our algorithm with Weil codes of length-67, 127, 257, 521, and 1031 bits.

Table 1: Design parameters of proposed NES machine learning method.

Parameter Description Value Gaussian proposal variance (σ2) 0.1 learning rate (for ` < 500) 10−4 learning rate (for ` > 500) 5 · 10−5 hidden layer size 2K` number hidden layers 2 output activation tanh network optimizer Adam [45] batch size (N) 100 number iterations 10, 000 From our NES method, we generate sequences for family sizes of 3 codes up to 31 codes. We additionally compare our proposed algorithm with the best performing code family across 10, 000 sampled families of Gold codes and Weil codes. In a few of the sample runs of Gold and Weil codes, we observed a large deviation in the auto- and cross-correlation cost components which frequently leads to worse performance on the overall objective defined in Eq. (17). In these instances, we would resample the conventional code families, leading to an improvement in the performance metric of the Gold and Weil codes. The details of our NES machine learning algorithm are indicated in Tab. 1. We utilize a slightly smaller learning rate for learning families with sequences of over length-500 bits, since we observed more consistent performance with a reduced learning rate for these longer sequence test cases. We use the hyperbolic tangent function as our bounded output activation function and Adam [45] as the optimizer of the GNN. Furthermore, we compare the performance of our algorithm with that of an analogous genetic algorithm which is set to optimize over the same maximization evaluation objective defined in Eq. (17). The GA implementation uses fitness proportionate selection [25], which stochastically selects a candidate solution from the population with a probability proportional to its normalized objective. Selected individuals are recombined using uniform crossover [46] which allows for greater recombination of candidate solutions and was observed to have better performance during initial testing. We additionally incorporated elitism [47], which ensures that the genetic algorithm will continuously improve during its optimization process.

Table 2: Design parameters of genetic algorithm implementation.

Parameter Description Value elite rate 0.01 mutation rate 0.005 selection method fitness proportionate [25] crossover method uniform [46] population size 100 number iterations 10, 000

In Tab. 2, we describe the parameters of the analogous genetic algorithm. For the GA to be comparable with the NES method, we set its population size to be the batch size of the NES machine learning algorithm, and we set the number of iterations for both algorithms to be the same.

2. Effect of Baseline Incorporation on Learning Performance Fig. 8 demonstrates the effect of incorporating the constant mean baseline on the learning performance during training, as measured by the normalized maximum correlation objective, i.e. max{fAC , fCC }, which the algorithm seeks to minimize. For both of the code length scenarios shown in Fig. 8, we observe that the incorporation of the baseline, shown in violet, leads to significant improvement in the learning performance during training, due to the reduction of variance in the gradient estimate. In particular, the baseline incorporation allows for more systematic learning and more consistent improvement on the correlation objective.

3. Comparison with Best Performing Gold and Weil Codes Fig. 9 shows the converged normalized mean-square auto- and cross-correlation performance of our NES algorithm after training, comparing it with the best Gold and Weil code families of equal-length. We plot the final performance as a function of the code family size, with the normalized auto-correlation component fAC represented by the dashed lines and the normalized cross-correlation component fCC indicated by the solid lines. Indeed, we observe in Fig. 9 that for all code lengths, the proposed NES method in violet outperforms the best Gold code (in gold) and Weil code (in blue) families for both the auto- and cross-correlation objectives. We additionally observe in Fig. 9 that to perform well on the objective function defined in Eq. (17), the algorithm learns to equalize the auto- and cross-correlation objectives, in order to avoid compromising one objective for the other and perform better on the overall maximization objective. Furthermore, Fig. 11 shows the relative performance improvement of our NES method with respect to the well-chosen Gold and Weil codes on the maximization evaluation objective, i.e. max{fAC , fCC }. The ratio of performance improvement with respect to Gold codes is defined as f Gold − f NES ratio of improvement wrt Gold codes := , (20) f Gold (a) 63 bits, 5 codes (b) 127 bits, 5 codes

Figure 8: Training performance on the normalized maximum correlation objective, i.e. max(fAC , fCC ), of the NES machine learning algorithm, with (violet) and without (green) baseline incorporation. We observe that baseline incorporation leads to faster and more consistent learning performance during training. where f Gold represents the performance of a well-chosen Gold code family on the maximization objective and f NES represents the corresponding performance of the NES method for the same family size and bit sequence length. The ratio of performance improvement with respect to the Weil code results is deﬁned similarly. Fig. 11 shows the relative performance improvement of the NES method with respect to the Gold and Weil codes. We observe that for a given sequence length, the relative performance improvement diminishes as the code family size increase, although there is still always a positive improvement in all cases. Comparing with Figs. 9 and 10, we observe that the diminishing relative improvement is accompanied with a saturation in the correlation performance objectives as the number of codes increases for each code length. In particular, as the size of the code family increases, the increase in the correlation performance also decreases. Interestingly, in Fig. 11, we generally observe an increase in the relative performance improvement as the length of the code sequence increases. This pattern could be due to the fact that the family size of the complete set of Gold codes and Weil codes correspondingly increases with the sequence length, and the NES method may have greater freedom to tailor the design of the spreading code family to the desired, smaller number of sequences in the given code family. Indeed, Fig. 9 also demonstrates empirically how designing spreading code sequences for larger family sizes can lead to worse performance on the mean-square auto- and cross-correlation objectives, for all types of code sequences. (a) length-63 codes (b) length-67 codes

(e) length-511 codes (f) length-521 codes

Figure 9: Comparison of the proposed NES machine learning method with that of well-chosen Gold and Weil code families as a function of family size. The normalized auto-correlation components fAC are represented by the dashed lines, while the normalized cross-correlation components fCC are indicated by the solid lines. We observe that for the conducted tests of all spreading code lengths, the NES method outperforms the best Gold and Weil codes in both correlation objectives. (a) length-63 codes (b) length-67 codes

Figure 10: Comparison of the proposed NES machine learning method with that of well-chosen Gold and Weil code families as a function of family size for length-1023 and length-1031 sequences. The normalized auto-correlation components fAC are represented by the dashed lines, while the normalized cross-correlation components fCC are indicated by the solid lines. We observe that for the conducted tests of all spreading code lengths, the NES method outperforms the best Gold and Weil codes in both correlation objectives.

(a) Gold codes (b) Weil codes Figure 11: Ratio of performance improvement of the NES machine learning method with respect to well-chosen families of Gold and Weil codes on the maximization evaluation objective, i.e. max{fAC , fCC }. For each sequence length, we observe that the relative performance improvement diminishes as the code family size increases, although there is still always a positive improvement. Interestingly, we generally observe an increase in the relative improvement as the length of the code sequence increases. 4. Comparison with Analogous Genetic Algorithm + Elitism Finally, we compare our NES method with that of the analogous genetic algorithm. Fig. 12 shows separate plots of the normalized auto-correlation and cross-correlation performance for length-127 and length-257 families of spreading code sequences, while Fig. 13 shows the results for the length-511 and length-521 spreading code families, and Fig. 14 shows the results for the length-1023 and length-1031 spreading code families. Indeed, we observe that our NES method consistently outperforms the GA in the overall maximization evaluation objective across all sequence lengths. Furthermore, in nearly all test cases, the NES method outperforms the GA in both auto- and cross-correlation obejctives. (a) length-127 codes (auto) (b) length-127 codes (cross)

(c) length-257 codes (auto) (d) length-257 codes (cross) Figure 12: Comparison of the proposed NES machine learning method with that of the analogous genetic algorithm with incorporated elitism for sequences of length-127 and 257 bits. We observe that for all spreading code lengths, the NES method performs better than the genetic algorithm implementation in ﬁnding low-correlation spreading codes in the overall maximization test objective. Furthermore, in nearly all test cases, the NES method outperforms the GA method in both the auto- and cross-correlation objectives. (a) length-511 codes, auto (b) length-511 codes, cross

Figure 13: Comparison of the proposed NES machine learning method with that of the analogous genetic algorithm with incorporated elitism for sequences of length-511 and 521 bits. We observe that for all spreading code lengths, the NES method performs better than the genetic algorithm implementation in ﬁnding low-correlation spreading codes in the overall maximization test objective. Furthermore, in nearly all test cases, the NES method outperforms the GA method in both the auto- and cross-correlation objectives. (a) length-1023 codes, auto (b) length-1023 codes, cross

Figure 14: Comparison of the proposed NES machine learning method with that of the analogous genetic algorithm with incorporated elitism for sequences of length-1023 and 1031 bits. We observe that for all spreading code lengths, the NES method performs better than the genetic algorithm implementation in ﬁnding low-correlation spreading codes in the overall maximization test objective. Furthermore, in all of the above test cases, the NES method outperforms the GA method in both the auto- and cross-correlation objectives.

5. Scaling of Network Size with the Size of the Learned Code Family With this proposed method, the total number of parameters in the GNN, and correspondingly the memory required for the network to train, scales quadratically with the length of the total code set to be determined, i.e. K` total binary values for length-` binary sequences and K codes. Indeed, these network parameters are only necessary during the training phase of the algorithm, while the codes are being learned. Once the algorithm converges to a final, optimized spreading code family, the network parameters are no longer needed and can safely be discarded. However, the memory required during training correspondingly scales quadratically with the length of the total code set, thereby requiring greater investments in computing resources. One could reduce memory requirements for training large-scale neural networks by modeling network parameters with lower precision as well as by exploring the use of sparsity in the network model, where some of the network parameters are set to zero and are correspondingly no longer necessary to store in memory. In this work, we used a fully connected network model; however, leveraging sparsity has been shown to significantly reduce the number of network parameters required for a deep networks while achieving similar or even better performance [48] [49]. Another advantage to utilizing sparse models is the corresponding reduction in the number of gradient computations as well as the size of the momentum buffer for the optimizer, thereby also leading to reduced memory requirements during training. V. CONCLUSION In this work, we developed a machine learning framework to devise a family of high-performing spreading code sequences which achieve low mean-square periodic auto- and cross-correlation objectives. We utilize a natural evolution strategy algorithm using a generative neural network to model a Gaussian proposal distribution over the space of binary spreading code families. Using the Monte Carlo-based gradient estimate, we directly optimize the proposal distribution over the design search space. Furthermore, we believe this is the first work to develop a machine learning method to design navigation spreading code families. We further demonstrated the ability of our algorithm to construct higher performing codes than well-chosen conventional codes, including Gold codes and Weil codes, for sequences of up to length-1023 and 1031 bits and family sizes of up to 31 codes. We additionally demonstrated that the spreading code families generated from our method consistently outperformed those obtained from a genetic algorithm with incorporated elitism on the overall objective function, and nearly always outperformed the GA-optimized code families in both components of the objective function.

VI. APPENDIX: BACKGROUND ON NEURAL NETWORKS At a high-level, an artificial neural network is a collection of parameterized linear and nonlinear functions, which altogether represent a complex and flexible mathematical model [50]. The parameters of the network are optimized by the machine learning algorithm, or “learned,” in order to recognize potentially complicated patterns which are difficult to describe within a provided set of data. The key structural component of the neural network architecture is an artificial neuron, which is pictorally represented in Fig. 15.

Figure 15: Depiction of an individual artiﬁcial neuron, the key structural component of an artiﬁcial neural network.

A neuron consists of three main components: (1) weight parameters, (2) a bias parameter, and a (3) nonlinear activation function as shown in Fig. 15. A neuron can have any number of inputs; in Fig. 15, we illustrate a neuron with M inputs. Each input i ∈ {1,...,M} is multiplied by its corresponding weight wi which is a parameter of the neuron, then the weighted inputs are summed and added to the bias term b, which is also a parameter of the network. Finally, the summed term, which is a scalar value for a particular neuron, is input to a nonlinear activation function represented as f. Indeed, the notation f(·) is commonly used in machine learning to represent the nonlinear activation function of a neuron, and in this case should not be confused with the objective function used for optimization. Thus, the explicit expression for the mathematical operations performed by an individual neuron is

M ! X y = f b + xiwi , (21) i=1

th where y is the scalar value output by the neuron and xi is the i input value to the neuron. Nonlinear activation functions have a domain that is continuous and unbounded, i.e. R, and generally a range that is continuous, which may or may not be bounded. Common bounded-output activation functions include the logistic sigmoid function, commonly called a sigmoid function, as deﬁned in Eq. (22), as well as the hyperbolic tangent function deﬁned in Eq. (23): 1 σ(x) = (22) 1 + exp(−x) exp(x) − exp(−x) tanh(x) = . (23) exp(x) + exp(−x)

A commonly utilized unbounded activation function is the rectiﬁed linear unit, commonly called ReLU, deﬁned in Eq. (24): ReLU(x) = max(0, x). (24)

An artiﬁcial neural network consists of layers of “stacked” artiﬁcial neurons, where the output of each layer of neurons acts as the inputs to the next layer of neurons, as depicted in Fig. 16. The input layer is simply represented as a set of constant values and is correspondingly represented as a lighter shade in Fig. 16, while the neurons in the output layer still maintain the complete neural structure of weights, biases, and an activation function as in Fig. 15. Notice that the number of neurons in the output layer is equivalent to the dimension of the desired output from the neural network.

Figure 16: Depiction of an artiﬁcial neural network, consisting of layers of “stacked” artiﬁcial neurons.

The layers of neurons which do not correspond to the input layer or the output layer are called the hidden layers of the network. The number of neurons in each hidden layer can be selected by the designer of the network architecture. Additionally, the neural network is typically fully connected, where each neuron from the previous layer is input to all neurons in the following layer. For more background on neural network architectures or general machine learning concepts, the authors recommend referencing [50] and [51] for a valuable and accessible introduction.

ACKNOWLEDGEMENTS This material is based upon work supported by the Air Force Research Lab (AFRL) at Kirtland Air Force Base, NM under grant number FA9453-20-1-0002. This material is additionally based upon work supported by the National Science Foundation Graduate Research Fellowship under grant number DGE-1656518. Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the Air Force Research Lab or the National Science Foundation.

REFERENCES [1] T. Cozzens. (2020) First GPS III satellite now available. Accessed: 4 September 2020. [Online]. Available: https://www.gpsworld.com/first-gps-iii-satellite-now-available/ [2] A. F. R. L. U.S. Air Force. (2020) Navigation Technology Satellite - 3 (NTS-3. Accessed: 5 September 2020. [Online]. Available: https://afresearchlab.com/technology/space-vehicles/successstories/nts-3#:~: text=Set%20to%20launch%20in%202022,architecture%20for%20satellite%20navigation%20technology [3] D. Chapman, J. Hinks, and J. Anderson, “Way, way out in front – Navigation Technology Satellite-3: The vanguard for space-based PNT,” Inside GNSS, vol. 15, no. 4, 2020. [4] B. W. Parkinson and S. T. Powers, “Part 1: The origins of GPS, and the pioneers who launched the system,” GPS World, 2010, Accessed: 7 September 2020. [Online]. Available: https://www.gpsworld.com/origins-gps-part-1/ [5] S. Erwin, “Air Force experiment NTS-3 could point the way to the next generation of GPS,” SpaceNews, 2019, Accessed: 7 September 2020. [Online]. Available: https://spacenews.com/air-force-experiment-nts-3-could-point-the-way-to-the-next-generation-of-gps/ [6] Department of the Air Force, Air Force Materiel Command, “Navigation Technology Satellite-3,” 2016, Request for Information (RFI–RVKVE-NTS-3). Accessed: 5 September 2020. [Online]. Available: http: //www.fbodaily.com/archive/2016/03-March/24-Mar-2016/FBO-04057628.htm [7] R. Gold, “Optimal binary sequences for spread spectrum multiplexing (corresp.),” IEEE Transactions on information theory, vol. 13, no. 4, pp. 619–621, 1967. [8] Z. Guohua and Z. Quan, “Pseudonoise codes constructed by legendre sequence,” Electronics Letters, vol. 38, no. 8, pp. 376–377, 2002. [9] J. J. Rushanan, “Weil sequences: A family of binary sequences with good correlation properties,” in 2006 IEEE Interna- tional Symposium on Information Theory. IEEE, 2006, pp. 1648–1652. [10] F. van Diggelen, “Who’s your daddy? why GPS will continue to dominate consumer GNSS,” Inside GNSS, March/April 2014. [11] ——, “High-sensitivity gnss,” Position, Navigation, and Timing Technologies in the 21st Century: Integrated Satellite Navigation, Sensor Systems, and Civil Applications, vol. 1, pp. 445–479, 2020. [12] C.-P. Lu, “Ai, native supercomputing and the revival of moore’s law,” APSIPA Transactions on Signal and Information Processing, vol. 6, 2017. [13] M. Stular and S. Tomazic, “Maximum periodic correlation of pseudo-random sequences in cdma,” in 2000 10th Mediter- ranean Electrotechnical Conference. Information Technology and Electrotechnology for the Mediterranean Countries. Proceedings. MeleCon 2000 (Cat. No. 00CH37099), vol. 1. IEEE, 2000, pp. 420–423. [14] L. Welch, “Lower bounds on the maximum cross correlation of signals (corresp.),” IEEE Transactions on Information theory, vol. 20, no. 3, pp. 397–399, 1974. [15] D. Sarwate, “Bounds on crosscorrelation and autocorrelation of sequences (corresp.),” IEEE Transactions on Information Theory, vol. 25, no. 6, pp. 720–724, 1979. [16] V. Sidelnikov, “Cross correlation of sequences,” Probl. Kybern, vol. 24, pp. 15–42, 1971. [17] V. M. Sidelnikov, “On mutual correlation of sequences,” in Doklady Akademii Nauk, vol. 196, no. 3. Russian Academy of Sciences, 1971, pp. 531–534. [18] M. D. Burroughs and S. G. Wilson, “On the performance of 4-phase sequences in asynchronous cdma systems,” in Wireless Personal Communications. Springer, 1996, pp. 171–182. [19] M. Pursley, “Performance evaluation for phase-coded spread-spectrum multiple-access communication-part i: System analysis,” IEEE Transactions on communications, vol. 25, no. 8, pp. 795–799, 1977. [20] M. Stular and S. Tomazic, “Mean periodic correlation of sequences in cdma,” in Proceedings of IEEE Region 10 International Conference on Electrical and Electronic Technology. TENCON 2001 (Cat. No. 01CH37239), vol. 1. IEEE, 2001, pp. 287–290. [21] H. Ganapathy, D. A. Pados, and G. N. Karystinos, “New bounds and optimal binary signature sets—part i: Periodic total squared correlation,” IEEE transactions on communications, vol. 59, no. 4, pp. 1123–1132, 2011. [22] ——, “New bounds and optimal binary signature sets-part ii: Aperiodic total squared correlation,” IEEE transactions on communications, vol. 59, no. 5, pp. 1411–1420, 2011. [23] S. Wallner, J.-A. Avila-Rodriguez, G. W. Hein, and J. J. Rushanan, “Galileo E1 OS and GPS L1C pseudo random noise codes-requirements, generation, optimization and comparison,” in ION GNSS 20th international technical meeting of the satellite division, 2007, pp. 25–28. [24] European Union, “European GNSS (Galileo) Open Service signal-in-space interface control document,” OS SIS ICD, Issue 2.0, Jan 2021. [25] J. Holland, “Adaptation in artificial and natural systems,” Ann Arbor: The University of Michigan Press, p. 232, 1975. [26] J.-A. Avila-Rodriguez, S. Wallner, and G. W. Hein, “How to optimize GNSS signals and codes for indoor positioning,” in Proceedings of the 19th International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS 2006), 2006, pp. 2418–2426. [27] F. Soualle, M. Soellner, S. Wallner, J.-A. Avila-Rodriguez, G. WHein, B. Barnes, T. Pratt, L. Ries, J. Winkel, C. Lemenager et al., “Spreading code selection criteria for the future GNSS Galileo,” in Proceedings of the European navigation conference GNSS, 2005, pp. 19–22. [28] J. O. Winkel, “Spreading codes for a satellite navigation system,” Oct. 11 2011, US Patent 8,035,555. [29] T. Y. Mina and G. X. Gao, “Devising high-performing random spreading code sequences using a multi-objective genetic algorithm,” in Proceedings of the 32nd International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS+ 2019), 2019, pp. 1076–1089. [30] L. Huang, H. Zhang, R. Li, Y. Ge, and J. Wang, “AI coding: Learning to construct error correction codes,” IEEE Transactions on Communications, vol. 68, no. 1, pp. 26–39, 2019. [31] T. Y. Mina and G. X. Gao, “Designing low-correlation GPS spreading codes via a policy gradient reinforcement learning algorithm,” in Proceedings of the Institute of Navigation GNSS+ conference (ION GNSS+ 2020), 2020. [32] S. Wallner, J.-A. Ávila-Rodríguez, and G. W. Hein, “Codes: The PRN family grows again,” Inside GNSS, Septem- ber/October 2011. [33] S. W. Golomb et al., Shift register sequences. Aegean Park Press, 1967. [34] Navstar GPS Joint Program Office, “Interface specification: Navstar GPS space segment/navigation user interface,” IS-GPS-200 Revision L Navstar Global Positioning System, May 2020. [35] J. J. Rushanan, “The spreading and overlay codes for the L1C signal,” Navigation, vol. 54, no. 1, pp. 43–51, 2007. [36] A.-M. Legendre, Essai sur la théorie des nombres. Courcier, 1808. [37] S. Y. Yan, Number theory for computing. Springer Science & Business Media, 2002. [38] I. Rechenberg, “Evolutionsstrategie: Optimierung technischer systeme nach prinzipien der biologischen evolution,” 1973. [39] D. Wierstra, T. Schaul, J. Peters, and J. Schmidhuber, “Natural evolution strategies,” in 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence). IEEE, 2008, pp. 3381–3387. [40] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber, “Natural evolution strategies,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. 949–980, 2014. [41] M. J. Kochenderfer and T. A. Wheeler, Algorithms for optimization. Mit Press, 2019. [42] T. Schaul, T. Glasmachers, and J. Schmidhuber, “High dimensions and heavy tails for natural evolution strategies,” in Proceedings of the 13th annual conference on Genetic and evolutionary computation, 2011, pp. 845–852. [43] P. W. Glynn, “Likelihood ratio gradient estimation for stochastic systems,” Communications of the ACM, vol. 33, no. 10, pp. 75–84, 1990. [44] S. Mohamed, M. Rosca, M. Figurnov, and A. Mnih, “Monte carlo gradient estimation in machine learning,” Journal of Machine Learning Research, vol. 21, no. 132, pp. 1–62, 2020. [45] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [46] G. Syswerda, “Uniform crossover in genetic algorithms,” in Proceedings of the 3rd international conference on genetic algorithms, 1989, pp. 2–9. [47] S. Baluja and R. Caruana, “Removing the genetics from the standard genetic algorithm,” in Machine Learning Proceedings 1995. Elsevier, 1995, pp. 38–46. [48] N. S. Sohoni, C. R. Aberger, M. Leszczynski, J. Zhang, and C. Ré, “Low-memory neural network training: A technical report,” arXiv preprint arXiv:1904.10631, 2019. [49] H. Mostafa and X. Wang, “Parameter efficient training of deep convolutional neural networks by dynamic sparse reparam- eterization,” in International Conference on Machine Learning. PMLR, 2019, pp. 4646–4655. [50] C. M. Bishop, Pattern recognition and machine learning. springer, 2006. [51] Y. Bengio, I. Goodfellow, and A. Courville, Deep learning. MIT press Massachusetts, USA:, 2017, vol. 1.