Application of Unified Array Calculus to Connect 4-D Spacetime Sensing with String Theory and Relativity∗ Research Article

Journal of Geodetic Science • 3(4) • 2013 • 266-279 DOI: 10.2478/jogs-2013-0032 •

Application of unified array calculus to connect 4-D spacetime sensing with string theory and relativity∗ Research Article

U. A. Rauhala

San Diego, California, USA

Abstract: Array algebra of photogrammetry and geodesy unified multi-linear matrix and tensor operators in an expansion of Gaussian adjustment calculus to general matrix inverses and solutions of inverse problems to find all, or some optimal, parametric solutions that satisfy the available observables. By-products in expanding array and tensor calculus to handle redundant observables resulted in general theories of estimation in mathematical statistics and fast transform technology of signal processing. Their applications in gravity modeling and system automation of multi-ray digital image and terrain matching evolved into fast multi-nonlinear diﬀerential and integral array calculus. Work since 1980’s also uncovered closed-form inverse Taylor and least squares Newton-Raphson-Gauss perturbation solutions of nonlinear systems of equations. Fast nonlinear integral matching of array wavelets enabled an expansion of the bundle adjustment to 4-D stereo imaging and range sensing where real-time stereo sequence and waveform phase matching enabled data-to-info conversion and compression on-board advanced sensors. The resulting unified array calculus of spacetime sensing is applicable in virtually anymath and engineering science, including recent work in spacetime physics. The paper focuses on geometric spacetime reconstruction from its image projections inspired by unified relativity and string theories. The collinear imaging equations of active object space shutter ofspe- cial relativity are expanded to 4-D Lorentz transform. However, regular passive imaging and shutter inside the sensor expands the law of special relativity by a quantum geometric explanation of 4-D photogrammetry. The collinear imaging equations provide common sense explanations to the 10 (and 26) dimensional hyperspace concepts of a purely geometric string theory. The 11-D geometric M-theory is interpreted as a bundle adjustment of spacetime images using 2-D or 5-D membrane observables of image, string and waveform matching in the unified array calculus of applied mathematics.

Keywords: array algebra • extra dimensions • general inverses • geodesy • mathematical statistics • matrix • photogrammetry • quantum string/M-theory • relativity • tensor • uniﬁed calculus © 2013 U. A. Rauhala, licensee Versita Sp. z o. o. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs license, which means that the text may be used for non-commercial purposes, provided credit is given to the author.

Received 29-05-2013; accepted 03-12-2013

1. Introduction ergy/ exploration of Hubble and other space sensors. Many profes- sional societies and journals in mathematics and various branches of engineering overlap the mathematical surveying sciences of We are living in an interesting time period. A retiree like the au- photogrammetry and geodesy using diﬀerent terminologies and thor can follow the latest web news about string theories, quantum concepts that are just some variations of a unified math theory. computing, neutrino speed test, hunt of boson and exoplanets, Similarly, the M-theory of Witten, Susskind and others can explain background radiation, Mars/Moon/black holes/dark matter & en- five string theory variations and quantum relativity in spacetime physics, (Greene 2005), (Kaku 2006). Einstein spent last years of his life in futile attempts of unifying ∗Reprinted with permission from the American Society for Photogrammetry & Remote Sensing, Bethesda, Maryland. the general relativity with quantum theories of particle physics. Journal of Geodetic Science 267

Generations of scientists have continued this work resulting in standing on these matters by the author, which the author hopes five early versions of 10-D string theories that all include apar- to be useful or hopes to provoke some thinking by the next gener- ticle called graviton. M-theory added one more dimension that ation of spacetime scientists. could unify the five early interpretations. Popular TV series, In- ternet and Wikipedia blogs got me interested in the hidden ‘ex- 2. Photogrammetry and geodesy of spacetime geometry tra dimensions’ to model the tiny 1-D strings that are moving as a function of time like image and laser scanners. So, are the six ex- This paper will summarize some basic ideas of array calculus in 4- terior orientation parameters of an image scanner in photogram- D photogrammetry and geodesy of spacetime sensing to fertilize metry somehow related to the extra six space dimensions of parti- the ongoing work also in string/M-theory and advanced math sci- cle physics? M-theory needs one more dimension. So needs pho- ences. Popular books of string theory are predicting that some new togrammetry in the collinear equations of conformal bundle ad- math or observational technology may be required to confirm the justment where the 2-D image coordinates are the observables. string theory, (Greene 2005), (Kaku 2006). It could well be the uni- M-theory also works with 5-brane observables in 11 dimensions in fied array calculus and latest technologies of spacetime sensing. the same duality mode as Independent Model triangulation with Einstein called a pioneer of tensor calculus, Levi-Civita, as the the bundle adjustment. Is the string/M-theory a 4-D formulation of horseman vs. foot soldier in its applications of mathematical photogrammetry, or vice versa? If this is true, they are dual parts of physics. I have been riding with the horsemen of photogramme- a spacetime theory where the proven solutions of spacetime sens- try and geodesy and have spent over 30 years as a foot soldier on ing could translate into spacetime physics (and vice versa) in a uni- factory floor, prototyping new technologies and system concepts fied M-theory of array calculus. of 3-D and 4-D surveying. As a retiree, I am climbing in this pa- This paper of about 15 pages attempts to translate the verbal de- per on an elevated chair above the field of mathematical physics scription of string/M-theories from the referenced popular books overlooking the multiverse of related math sciences. I am going to of thousands of pages into 4-D photogrammetry and geodesy of a interpret the ‘Theory of everything’ (Toe) as the spacetime Math or purely geometric formulation using the universal math language M-Toe problem of unified array calculus in the mixed layman lan- of matrix, tensor, differential and integral calculus. The result- guages of photogrammetry, geodesy, mathematical statistics and ing purely geometric string/M-theory of 4-D photogrammetry and general inverse theory, relativity, quantum and string/M-theories. geodesy applies the expanded matrix and tensor calculus of array The goal of this paper is to alert the various communities of space- algebra in a dual inverse and estimation theory of loop inverses. time sensing to join the physicists to find some practical proofs The resulting unified math of array calculus is applied to interpret of the string/M-theory by exploiting and refining the image and some concepts of quantum and relativity theories based on the range sensing tools of latest technologies. proven technologies and system concepts of spacetime sensing. The starting ideas of array algebra were born as a follow-up to my The early unified matrix and tensor operators enabled fast least 1968 HUT diploma work in Finland about the combined adjust- squares solutions of gravity networks and automated photogram- ment of photogrammetric and geodetic observables of 3-D space metric mapping technologies with millions or even billions of mod- mapping. The combined adjustment idea was prompted by my eling parameters and observables, including extra constraints and field work as a student in urban area and cadastral surveying, to- observables as extra dimensions or border parameters. This work day dominated by the Global Positioning System (GPS) and satel- has evolved into further expansions of the unified matrix and ten- lite (Google Earth, Bing Map, etc.) or high-resolution aerial ortho sor calculus in nonlinear differential and integral calculus. Some image maps and laser/radar/lidar mapping of geographic informa- results of this work will be outlined with only some verbal description systems used, for example, in smart phone, car, boat, airplane tion in the language of spacetime sensing. They appear of inter- and other navigation systems. est in spacetime physics but the detailed work of connecting these High-resolution image mapping of 4-D object space coordinates vast fields is left to you, the reader and future generation of scien- x,y,z,t of all (billions or trillions) points in even a small area (city, tists. country, Earth, Moon, Mars) is challenging as it typically requires an To assist the reader in this future work, I am submitting my personal aerial or satellite imaging mission with thousands of overlapping lines of thoughts, comments and interpretations about the most images from multiple views. The early geodetic surveying tech- controversial issues in modern sciences during the past century. niques of discrete distance, elevation and angle measurements They might be incorrect, too preliminary and unproven or improv- could produce only sparse 2-D or 3-D space locations of station- able in the foreseeable future like most of the theoretical work in ary survey monuments. Aerial and satellite photogrammetry cap- string theory of particle physics since 1960’s. They still may provide tures several image views of the visible 4-D spacetime onto 2-D im- a starting math idea or a ‘thought experiment’ in the translation age coordinates x’,y’. The images are viewed and measured in 3-D from the proven spacetime geometry and adjustment calculus to stereo instruments. Since early 1980’s the era of digital imaging particle physics in a team effort of surveyors, physicists and other and automated image matching improved this process with po- spacetime mathematicians. They represent the personal under- tential future applications also in the micro world of strings. 268 Journal of Geodetic Science

Regular spacetime images are projections of a vast number of 4-D tering) distortion of the optics or by small atmospheric, gravimet- points, features and surfaces on any camera view at image shutter ric and other physical eﬀects. These distortions are usually pre-

time ti. A unique least squares solution of 3-D space coordinates calibrated or correctable in the observables such that the remain- for any given point requires that it is stationary when imaged from ing random errors are close to the unit noise level. They are partially at least two diﬀerent views. A capture of 4-D arbitrarily moving compensated by the adjustable (and therefore slightly distorted) path of points or features requires the use of two or more stereo ten modeling dimensions of each equation in a bundle adjustment cameras with synchronized imaging times. The task is 1) setting of images with nominally zero attitude angles of traditional aerial up the collinear equations of observable image coordinates x’,y’ of cameras. a few discrete tie and control points as a nonlinear function of the Control points, features or other entities (such as point clouds of unknown spacetime coordinates and image orientation parame- known terrain shape and location or orbital GPS and inertial navi- ters, 2) solving the resulting least squares bundle adjustment and gation constraints among the exterior orientation parameters) are 3) then applying the adjusted orientation parameters for 4-D map- known within a weight close to the noise level of observables. They ping of remaining points (including both object and image space define the object space datum (Bianchi identities) or the orienta- times) with the validated quality control of least squares error prop- tion and scale of space coordinates in the Gaussian full-rank esti- agation. mation theory. They together with the unknown tie point coordi- Early computers could digest the bundle adjustment of only very nates are allowed to adjust within their a priori weight of the bun- sparse and manually observed control and tie points to estab- dle adjustment where all n images have the six exterior orientation

lish the six unknown exterior orientation parameters for each im- (sensor space location and attitude angles at exposure time ti) pa- age. Control points (today GPS) of known coordinates are usu- rameters. The m discrete control or tie points have three space co- ally made visible or identifiable for automated measurement on ordinate unknowns each. The object and image space time coor- the images by centering high-contrast targets on them. Natural dinates of these few control and tie points were constrained to be tie points cover stereo overlap areas on well-defined horizontal stationary during standard imaging missions of a block with side- and vertical locations in a sparse grid pattern such as 5×5 or 9×9 overlapping strips of stereo image sequences along the typically for a square image format of the camera. Array calculus could parallel sensor paths. automate the manual tie point mensuration technique by 1) au- Tie points are seen from multiple image views connecting over- tomating the epipolar multi-ray image matching to produce Digi- lapping ‘2-branes with sticky open strings’ in terms of M-theory, tal Surface Models (DSM) in a free local datum or ‘local multiverses (Greene 2005 p.390). They enable a unique unbiased solution of stereo models’ and 2) by automated shape matching of DSM for the inverse problem or 3+6=9-D space reconstruction. The patches to provide fictitious tie point observables for bundle ad- 3m+6n (say, 10,000-100,000) discrete block modeling parameters 2 justment within a unit weight of 1/ (few milli-pixels) . of these dimensions are solved in a least squares bundle adjust- Collinear equations (‘field equations’ of spacetime imaging) ex- ment using 2m¯ n observables x’,y’ of overlapping images where m¯ press the observed image coordinates x’,y’ as nonlinear func- (say, 25-81) is the average number of observed (visible) tie/control tions of the 4-D spacetime coordinates of a discrete object point points per image. The unbiased 4th coordinate or the time diﬀer- j=1,2,...m and as functions of the six exterior orientation param- ence when the light started its travel from any object point j to eters of image i=1,2,...n. The image coordinate observables x’, y’ reach its various (say, 9-15) imaging locations i can be computed are also linear functions of small (curled-up) corrections to interior after the bundle adjustment. The time becomes a valuable extra orientation parameters of the sensor, shared by the equations of observable if the object is a pulsar, GPS satellite, Lidar antenna or – all observables. Assuming the 3-D space locations of all targets to any 1-D vibrating string sending a recordable waveform for phase be stationary and the interior orientation pre-calibrated, we have matching to observe the signal travel time or object-to-sensor dis- 3+6=9 space domain dimensions. They are complemented by the tances. For images taken from only one relative view, like telescope projective dimensionality reduction factor or the focal length of the images of stars taken from Earth orbit, the 3+6+3=12 parameter camera. It converts the 3-D object space and camera parameters collinear transform of image coordinates can be reversed in a con- onto the 2-D focal plane (membrane or 2-brane) coordinates x’,y’. formal solution of a spacetime slice to find the two space domain Modern digital sensors (such as telescopes) are scanners where a directions from the imaging location toward each 3-D point using string or strings of 1-D sensor elements form the 2-brane image the replicated 6 exterior and 3 interior orientation parameters. during the scan time, related to the spin time of particle physics. A stereo pair (overlap of images of two diﬀerent perspective views) In addition to the basic 10 space domain parameters of each without any other control than the local gravity gradients provides collinear equation, we have two ‘hidden’ physical parameters of in- a 4-D local or free net adjustment solution. Neighboring over- terior orientation or the, usually pre-calibrated, principal point to lapping stereo pairs can be interpreted as ‘brane-world’ observ- define the origin of the observed image coordinates. Small addi- ables of colliding parallel universes of local spacetime coordinates tional systematic errors of interior orientation are caused by man- with 4-D shear errors. They were minimized in a traditional Inde- ufacture of the sensor such as the radial and tangential (or decen- pendent Model (IM) block adjustment with 7-parameter similarity Journal of Geodetic Science 269 transforms of all stereo models using leveling (gravity gradient) ob- (of Apollo program ‘moon pictures’) imbedded in a calibrated glass servables to align the object space elevations z with gravity. The plate that were projected on the physical film membrane at the IM math model explains why most living creatures have two eyes, time of image exposure. The observed 2×5×5=50 dx’, dy’ film inner ear gravity sensing and balancing sensors with built-in brain and other distortions can then be corrected at all other measured computer to convert the multi-brane observables of eyes, ears and image point locations by interpolation or filtering. other sensors into local (5-brane) information for survival and mak- The separable 2-D interpolation model of film deformations ing sense of the surrounding spacetime events. prompted an array algebra expansion of matrix and tensor calcu- The starting point in modeling a disjointed stereo pair consists of lus. An important step, leading to the expansion of matrix and ten- 26-dimensional (Ramanujan) space as each image has 13 dimen- sor calculus in inverse problems, was the 1-D (string) formulation sions or 4 local spacetime coordinates, 6 exterior and 3 interior ori- of Gaussian estimation of adjustment calculus using a dual set of p entation parameters. This is chaotic imaging with no tie point ob- modeling parameters L0 in the observable domain and their tradi- servables available between the images and all 12 exterior and 6 tional n > p linear transform (say, biased quantum) domain mod- interior orientation parameters are unknown with no or poor a pri- eling parameters X of function ori weights.

A regular two-eye stereo vision has 4+7=11 dimensions with f(u) = a X = [a1(u), a2(u), . . . an(u)] X 1,n n,1 n,1 known interior orientation of both cameras. Five of their 6+6=12 ∑n ∑n exterior orientation parameters are fixed such that the aligned eyes i−1 2 = ai(u)xi = u xi = 1 · x1 + ux2 + u x3 ... (cameras) get the same y’ image coordinate for stereo fusion of i=1 i=1 a 3-D/4-D vision system. This provides the epipolar geometry of (1) stereo vision where planes through the two imaging locations in- tersect the focal spheres of both eyes along conjugate 1-D epipolar where the row vector a contains i = 1, 2,... n basis functions of strings. Mensuration of the 3-D/4-D point variant model of 12-5=7 variable u, such as polynomials, and column vector X contains the conformal orientation parameters in 11 dimensions is reduced to traditional modeling parameters. the fast 1-D image (string) matching of x-shifts along epipolar lines The unknown values of the alternating dual parameter sets are of a local stereo pair. linked together by a consistent system of linear equations called The human eye-brain computer can register and make sense of all herein as dual ‘parameter exchange transform’ points of the stereo view in real-time - a few million points over 60 times a second. Still, the manual measurement of x’,y’ image coor- A0 X = L0 (2a) dinates of one control or tie point for the bundle adjustment used p,n n,1 p,1 to take seconds per point. One goal in array algebra inventions since 1970’s was the development of new technologies and their where the ‘problem design’ matrix A0 of rank p consists of p row systems integration for automated stereo mensuration and valida- vectors a evaluated at p well-separated variable locations of u such tion by real-time epipolar bundle and IM adjustments. A series of that its rectangular minimum norm m-inverse exists. For example, array algebra expansions of matrix, tensor, diﬀerential and integral these variable locations could provide a fast known exact (say, a calculus provided the new math tools of unified array calculus. priori simulated) solution of Eq. (2a) for true values of vector X us- 3. Multi-linear array calculus of 4-D spacetime ing the true values of vectorL0 by

The basic idea of multi-linear array algebra was born in my KTH f(u) = k L0 = a X , doctoral work of the combined photogrammetric and geodetic ad- 1,p p,1 1,n n,1 m justment theories, reported in (Rauhala 1972-1976). Their testing k = a A0 , 1,n resulted in a grid based self-calibration technique of Hasselblad El n,p m m Data (close range ‘moon camera’ of Apollo missions) as an inte- X = A0 L0 + ( I − A0 A0)U , n,1 n,p p,1 n,n n,n n,1 gral part of the bundle adjustment. This gridded math model then m T T −1 evolved into fast 2-D and 3-D finite element array solutions of grav- A0 = A0 (A0A0 ) . (2b) ity network modeling and automated image matching to feed on- line bundle adjustment as a validation tool of surface models of the Traditional tensor calculus of mathematical physics is more or less mapped object such as the surface layers of the Earth. restricted to the square regular inverse matrix vs. the rectangu-

2-D gridded interpolation parameters were introduced for the lar m-inverse of matrix A0 in Eq. (2b). The square full-rank special (small curled-up) linear corrections of interior orientation, in addi- case p = n has also dominated the classical math sciences. In the ′ ′ tion to the focal length f and principal point x0, y0, found from the general case p < n there is a complete set of biased parameter projection center as the perpendicular intersection with an ideally transform solutionsX with arbitrary vector U of ‘hidden’ dimen- flat focal plane. This camera has a5×5 grid of front-reseau crosses sions behaving like the quantum wave function. All solutions X 270 Journal of Geodetic Science

th of the inverse transform of Eq. (2b) satisfy the consistent forward The 4 row of matrix K expresses the observed angle in Lobs as a

transform of Eq. (2a) into the estimable (uniquely predictable and linear function of the true values L0 of the triangle sides. In se-

observable) parameter set L0 that spans the observable space. quential least squares adjustment, the three observed distances The unbiased full-rank Gaussian estimation is valid for over- (first three elements) of Lobs can be considered as the initial val-

determined m>p observed values Lobs by considering L0 (input ues of L0 and then updated with the redundant angle observable values of exact perturbation solutions) as the unknown but unbi- using the technique of linear condition adjustment, a predecessor asedly estimable set of p basis parameters by replacing X (output of the wavelet theory and Kalman updating by using extra observ- values of exact perturbation solutions) in the linear systems of least ables as extra dimensions or parameters. The adjusted distances squares equations by can then be converted into six adjusted coordinates of three free orientation parameters by selecting two triangle points to form the m adjusted base and intersecting the third point coordinates with the K L0 = Lobs + V ⇔ A X = Lobs + V ,K = A A0 . (3) m,p p,1 m,1 m,1 m,nn,1 m,1 m,1 m,p m,n n,p other two adjusted distances. The resulting triangle provides the same adjusted angle as in the parametric Gaussian adjustment of m L using L in Eq. (4). The design matrix K = AA0 interpolates the m actually observed 0 obs values from the p variable locations of L0 to the m observed loca-

tions of Lobs. Without a loss of generality, the first p observed loca- Let us now experiment with the three possible choices of the scale tions could coincide with the p chosen locations of the true values invariant 2×6 parameter transform matrix A0 for design matrix A

L0 such that the leading p,p horizontal partitioning of matrix K is a of the same four observed values where p < rank(A). One choice unit matrix and the m-p redundant observations are expressed as could use the two unmeasured angles as the basis parameter set

local interpolations from the nearest unknown values of L0. This L0 spanning the angular or scale-free conformal space. The result- holds true also in nonlinear least squares perturbation problems, ing least squares estimates of the two angular parameters are their (Rauhala 2002 a,b). Best Linear Unbiased Estimators (BLUE) of minimum variance. They The least squares solutions of both sets of parameters in Eq. (3) are are the same as in the first example although the resulting Lm- found by inserting the full-rank adjusted least squares solution of inverse does not satisfy the traditional Bjerhammar-Rao definition

L0 into the inverse transform (exact perturbation solution) of X in AGA = A of a general matrix or least squares inverse. The biased Eq. (2b) by parametric solution of X now absorbs also the scale bias. Yet, the projections from any biased hyperspace of these parameters pro-

L L T −1 T duce unbiased estimates in predicting any angular or conformal Lˆ0 = K Lobs,K = (K K ) K , p,1 p,m m,1 subspace of the triangle. The hidden dimensions involving U of ˆ mˆ m Lm Lm Eq. (4) collapse in the same fashion as the quantum wave function X = A0 L0 + (I − A0 A0)U = A Lobs + (I − A A)U Lm m m L T T T T −1 of particle observables because the general estimability condition A = G = A0 (AA0 ) = QA ,Q = A0 (A0A AA0 ) A0. A0 = A0GA, and therefore also A0(I − GA)U = 0, is satis- (4) fied under the Gauss-Markov model, (Rauhala 1976-1981). Raoand Mitra (1971 p. 101) pointed out that Moore originally derived the The Lm-inverse is one class of loop inverse operators that expand pseudo-inverse as a constrained inverse with and without the ex- the theory of general matrix inverses G of the design matrix A of tra parameters (or Nordstrom and Kaluza-Klein extra dimensions). a linear system of equations in Eq. (3). In the limiting case of p = They were expanded in the theory of loop inverses where the Lm- rank( A ) it reduces into the Moore-Penrose pseudo-inverse. m,n inverse employs a combination of the full-rank rectangular L- and Loop inverses explain and expand the general estimation theory m-inverses in the dual loop of estimation processes. of mathematical statistics, potentially unifying the relativity, quantum and string theories, (Rao and Mitra 1971), (Rauhala 1976 pp. Let us proceed to the multi-linear array expansion of the loop in- 81-120, 1981). An example from (Rauhala 1974 p.134) is given in verse and unified math theory. The 2-D interpolation model for (Rauhala 2010) that may have some relevance to a conformal cos- small systematic corrections of the interior orientation (after elim- mology theory, (Penrose 2011): We have measured all three sides ination of the unstable film deformations) was superior over tra- of 2-D triangle with 2×3 = 6 plane coordinate parameters and one ditional calibration techniques while expanding the ‘fast’ (such as of the three angles. The rank of the design matrix is three in a free Fourier) transforms of signal processing and tensor calculus to the net adjustment where the scale of the measured distances is based general estimation theory of loop inverses. It also provided the on an assumed constant speed of light. starting point of expanding the ‘fast’ gravity modeling of physical

A pseudo-inverse choice of the 3x6 parameter transform matrix A0 geodesy where the 2-D fast adjustment problem can be solved by partitions first 3 rows of the 4×6 design matrix A for the 3 observed matrix notations of tensor products. Einstein’s summation conven- distances. The 4×3 design matrix K of the unknown true values tion of indical tensor notations (causing a major restriction of ten-

of distances L0 consists of the 3×3 unit matrix partition as param- sor vs. matrix calculus) was replaced in three and higher dimen- eters L0 span the entire observable space of distances and angles. sions by the flexible contraction rules of matrix calculus in gen- Journal of Geodetic Science 271 eral array solutions of the loop inverse estimation theory, (Rauhala sumed to be zero or the 3×3 rotation matrix is unity in the typical 1972-1976). Lorentz transforms of literature with parallel transported coordi- The industrial R&D applications of array algebra in USA since 1975 nate frames such that the speed of light c remains constant in both started from on-line validation and automated editing of terrain coordinate systems. This is achieved by a common time scale fac- models, used in contouring of topographic maps and produced by tor St in all four scale factors of x,y,z,t by (McConnell 1931, 1957) early correlators (with slow manual edit) of Helava analytical plot- √ 2 2 ters. Array algebra studies in late 1970’s found 2-D and 3-D Singular Sx = St 1 − (u2/c) − (u3/c) = St if u2 = u3 = 0 √ Value Decomposition (SVD) array solutions using Karhunen-Loeve 2 2 Sy = St 1 − (u1/c) − (u3/c) = 1 if u2 = 0 (spin 2 = 4 parallel spin 4 FFTs) and cosine (spin 1 = two parallel √ 2 2 Sz = St 1 − (u1/c) − (u2/c) = 1 if u3 = 0 spin 2 FFTs) transforms for fast finite element solutions. The work √ ′ −1 2 led to 1985-95 techniques of global and entity least squares match- St = rij /rij = cos (α) = 1/ 1 − (u/c) , ing of terrain or feature extraction with automated on-line vali- √ u2 = u2 + u2 + u2, sin(α) = u/c = (r2 − r′ 2)/r . dation. Work since 1995 resulted in system concepts of 4-D pho- 1 2 3 ij ij ij togrammetry and range sensing where the automated software is (5) integrated into the on-board sensor hardware. Distance r = cdt is between the object space location x,y,z Workers in search of concrete string theory observables and their ij (where light starts the travel) and the space location when light adjustment models may want to study the covariance (related to reaches the moving observer (say, camera) after travel time dt. The R and 1/R dualities of M-theory) functions of Bjerhammar, Hardy, ‘relativistic’ distance r′ between these two points of relative move- Hirvonen, Moritz etc. gravity methods and collocation techniques ij ment dx, dy, dz during time dt is called spacetime ‘interval’ of involving least squares network solutions of N-point problem with metric r′ 2 = (cdt)2 − dx2 − dy2 − dz2. This metric is invari- N=millions or billions of finite element nodes or buried point ij ant for both (input/output) coordinate systems of Lorentz trans- masses, (Rauhala 1976). Collocation applies a separable signal form. The relativistic distance coincides with the initial (dt = 0) model for the residual of a low-order physical trend function, such object-to-observer distance when the relative movement vector is as Nordstrom and Einstein field equations for the dark matter and either small or nearly perpendicular to the line of sight between energy, (Ravndal 2004), (Ma and Wang 2012). The global network the points. solution of N-point problem may provide some new insight on One could expect that the special theory of relativity applies to the cosmology concept of dark matter and energy requiring more satellite imaging where the sensor velocity should cause signifi- work. Some innovative analysis is needed to exploit the fast array cant relativistic eﬀects in the fashion of GPS. The literature often solutions, as in the Bjerhammar KTH method and fast ‘corner turn- mentions GPS and gravimetric bending of light passing the sun as ing’ of initially non-separable models, (Bjerhammar 1975), (Rauhala proofs for relativity theory. Traditional quantum theories of par- et al. 1989), (Hotine 1969 p. 323), (Heiskanen and Moritz 1967), ticle physics could not be explained in terms of special relativity (Sjoberg 2013). However, first we need to connect some basic ideas and they did not mix with the gravimetric force of general relativity. of relativity and quantum theories to the general estimation theory However, the verbal description of the quantum string/M-theories of array calculus. in the popular physics books of the references sounds like any el- 4. Lorentz transform and imaging theory of special relativity ementary course of aerial photogrammetry! This physics literature appears unaware or fails to state the main diﬀerence of the metric Lorentz transform was the 1905 starting point of relativity theory special relativity from the quantum theory of conformal imaging: in the thought experiment of train moving along a river bank, (Ein- Image geometry of a bundle of scattered light rays is in- stein, 1916, 1952). The problem is restated in terms of photogram- variant on the speed of light and camera velocity when metry as follows: A satellite with a camera and atomic clock on- the exposure time is controlled by a shutter in a moving board is moving at relative velocity u in orbit around the Earth for camera. The above Lorentz scale transforms are valid for an active densification of GPS world network and 4-D spacetime mapping of event shutter (say, a flash or short burst of photons) at one x,y,z,t every pixel of every image on the digital surface fabric of the visi- spacetime location of the Lorentz input frame toward the camera. ble Earth. Derive and solve for the equations expressing the image They are valid for a GPS receiver of active range sensing – but not in space observables x’,y’ in terms of the spacetime object space loca- typical passive imaging where the shutter is located in the camera! tions of all surface elements and modeling parameters of the cam- Light scattering from space location x,y,z in continuous quan- era. What happens if we do not know the velocity u and imaging tum mode of spherical wave front toward the camera is immune to times? the special relativity in a similar fashion to quantum entanglement. Lorentz transform expands the 3-D conformal (angular) similar- The light keeps scattering from a stationary object point x,y,z of the ity transform to distance based metric line transforms by adding Lorentz input frame in all possible directions, including the “pho- the time dimension and relative velocity vector u of the mov- tonic pipelines” tracking the trajectory of camera lens. The pho- ing origin shifts of two frames. The three rotation angles are as- tons along these rays, ready at any given time to enter the camera, 272 Journal of Geodetic Science

reveal the geodesic line directions perpendicular to the spheri- spacetime sensing techniques of the electromagnetic force? The cal wave front at the intersection point of the sensor path at the strong and weak nuclear forces act on very short pipelines of travel

moment of exposure time ti. These line directions point toward time that this angular imaging mode makes sense in the light of their starting (object space) points j of time coordinate ti −rij /c spin phase matching discussed in section 8. and are invariant of their diﬀerent travel times rij /c at the speed of The passive imaging mode of telescopes and other space sensors light c. Because the photons of all visible points j of the imaged has provided concrete proofs of some cosmic theories based on area already arrived in their quantum state at the lens at the right Einstein’s theory of relativity – although its imaging principle of (geodesic or Fermat’s shortest path) moment ti, it takes no or neg- quantum theory does not obey the starting point, the special the- ligible time to encode their two arrival directions on the focal plane ory, of relativity! It is well-known that an image of multiple dis- coordinates x’, y’ after the shutter opens and their quantum wave tant galaxies today show their shape (state of time events) millions

functions collapse. They appear to start their travel from all station- (rij /c) years ago when the light left them – but who could expect to ary object points j of varying distances rij from the camera location capture the accurate direction measurements to their visible parts point i at the same instant as they form the image. in 4-D spacetime from a single image taken on Earth millions of Collinear imaging equations of ‘4-D quantum photogram- years later at exposure time ti! The two image coordinates and cal- metry’ are equivalent to the photons having an inﬁnite ibrated focal length f are equivalent to two (horizontal and verti- speed of light because they correctly express the direc- cal) theodolite directions toward the points to be mapped by in- tersections from two or more survey stations. Photogrammetry tional equations of image coordinates at exposure time ti when all pipelines of photons point toward their starting exploits these two indirect angle observables of millions of space- points j in the object space. In contrast to quantum entangle- time points in one image of one (camera arrival) time epoch in ment, the information (sensor-to-object directions and scattered Lorentz input domain without necessarily knowing the exact imag- energy intensity) mediated by the pipelined photons of light rays ing times or (other than crude local gravity gradient) orientation have also the apparent speed of infinity under the assumption and location of the camera at that instant. that the energy carried by the photons inside each pipeline starting The six exterior orientation parameters of each image are es-

from object point j remains constant during the travel times rij /c. timable in the view of an observer in Lorentz input space. They This is often the case for most pipelines j where no changes in the are not some invisible and biased Calabi-Yau domain parameters X object space events take place within such a short time, such as or U used today to explain the string theory observables of parti- the death of Schrodinger cat or sun dying within the few minutes cle colliders. Projections of the hidden Calabi-Yau dimensions and before the exposure time. Obstructions may emerge in the line of parameters onto 4-D spacetime get observable in the loop inverse sight causing the well-known illumination bias of gray values to be transition theory of mathematical statistics from biased operators discussed in section 6. Photons arriving at the camera focal plane to the unbiased estimators, (Rauhala 1976 p.108, 1981), (Grafarend

from all directions at the same exposure time ti are captured onto and Schaﬀrin 1974), (Sjoberg 1975). The exterior orientation pa- the 2-D image plane of the 4+6=10-D (or 4+7=11-D including rameters of spacetime sensing have been solved since 1950’s in the focal length f) hyperspace by collapsing their quantum wave the bundle adjustment of image triangulation that has prompted functions and mediating their energy into the observable gray val- some advances in modern calculus, including the unified matrix ues, (Hawking 1988). Their direction cosines (image coordinates and tensor calculus of array algebra and its expansions of diﬀeren- divided by the focal length) point toward their 3-D space locations tial and integral calculus.

at the imaging moment ti within the weight of measured image 5. Bundle adjustment techniques of array calculus coordinates and realizable collinear equations of the sensor orientation parameters. The early history of bundle adjustment and matrix calculus since In photogrammetry, the photons reaching the sensor focal plane 1950’s was summarized by one of its pioneers in (Brown 1974). It and mediating the image geometry and gray values committed a involved the early satellite imaging systems of the Cold War era and false start (with individual start times from the object point j) at enabled the moon control network and image mapping of Apollo some point and time along the imaging rays and are disqualified programs with access to early computers. Efficient mini- and mi- in a distance race of Einstein’s thought experiment. The resulting crocomputers coupled with the software development of fast array collinear geometry provides the image coordinate x’,y’ observables algebra and commercial applications enabled the R&D programs for reconstruction of the 10/11-D hyperspace that was projected of digital photogrammetry since late 1970’s. onto the 2-D image. Some interesting questions are: 1) Are the Array algebra development of automated terrain and feature ex- string/M-theories of particle physics dual to this math model of an traction expanded and integrated the use of bundle adjustment image (related to Susskind hologram)? and 2) Is this passive ‘quan- as an automated validation tool to replace the human operator tum pipeline’ short-cut mode applicable also to the mediator parti- in these tedious and costly editing tasks of image mapping. All cles of other three forces of universe in the fashion of sun’s gravity surface elements of the mapped object space can be considered on the Earth or eye-brain stereo vision, cell phone, TV and other as observables in a hybrid epipolar bundle adjustment related to Journal of Geodetic Science 273 its Independent Model (IM) predecessor of Ackermann and others. • Elimination of the extra point variant parameter sij (shared These techniques may become applicable in ‘particle photogram- by three observables x’,y’,z’ per point and one magnitude metry and full waveform range sensing’ where string matching weight equation) is straight forward in the resulting lin- provides the observables. As a follow-up anecdote to (Helava earized normal equations. This parametric modeling and 1988), simulated examples of 1-D and 2-D periodic string match- elimination appears related to Nordstrom and Kaluza-Klein ing since 1976 as true values of discrete least squares matching res- theories of adding the 5th extra dimension to gravity field cued some funding in my industrial R&D on three occasions. One equations, unifying them with Maxwell equations, (Ravn- of them simulated the explosive ‘big bang’ convergence of global dal 2004). The extra dimensions can also be interpreted finite element solution resulting in the integral perturbation theory as added or deleted observation or constraint equations in of nonlinear array calculus, (Rauhala 2002a). augmented bordering and free net adjustment techniques This paper attempts to call for joint efforts among the various com- of loop inverses, (Brown 1974), (Rauhala 1975). Loop in- munities in spacetime surveying sciences and particle physics to verse partitioning and eliminating singularities expanded find some concrete proofs of the string theory. It cannot detail the block matrix techniques in (Rauhala 1974 pp.59-70). It was past and new bundle and IM math modeling or practical solution further expanded to Cholesky block solution of bundle ad- techniques, other than summarizing some guide lines for a new justment and to multi-dimensional array or tensor decom- work in computational string theory: posing where the eliminated√ parameters are expressed as purely imaginary i = −1 part of observables in terms of • The math model of image observables x’,y’ of collinear the remaining parameters, (Rauhala 1975) equations are found from the pinhole camera geometry by scaling the horizontal space coordinate differences ∆x, ∆y • Block matrix normals of the bundle method are solved by among a camera station i and imaged object point j by the first eliminating some or all exterior orientation parameters implicit scale factor s= f/∆z after the 3×1 column vector as the match results x’,y’ of image observables of a strip are of differences ∆x, ∆y, ∆z is multiplied by the 3×3 rotation ordered per image i of a multi-ray stereo sequence. The matrix of the three exterior orientation angles of the sensor space coordinates of points j and self-calibration parameters of the sensor are left to the last two banded-border • D.C. Brown further scaled all terms in ∆x/∆z, ∆y/∆z by the groups of reduced normal equations because 1) the self-

distance rij of object point j to projection center i thereby calibration parameters need all image observables of the converting the rotated coordinate differences into direc- block or some strips accumulated on the reduced normals tion cosines with proper magnitude or continuity weights, 2) coordinate differences ∆x, ∆y, ∆z can be used as adjustable pa- • I started from the technique of my geodesy professor rameters or direct GPS and/or inertial navigation observ- Hirvonen (1965), the inventor of Finnish bundle method ables to control the sensor path locations and 3) geode- based on his pioneering USA sabbatical research of satel- tic observations among object space points j can be added lite geodesy and photogrammetry in early 1960’s. The fo- without destroying the sparseness of the resulting normal cal length f was treated as an image space observable z’ equations using the ideas of Array Relaxation, Brown’s aug- but with an infinite weight such that the implicit scale fac- mented bordering and the sequential strip or sub-block ad- tor s= f/∆z remained the same as in traditional collinear justment technique of double tie points equations, (Hallert, 1964) • The double tie point technique adds a second set of co- • In early 1970’s the main error source for deformations of the ordinate unknowns with weighted equality or difference adjusted space locations j was found by D.C. Brown to come constraints for tie points on the side-overlap of new image from the lack of flatness of the film and its support platen strips (5/9-branes) that otherwise cause an increase of the causing systematic deviations from an ideal focal plane as- bandwidth of banded-border normal equations. In satellite sumed in nonlinear collinear equations. Array algebra tech- photogrammetry, it may take days, weeks or months be- niques solved the problem 1) by introduction of an explicit fore the observables are available for neighboring strips or

point variant scale parameter sij at every measured im- sub-blocks so it makes sense to adjust one new sub-block age point as the explicit seventh space modeling dimen- at a time thereby resulting in shear errors at points seen on

sion of all images in addition to the image ti variant the overlap of other sub blocks. The past block solutions six exterior orientation parameters but with weaker a pri- are updated as new tie point data get available as if the en- ori magnitude weights for observed z’ than for x’,y’, and 2) tire combined block had been readjusted using all accumu- by data mining two separate sensor variant dx’ and dy’ lated observables, (Mikhail and Helmering 1973). This en- linear self-calibration parameters at a regular grid, such as ables blunder detection as it is more diﬃcult to measure 5×5, for images taken with the same sensor same object points from (say, 9-15) diﬀerent perspective 274 Journal of Geodetic Science

views and time periods due to illumination, seasonal and sights from the camera locations. Photogrammetry is sharing the local scene changes problems of future ‘string sensing of quantum uncertainty and foam obstructions’ between the sensor (say, epipolar ‘string stereo • Finite equality weights in the sequential adjustment cause pair’ employing s-particles to balance quantum jitters in their ob- less deformation and shear errors than tight equality served spin parallaxes of string matching) and the fabric of space- weights. Using only one set of coordinate parameters per time layers that we wish to determine. Aerial and satellite images point j, in the fashion of (Dolloff and Settergren 2010), have clouds, ground clutter (object surface ‘foam’), moving vehi- is equivalent to infinite equality weights in the double- cles and other natural or manmade objects on different relative tie technique. The double-tie technique of finite weights object space locations in images taken only few seconds or min- saves the sparse banded-border Cholesky factorization of utes apart. Buildings, trees and other features above the terrain sub-blocks until new shear errors with neighboring sub- surface obstruct the stereo view (with disturbing scatter and noise) blocks get available for updating. As in the covariance of surrounding terrain surfaces. Only the top portions (roofs, tree computations, only the fast forward/backward reductions tops etc.) of the terrain canopy are visible in two or more consec- of few elements in the right-hand side column vector or utive images from the sensor path. These dissimilar image spots vectors are needed by reusing the stored sparse left-side prevented or reduced the automation success of early 2-ray stereo Cholesky factorization mapping instruments causing costly manual edit. The global finite element solution of the LSM technology in multi-ray imaging re- • The evolution of nonlinear array algebra since late 1980’s solved or reduced this problem. reformulated the bundle adjustment in its hybrid epipo- A self-corrective illumination bias parameter db is added to each 2- lar mode related to IM. The idea started from Meissl’s work D node point to absorb the gray value effect of the local disturbed of differencing GPS sequence observables, (Meissl 1979), scatter before it ruins the nonlinear shift solutions. As discussed (Grejner-Brzezinska 1995). It also resulted in direct nonlin- earlier, the observed reference and slave image gray values result ear solutions by tensor and array techniques of inverse Tay- from the collapsed wave functions of photons mediating the en- lor expansion (Blaha 1994), (Rauhala 1992, 2002a-b). This ergy from an identical object point location j. They should match work introduced short-hand matrix and array notations for at conjugate locations defined by the slowly varying and com- exponential contractions of high order partials with a vec- pact global y-shifts and local x-shifts depending on terrain eleva- tor or matrix in differential array calculus enabling a paper- tions and base-to-height stereo geometry. GLSM determines the and-pencil derivation for the closed-form (vs. iterative) per- global epipolar x-shift (elevation variation) network of slave image turbation solution of inverse Taylor expansion. It evolved to match the conjugate reference gray value locations. The weight into the non-perturbation theory of direct solutions for of local LSM ‘field equation’ in the global network adjustment isthe nonlinear consistent and least squares systems of equa- inverse of local x-shift covariance or the reduced normal equation tions as discussed in more detail in next sections. Their ap- of local x-shift (after elimination of dy and db) scaled by the mini- plications in image and terrain matching of fast integral ar- mized variance of local LSM residuals, (Rauhala 2002a). This weight ray calculus provided the automated observables for a real- gets proportional to the contrast variations of f ’2 in match win- time 4-D epipolar spacetime bundle adjustment, (Rauhala x dows of the reference image and inversely proportional to the local 2010). In future, they may get available also in experimen- match error among them. This reduces the weight of the ‘adverse tal string/M-theory of particle physics. areas’ where large residuals of gray value differences are partially 6. Automated image (2-brane) observables of global least squares absorbed by the change detection bias parameter db. Featureless matching areas of no or small contrast variations get small weights as if they were not measured at all, so we need the global continuity equa- Global Least Squares Matching (GLSM) of Finite Element surface tions of a flexible finite element fabric to fill smooth dx valuesat modeling was enabled by fast array algebra solutions of Karhunen- these nodes without the otherwise costly operator intervention or Loeve transforms and sparse Cholesky network solutions of dif- edit. ferential equations, (Rauhala 1977-1980), (Rauhala et al. 1989). A

dense (typically 2×2−3×3 pixel spacing) 2-D grid L0 of about one Another role of the weighted (Riemann) continuity equations is million match points is allocated on a suitable size reference image the pull-in of x-shifts toward the true surface and realistic weight patch (say, 2×2−4×4 K2 pixels) of gray values f(x,y). The task is to values. The third role is optimal smoothing of the adjusted sur- locate the epipolar shifts dx, dy at these known reference image face model. These goals are met by using a minified image pyra- grid locations x,y to match the slave gray values g(x−dx, y−dy) of mid for the pull-in, with tight continuity weights and large over- one or more stereo slave images using least squares estimation. lapping windows of double node spacing in first iterations of each The images are projections of electromagnetic scatter from 4-D level and then releasing the weights for regular 2×2−3×3 node spacetime locations of an object surface reaching the camera at spacing and less overlapping windows. Three expansions of non-

a given exposure time ti with various obstructions in the line-of- linear estimation theory were found to increase the local pull-in Journal of Geodetic Science 275 range, (Rauhala 2002 a-b, 2010): 1) Improved perturbation theory array) is in extended matrix notations of (Rauhala 2002a) of a large uncertainty basket of initial values makes a simultane- ous merged solution from multiple initial values using Taylor array ∗∗ ′ ′ ′ −1 1/2F”dX 2 + F dX + F0 = 0 ⇔ dXˆ = −(F + Fˆ ) 2F0 expansion of nonlinear normal equations, 2) All high order Taylor m,n,n mn n,1 m,1 array derivatives are exploited resulting in the direct closed-form for m = n = rank(F ′), inverse Taylor solution and 3) Nonlinear robust estimation is ap- Fˆ ′ = F ′ + F”dXˆ ∗∗1 = (F ′T F ′ − 2F T F” )1/2 plied in first iterations of each level using powers for minimized 0 T residuals smaller than the second power of Gaussian least squares by complex Cholesky factorization, ∑ of minimum variance. A priori magnitude weights are increased F”dXˆ ∗∗1 = f”(i, j, k)dxˆ(k), toward high-resolution iterations of relaxed continuity weights to k ∑ ∑ prevent divergence while sharp local elevation or scene illumina- F”dX ∗∗2 = f”(i, j, k)dx(j)dx(k), tion changes are captured. j k High computational speed of 2-D finite element array solution ∑ F T F” = n′(1, j, k) = f”(i, j, k)f (i),F T F” = n′(j, k, 1). is found by two successive 1-D banded Cholesky solutions using 0 0 0 T 1,n,n i n,n modified corner-turning of separable multi-linear transforms. Per- (7) sonal computers already achieved the solution speed of millions of nodes/sec/iteration by late 1990’s, prompting the real-time image These full-rank special cases have several expansions in the ‘multi- data-to-info conversion and compression idea of hardwired signal nonlinear’ diﬀerential and integral array calculus and least squares processors on-board the sensor, (Rauhala 2010). relativity theory discussed at the end of this paper where more de- Kaku (2006) predicts that the final solution of the unified string tails are given for the extended tensor contraction rules of array theory comes from pure mathematics in one and a half inch long calculus in Eq. (7). The above solution of consistent quadratic full- formulae and that its practical proof may come from listening to rank equations avoids the ill-conditioned special case of classical low-octave vibrations of a string. This perhaps can be achieved us- calculus where the scalar f” of the 3-D second order tensor partials ing the earlier discussed string matching techniques. They exploit has to be inverted. The uncovered rule of Eq. (7) is that the con- the local closed-form nonlinear solution of inverse Taylor expan- stant column F is inverse multiplied by the average first derivative sion and connect all (millions or billions) node points into a flexi- 0 matrix of the initial and solution points. The derivatives at the solu- ble finite element fabric of multi-linear continuity equations. The tion points are found by the array square root of Cholesky, (Rauhala new super and hyper iteration techniques of inverse Taylor solu- 1975, 1976), thereby opening the branching possibility of search- tions need no truncation by combining several Newton-Gauss (N- ing for multiple roots of general nonlinear equations. The reader is G) iterations into one. For example, the super iteration of solv- urged to check out the direct column vector solution of Eq. (7) in ing a consistent system of nonlinear equations is found by using the special scalar case m=n=1. two N-G iterations with the same inverse matrix at the point of ex- The closed-form array solutions have avoided discovery since the pansion to get dx and ddx parametric vector corrections, (Rauhala, times of Newton, Gauss and Einstein as the traditional scalar, vec- 2002a-b, 2010). The traditional linear N-G correction dx is comple- tor, matrix and tensor operators are too cumbersome, if applica- mented with the eﬀect ddx of all neglected high order terms. The ble at all, in their derivation even at its simple starting point of the direct solution is found by two more steps with the same inverse of 0 quadratic forward Taylor expansion in Eq. (7). A pioneer in its in- derivative matrix F’1.5 evaluated at x + dx + ddx/2. The correction dical tensor formulation, my former colleague Dr. Georges Blaha, vector ddx+dy is found by multiplying the already computed dis- 0 used some 40-50 pages in deriving the Q-surface solution of non- crepancy vector at F(x +dx) with the inverse matrix of F’1.5. The fi- linear sequential or Kalman least squares estimators involving the nal correction ddy is found by repeating this inverse multiplication 3rd and lower order partial derivatives, (Blaha 1994). The above ar- with the updated vector F(x0+ dx+ddx+dy). This can be written ray equations have evolved in my studies since 1980’s with unlim- in the full-rank special case as: ited Taylor terms until a recent break-through of an ‘Einstein mo- 0 X2 = X + dX + ddX ment in reading the Mind of God’. 0 ′ 0 −1 0 0 = X − F (X ) [F(X ) + F(X + dX)] 7. Integral least squares matching in 4-d photogrammetry

Xˆ =X2 + dY + ddY ⇔ F(Xˆ ) = 0 inverse Taylor solution Many math models of least squares and robust estimation are hy- without truncation brid where some linear terms are added to correct the bias of the (ddX+dY ) + ddY + ... = −F ′(X 0 + dX + ddX/2)−1× observables that often distort the parameters of the nonlinear so- 0 lution. A problem (called fool’s paradise by D.C. Brown) is that [F(X + dX) + F(X2 + dY ) + ...]. (6) adding the linear model causes high correlations and large net- For example, the quadratic tensor solution of a consistent nonlin- work deformations with the linearized parameters of the nonlinear array equation (where the second derivatives form the m,n,n ear model unless both parameter groups are ‘tamed’ using some 276 Journal of Geodetic Science

magnitude, continuity or other constraints, including the stochas- location of moving features embedded on the 3-D fabric of space tic weighting of parameters as observables, (Brown 1974). (say, DSM). An example of the scalar integral least squares solution The discussed expansions to matrix, tensor, diﬀerential and inte- is 1-D line or string matching using partitioned linear and nonlinear gral calculus uncovered the compensation eﬀect of any nonlinear scalar parameters db, dx to perturb the quadratic reference gray function such as LSM with the additive linear bias correction pa- value function to match the quadratic slave gray value function g(x) rameter db, related to the concept of hidden and extra dimensions of physics. This enabled 4-D change detection to determine the

′ 2 1db + f(x + dx) = g(x) + v(x + dx), f(x + dx) = f0 + f (x + dx) + 1/2f”(x + dx) , ∫ ∫ ∫ v(x + dx)2 = min ⇔ v ′(x + dx)T [db + f(x + dx) − g(x)] = f ′(x + dx)T [db + f(x + dx) − g(x)] = 0 [ / ] [ ] ∂[1db + f(x + dx)] 1 f ′(x + dx)T = /∂db = ∂[1db + f(x + dx)] ′ ∗∗ ∂dx f + f”(x + dx) 1 ∫ [ ∫ ] 1 db + f(x + dx) − g(x) = 0 ⇒ f ′(x + dx)T [f(x + dx) − g(x)] = ∫ . (8a) [f ′ + f”(x + dx)][db + f(x + dx) − g(x)] = 0

The resulting two scalar nonlinear integral equations can be writ- low-octave string vibrations, (Kaku 2006). The phase matching of ten using high school calculus for any symmetric match window to these vibrations could provide new techniques of time-keeping, exploit the resulting cancellation of many terms. The linear part of range and velocity (gravity) sensing with extended GPS, radar, the first equation is used to eliminate db in terms of dx. Its substi- laser, lidar, interferometric and other technologies. This opens a tution to the second integral equation reduces to (Rauhala 1992) new aspect of 4-D photogrammetry as the 4th (time, scale, dis- in the well-known scalar special case when the general 3-D m,n,n tance, velocity) Lorentz transform element can be included in the partial second derivative array f” can be inverted in collinear equations to provide, e.g., direct camera-to-object distance observables. Photogrammetry has developed some indus- ′ ′ ′ ′ f”dx = g − f ⇒ dxˆ = (g − f )/f”, trial system concepts of retro-targeting, multi-lens imaging and in- ¯′ ¯ 2 dbˆ = g0 − (f0 + f dxˆ + 1/2f”dxˆ ). (8b) verse (projector) photogrammetry in close and medium range surveying with potential applications in particle physics, (Brown 1974, This integral solution technique is also applicable to LSM and nor- 1994), (Rauhala 2010). malized cross correlation of multiplicative db bias correction as discussed in (Rauhala 2010), (Ruyten 2002) with potential applications of radio astronomy and automated sky survey and monitoring systems. The question for the reader is to think about the general case As mentioned before, the discrete version of matching the phase of multiple elements in linear modeling parameters db and vector of a continuous waveform has served as a robust calibration tool dx of a nonlinear model when f” is the m,n,n array. in development and testing of the discrete GLSM and related ter- Some expansions of traditional matrix, tensor, diﬀerential and in- rain shape or feature based matching technologies since 1970’s. tegral calculus are needed to treat the general case where the pa- Recent work in full waveform matching in range sensing together rameters of the nonlinear model form a vector, matrix or multi- with the ideas of loop inverse string theory prompted an integra- dimensional array (membranes of M-theory) of the system concept tion of the inverse Taylor solution of array calculus with the integral in 4-D photogrammetry, including fast corner turning of nonlinear LSM solution of a periodic waveform, such as the sine function of a Cholesky array or tensor decomposing. The SIAM math community single tone of spinning strings. has published some basic ideas of the early multi-linear array algebra using diﬀerent terms and notations, (Rauhala 2002a), (Kolda and Bader 2009). A periodic shape of two (or a sequence of) vibrating 1-D strings 8. Low octave phase matching of 1-d spinning strings is ideal for integral matching as the LSM bottleneck or iterative re- The SPIE R&D community has reported nanotechnology imaging sampling of the slave function to the estimated (fractional) location 0 of THz rates that may open some practical ways of ‘listening’ to the dx of the reference is solved analytically by Journal of Geodetic Science 277

′ 0 0 f(x + dx) = sin f(x + dx), f (x + dx) = cos f(x + dx), gobs(x − dx ) = sin g(x − dx ), v ′(x + dx)T v(x + dx) = f ′(x + dx)T [sin f(x + dx) − sin g(x − dx0)] ∫ ∫ v(x + dx)2 = min ⇔ cos f(x + dx)T [sin f(x + dx) − sin g(x − dx0)] = 0 ∫ ∫ ⇒ cos f(2dx) sin f(2x) + sin f(2dx) cos(2x) ∫ ∫ − 2 cos f(dx) cos f(x) sin g(x − dx0) + 2 sin f(dx) sin f(x) sin(x − dx0) = 0. (9)

The scalar normal equation for a small shift parameter dx of the reference string f with respect to the latest shift estimate dx0 (used in resampling slave string g) becomes

cos f(dx) ≈ 1, cos f(2dx) = 1 − sin f 2(2dx), sin f(2dx) = 2 sin f(dx) cos(dx) ≈ 2 sin f(dx) ∫ ∫ cos f(x)[sin g(x − dx0) − sin f(x)] + [4 sin f 2(dxˆ0) cos f(x) sin f(x) ≈ 0] ⇒ sin f(dx) = ∫ ∫ . (10) cos f(x)2 + sin f(x)[sin g(x − dx0) − sin f(x)]

A closed-form solution of quadratic normal equations in the fash- example is the full-rank least squares solution of quadratic normal ion of Eq. (6) and (7) by inverse Taylor expansion with m>n discrete equations of over-determined Taylor expansion of Eqs. (6)-(7). It ex- observables Lobs and n parameters of column vector X requires pands the well-known special cases of applied mathematics since 2 a leap from the scalar solution of 1/2f”x +f’x+f0 =lobs and the the times of Newton, Leibniz, Euler, Gauss, Riemann, Ricci, Levi- L truncated N-G least squares iterations dX = F’ dLobs. A simplified Civita and others in the unified matrix and tensor notations

∗∗ ′ ′ ′ ′ T 1/2 −1 0 1/2N”dX 2 + N dX = N0 ⇔ dX + ddX = 2[N + (N N + 2N0 N”T ) ] [N0 + N(X + dX)], ′−1 T ′−1 1/2 −1 ′−1 dX = 2{I + [I + N (2N0 N”T )N ] } N N0, quadratic Newton-Gauss-Raphson Cholesky, ′ T ′−1 −1 ′−1 st ≃ (N + 1/2N0 N”T N ) N0 ≈ N N0, 1 N-R iteration of inverse Taylor for dX,ddX,dY,ddY 0 ′T 0 ′ 0 T 0 N0 = N(X ) = F dLobs, dLobs = Lobs − F0,N(X + dX) = F (X +dX) [Lobs − F(X + dX)], n,1 n,m m,1 m,1 m,1 m,1 ′ ′T ′ T ′T ′T T T N = F F + dLobsF”T ,N” = F F” + (F F”) + dLobsF”T , quadratic normal derivatives. (11) n,n n,m m,n n,n n,n,n n,m m,n,n n,n,n n,n,n

The unified matrix and tensor notations in Eq.(11) deal with multi- plications in global N-point field equations of general relativity in D arrays. A left-side array multiplication by a matrix performs the analogy to the GLSM technology of Section 6 are left for the reader. contraction of the first array index with the second matrix index, Notice that this local least squares gravity term is also aﬀected by replacing it by the first matrix index and the process is repeated the change (f”’) in the acceleration of the motion. Left-side row for all array columns. The lower transpose of an array exchanges multiplication of N” in Eq. (11) results in 1,n,n array and then in the the first index to be the last array index such that e.g. the 1,n,n symmetric n,n matrix using the lower transpose. The upper array array in Eq. (7) becomes an n,n matrix, (Rauhala 2002a). The left- transpose T exchanges the first and second array indices as in ma- T side row multiplication of m,n,n,n array f”’ by dLobs first produces trix notations. 1,n,n,n array and its lower transpose T turns it into the n,n,n 3-D An exponential array post-multiplication of the kth power of vector super symmetric array component of N” of acceleration or gravity dX (or matrix) contracts the last k array indices by the same vector in least squares relativity. Its detailed derivation and potential ap- dX or by the first index of a matrix replacing it by its second index, 278 Journal of Geodetic Science

Rauhala (2002a, 2010), as in N”dX**2 for k=2. The first array index squares adjustment. Manuscripta Geodaetica 19, 199-212. keeps track of the observables. The 2-D, 3-D etc. gridded observ-

ables Lobs require fast nonlinear array solution and compression Brown D.C., 1974, Evolution, application and potential of techniques to make real-time system solution of billions of param- the bundle method of photogrammetric triangulation. Com- eters feasible in 4-D photogrammetry and geodesy. These fast so- mission III ISP Symposium, September 2-6, Stuttgart. lutions may also be needed in the global least squares network expansion of relativity and M-theories of spacetime physics. Brown D.C., 1994, New developments in photogeodesy. PERS Vol. 60, pp. 877-894. 9. Summary and future work

Dolloﬀ J.T. and Settergren R., 2010, Worldview-1 stereo Photogrammetric 4-D spacetime sensing of various layers of im- extraction accuracy with and without MIN processing. ASPRS aged objects was found to expand Einstein’s special theory of rel- Convention Proceedings, March, San Diego. ativity in terms of quantum mechanics. The reason has a common sense explanation of “photonic pipelining” of light rays in passive Einstein A., 1916, 1952 5th Edition, Relativity. Crown Pub- illumination. The photons at the end points of all j pipelines have lishers Inc., New York. already reached the camera location i when the shutter opens and they are pointing toward their starting points j in the object space. Grafarend E. and Schaﬀrin B., 1974, Unbiased free net ad- The light rays therefore appear to have instant (entangled) pho- justment. Survey Review XXII, January. tonic energy transfers from object to image points regardless of

the actual point variant travel time rij /c through each pipeline at Greene B., 2005, The Fabric of the Cosmos. Vintage Books, New the constant speed c of light. The 4-D Lorentz transform of Ein- York. stein’s thought experiment with dual distance observers does not apply to the quantum process of imaging where the collinear math Grejner-Brzezinska D., 1995, Analysis of GPS data process- model absorbs the relativistic effect of sensor velocity and attitude ing techniques: In search of optimized strategy of orbit and angles by the sensor orientation parameters or dimensions. This Earth rotation parameter recovery. Report No 432, Ohio State finding together with the loop inverse theory of estimation and University, Department of Geodetic Science and Surveying, relativity removes a major controversy among the relativity and Columbus, Ohio, USA. quantum theories and explains the hidden extra dimensions of string theory. It may help in translations of the string/M-theories Hallert B., 1964, Fotogrammetri. P.A.Norstedt & Soners of particle physics in terms of their possible duality with 4-D space- Forlag, Stockholm. time sensing. Some steps of expanding the matrix, tensor, differential and integral calculus into a unified array calculus of space- Hawking S., 1988, A Brief History of Time. Bantam Books, time surveying and physics were outlined in a general theory of es- New York. timation and relativity. Its application focused on the 11-D inverse problem of imaging or reconstruction of spacetime from multiple Heiskanen W.A. and Moritz H., 1967. Physical Geodesy. projections of 2-branes using 1-D stereo vision x-shift observables Freeman and Co, San Fransisco. of fast epipolar string matching. The paper calls for joint efforts of physics, imaging, range sensing and math communities to ex- Helava U.V., 1988, Object-space least-squares correlation. pand today’s spacetime sensing techniques into the micro world PERS Vol. 54 No 6, 711-714. of particle physics and 4-D mapping of our solar system and galaxy neighborhoods. This effort needs finding the duality links of par- Hirvonen R.A., 1965, Tasoituslasku. Academy of Technical ticle physics with the outlined math models of a purely geometric Sciences, Helsinki, Finland. string/M-theory. It also calls for the development of advanced sensor technologies using on-board data-to-info conversion and com- Hotine M., 1969, Mathematical Geodesy. Essa 2 Mono- pression in future micro and macro space explorations. gram of USA Department of Interior, Washington D.C.

Kaku M., 2006, Parallel Worlds. Anchor Books, New York. References Kolda T.G. and Bader B.W., 2009, Tensor decompositions Bjerhammar A., 1975, Reflexive prediction. 6th Symposium on and applications. SIAM Review 51 Nr 3, 455-500. Mathematical Geodesy, April, Sienna Italy. Ma T. and Wang S., 2012, Gravitational field equations Blaha G., 1994, Non-iterative approach to nonlinear least- Journal of Geodetic Science 279 and theory of dark matter and dark energy. ArXiv:1206.5078v2 Rauhala U.A., 1980, Development of Array Algebra Algo- [Physics.gen-ph]11 July. rithms for Finite Element Filtering. Final report of DMAAC contract 700-78-C-0022P00002. McConnell A.S., 1931, 1957, Applications of Tensor Analy- sis. Dover Publications, Inc. Rauhala U.A., 1981, Note on general linear estimators and matrix inverses. Manuscripta Geodaetica, Vol 6. Meissl P., 1979, Direct solution of overdetermined alge- braic problems with examples from geometric geodesy. Rauhala U.A., 1992, Nonlinear array algebra in digital pho- Manuscripta Geodaetica 4, 309-358. togrammetry. ISPRS Vol. 29 B2 II.

Mikhail E. and Helmering R., 1973, Recursive methods in Rauhala U.A., 1996, ELSM and GLSR techniques of array photogrammetric data reduction. PERS, 983-989. algebra in shape matching and merge of multiple DEMs. Inter- commission II/III Working Group of ISPRS Congress, Vienna. Penrose R., 2011, Cycles of Time. Alfred A. Knopf, New York. Rauhala U.A., 2002a,b, Array algebra expansion of matrix and tensor calculus: Parts 1 and 2, SIAM J. Matrix Anal. Appl., Rao C.R. and Mitra S.K., 1971, Generalized Inverse of Ma- 21. trices and Its Applications, Wiley. Rauhala U.A., 2010, Array algebra automation of 4-D imaging Rauhala U.A., 1972, Calculus of matrix arrays and general and range sensing. ASPRS Proceedings of San Diego Spring polynomial and harmonic interpolation by least squares with Convention. new solutions in photogrammetry and geodesy. Fotogram- metriska Meddelanden IV:4, Royal Institute of Technology, Rauhala U.A., Davis D. and Baker K., 1989, Automated Division of Photogrammetry, Stockholm, Sweden. DTM validation and progressive sampling algorithm of finite element array relaxation. PERS No 4. Rauhala U.A., 1974, Array algebra with applications in photogrammetry and geodesy. Fotogrammetriska Meddelanden Ravndal F., 2004, Scalar gravitation and extra dimensions. IV:6. arXiv:gr-qc/0405030v1 6 May.

Rauhala U.A., 1975, Calculation of loop inverses. Fotogram- Ruyten W., 2002, Subpixel localization of synthetic refer- metriska Meddelanden 2:38. ences in digital images by use of an augmented template. Optical Engineering 41(3), 601-606. Rauhala U.A., 1976, A review of array algebra. Fotogram- metriska Meddelanden 2:38. Sjoberg L.E., 1975, Are general least squares estimators biased? Royal Institute of Technology, Division of Geodesy, Rauhala U.A., 1977, Array algebra as general base of fast Stockholm, Sweden. transforms. Mitteilungen Der Geodaetischen Institute Der Technischen Universitat Graz, Folge 29. Sjoberg L.E., 2013, The geoid or quasigeoid - which reference surface should be preferred for a national height system? Rauhala U.A., 1979, Intuitive derivation of loop inverses J. Geod. Sci. 3(2), 103-109. and array algebra. Bull. Geodesique 4.