Sequential Estimation in Robot Vision*

PEER.REVIEWED ARIIGTE

SequentialEstimation in RobotVision*

Armln Gruen and Thomas P. Kersten

Abstract of on-line bundle triangulation is closely tied to tle image Highly time-constrained robot uision applications rcquirc a coordinate measurement process involving a human opera- careful tuning and optimized interaction of a system's hard- tor. The main purposo of this fast sequential estimation is ware components, algorithmic complexity, software engineer- that of blunder detection at an early stage of the measure- ing, and task performance, The high accwacy processing of ment process with the utilization of quick r€measurement possibilities and better blunder control capabilities.Impor- full-ftame image sequencesfor image analysis and object space positioning is very time consuming. In both of tant characteristics of this application are, on ttre one hand, feature ("solution these processes,sequential estimation algorithms offer valua- t}le constaatly varying size of the state vector vec- ble alternatives to simultaneous approaches. This paper in- tor" in least-squaresadjustment terminology) of bundle ad- troduces an efficient estimation algorithm based on Givens justment consisting of tle exterior orientation pararrretersof photographs,the object point parameters,and, possibly, ad- transformations for use in point positioning and updating camera orientation data. In a test, an easy-to-use standard ditional parametprs for self-calibration. On tle other hand, the full covariance matrix of all system parameters is, if at video camera has been applied for image frame generation. The results of camera calibration and an accuracy test using all, only needed at the termination of the process. A third a 3o testfield are presented. The computing times of sequen- distinctive characteristic is the high and typically sparse pat- tial point positioning and camera ofientation arc given and terns of the matrices involved in tle estimation procedure (design matrix of observation equatious and normal equation in part compared to the values for the simultaneous adjustment. This clearly indicates the superior performance of the matrices of least squares).Given ttrese system characteristics, sequential procedurc. a number of sequential estimation algorithms have been compared to each other in the past. First, the Triangular Fac- lntroduction tor Update (rru) algorithm, which updates directly the upper Image sequencesplay an important rolo in photogramme$, triangle of tle reduced normal equations, was fou-ud to per- machine vision, aad robot vision. While in classical photo- form much better than the Kalman form of updating, both in grammetry, especially in aerial applications, data acquisition terms of computing times and storage requirement (Gruen, and processingis largely separated,this is no longer the case 1982; Wyatt, 1982).Later, the Givens transformationswere in modern applications where non-photographicsensor tech- found to be superior, in general, to the TFU (Runge, 1987; nology and digital processing techniquos are employed. Fast Holm, 1989), botl in computational performance and in the methods for data reduction are required, in particular, in ease of mechanization and software implementation. In the highly time-constrained robotics applications, but are also meantime, t}.e Givens algorithm has been implemented in a very often of advantage in less time-critical machine vision number of systems (Edmundson, 1991; Kersten et al., 7992). and digital photogrammetric projects. The classical data re- Already in the mid 80s, Gruen (1985a) envisioned semi-auto- duction processconsists of two major stages:image meas- matic or fully automatic digital real-time triangulation sys- urement and three-dimensional (ao) point positioning. tems for the future. We argue nowadays that machine vision These processcomponents are in general separatedfrom and, in particular, robot vision could draw substantial advan- each other. In each case,simultaneous algorithms can be re- tages from theso sequential approaches. formulated into sequential form for better time perform- This fact has been obviously acknowledgedby the com- ance, puter vision community, where, among others, two recent In image processing, well-known sequential formulations dovelopments are of particular interest. In Matthies ef o1, exist for incremental convolution operations (used in Iinear (1989), the Kalman filter is used to estimate a depth map filtering, resampling,image pyramid generation,etc.); in im- from image sequences.Typical for this approach is that age analysis, they are applied in the pixel location transfor- depth estimates and uncertaintv values are comDuted at each mations in orthophoto production (Baltsaviaset aI.,'J.997) pliel of miniffed (256 by 256 p;ls) video framei and that and in form of the Kalman filter in the tracking of line seg- these estimates are refined incrementally over time. The ments in image space(Deriche and Faugoras,1990). good performance of the Kalman filter is due to the fact that A well known example is that of on-line triangulation only one parameter (depth value) constitutes the state vector using sequential estimation techniques in point positioning and is updated. Errors in orientation and calibration of the with aerial photographs. Here the computational procedure video frdrnes and thus correlations between different elements of the depth map are not considered. Also, only lat- eral motion of ttre CCDcamera is assumed. *Presented at the ISPRSCongress, Commission V, Washington, D.C., Photogrammetric Engineering & Remote Sensing, 2-14 August 1992. Vol. 0r, No. 1, January1995, pp.75-82,

Institute of Geodesy and Photogrammetry, Swiss Federal In- 0099-7772/ 95/6 1 0 1-75$3.00/0 stitute of Technology,ETH-Hoenggerberg, CH-8093, Zurich, 1995 American Society for Photogrammetry Switzerland. and Remote Sensing

PE&RS PEER.REVIEWED ARIICTE

Zhang and Faugeras(1990) use the Kalman update Least-SquaresApproach for Estlmatlon mechanism to track object motion in a sequenceof stereo The Gauss-Markovmodel is the estimation model most frames.They track object features,called ntokens" (line seg- widely used in photogrammetriclinear or linearized estima- ments for example),in eD spacofrom frame to frame and es- tion problems. An observationvector I of dimension n X 1 is timate the motion Darametersof these tokens in a unified functionally related to a u X 1 parameter vector x through way (they also inte-gratea model of motion kinematics). (1) Their statevector consists,thus, of three angular velocity, l-e:Ax three translationalvelocity, and three translational accelera- The design matrix A is an n X u matrixwith n > u and Rank tion parametersof each object,and, in addition, six line seg- (A) : u. There is no need to work with rank-deficientdesign parameters gD ment for each line representation.As before, matrices in on-line triangulation. Rank deficient systems, the tokens are treated independentty here, which allows the causedby missing observations,generally do not allow for a statevector for estimation to remain small in size and may comprehensivemodel check. Observationsshould be accu- result in straightforwardparallelization for any number of to- mulated until the system is regular and can be solved using Kens. standard techniques.For rank deficiency causedby incom- (e.9,, Although we aro aware that very fast a video rate plete datum, seeGruen (rgesa). Sequentialleast-squares esti- of 25 Hz) solutions to tracking problems with affordable mation with pseudo-inversesis very costly (compare computer hardware still require substantialsimplification of Boullion and Odell (1977),p. 50 ffj. The vector e represents the measurementproblem at hand, we neverthelesspresent tho true errors. With the expectation E(e) = 0 and the disper- in this paper the generalsolution for sequentialpoint posi- sion operator D, we get tioning, basedon the bundle solution. The solution is object- point based.Other featuresmay be derived from these gD E(l) = 6*, (2a) point measurements,Any simplification in measurementar- D(l) : Cr : (zoP-t, and (2b) rangement,as for instancenon-moving sensors,may readily D(e) = C""= Ca. (zc) be derived from the generalconcept. Our solution may include self-calibrationparameters for systematicerror model- The estimation of x and oo2is usually attempted as unbiased, ing and statisticaltests for blunder detection.The sequential minimum variance estimation performed by means of least estimation procedure applies Givens transformationsfor up- squares, and results in dating of the upper triangle of the reduced normal equations pammeter vector * : (Ar Pa;-' 4t (3a) (Blais, 1983; Gruen, 1985a).We use Givens transformations residualvectorv = Ai - I, and"t, (3b) as opposedto a Kalman update becausein robotics t}le co- vrPv variance update of the parametervector is not required at variance factor (2o (3c) every stageand then, if at all, only at relatively sparseincre- r ments, Moreover, the varying size of the parametervector The architecture of A is determined by the type of triangula- (addition and deletion of new object points, addition of tion method used. As explained previously, we chosethe frame exterior orientation parameters)leads to very poor bundle method for the pu{pose of generality and rigidity. computational performance of the Kalman filter. For bundle adjustment, Equation 1 can be written as The approachpresented here treats only the so point positioning problem in a sequentialmode. A combination of se- -e:4x+&t-l;P (4a) quential algorithms in zD image measurementand so object point positioning within one unique systemis feasibleand where meaningful if, for instanceMultiphoto GeometricallyCon- x is the vector of object point coordinates; strained (urcc; image matching, which delivers object point t is the vector of orientation elements; coordinatessimultaneouslv. is executedfor full frames in a A, and A, are the associateddesign matrices;and pixel-by-pixel(iconic) mode. AIso, a completebundle solu- e, l, and P are the true error vector, constantvector, and tion with integratedimage matching could be basedon this weight matrix for image point obsewations, respectively, concept. Another generalizationis possible if moving objects x and t are consideredhere as unconstrained(free) paramo- are included in the system. ters. If observations are available for some or all of the object In the second section,a brief description of the Givens point coordinates,a second system of observationequations transformations,as applied to sequentialbundle triangulation is added, that is, with static object points, is given. The third section presents : - a test example using real irnagedata produced with an arbi- Ix l.; P". (4b) trarily moving video cameraover a 3Dtestfield. Similarly, observations for the orientation elements SequentialEstimation inthe Bundle Adjustment with Givens would add Tlansfomations -Br : Ix - l,i P, (4c) In this section we will Dresentin a concisefashion the for- The least-squaresprinciple, applied to Equations4a, 4b, and mulae of Givens transfo^rmationsas applied to sequentiales- 4c leads to t]le combined minimum timation in bundle systems.For a more comprehensive treatment,compare Blais (1983),Gruen (1985a),Runge vrPv + vlP" v" * vrrP,vr+ Min, (4d) (1987),and Holm (1989).In the following, our functional model for the bundle adjustmentwill be set up without the For the purpose of simplicity and without loss of gener- inclusion of parametersfor self-calibration.An extensionby ality, we will operate in the following derivations only with these parameiers is straiqhtforward and does not alter the the reduced minimum principle considerationsand concl"usionspresented here, vrPv + Min,

76 that is, we will consider only Equation 4a as observation eouations.- -€r.=A,rr) *o,,", -r.,'r,", (72) The resulting normal equationsare of the form [X,.,J [l-,1

^ The updated normal equationsof the stagek are of the IOrm ^'[{ : [s;x:] trl : ti;l tcl *hl: , (1s) N*:AT PA' l* :AT Pl [i:] with No:AT PA,, l,:6; P1 (6) No:AIPA,' i+ [X,_,],,n[1,,] N is further assumedto be regular. In an off-line environment,nent, Equation 6 is usually solved bvby applvinsapplying Gausscor and Choleskyfactorization. The former can formallvformally be describedescribed as an LU factorization, decomposing N into a product of + ATr"r4*rlr*r, N= f{* N.' i,:t,"i Iower and upper triangular matrices L and U, i.e., LNIN, lr=ltflF AT 6) P(r) l(k) -[r]: tll (7) 1i1- : lttg+a1 rorpr*rATt*r, N- : Prg,+4*,P,0,Afi",, and or, with L : UrD (D is a diagonal matrix), in the alternate formulation lllo : llto;+llr*rPr*rAlr*r.

The superscripts(0) indicate that, if new parametersx, -""[t]: (B) and t6,,are added, the column/row spacesof the origind No, tll No, and Nn matrices have to be extended by zero vectors and After the reduction of the right hand side, the solution the row spacesof the original vectors and by zero elements vector is computed from accordingly, The updating of the k - 1 stagenormals can be de- '[t]:' '['r] (e) scribed as by back-substitution. (N+^N): (14) In photogrammetrictriangulation, the factorizationis tf] tl i tl] usually done as a stepwise procedure,stopping the reduction The addition of the term AN to the k - 1 normals will of N right before it enterswhat was originally the N,, matrix. result in alterationsof the matrix factors L and U, i.e., This procedure leads to the pre-reducednormals No, i.e., l-:-l : N"i 1", (10) (u+au)lfl = rr*ar1-'tutl with N" - Nr - It$N;$.[o and l": l,-NlN;lI". SequentlalTreatment wlth Glvens Ttanstomatlons Sequentialestimation with orthogonaltransformations using Nn is finally factorized to an upper triangle N"" and f is qR decompositionis describedin Lawson and Hanson obtained by back-substitutionfrom (1,974).Both additions/deletionsof column and row vectors of the A matrix are discussedthere. Householdertransforma- : Nn"i l"r. (11) tions as well as Givens rotations are used. The mechanization of this offline factorization algorithm Blais (1983)recommended the application of Givens ro- takes advantageof the fact that N* is a block-diagonalmatrix tations for the sequential treatrnent of surveying and photo- with 3 by 3 submatricesalong the diagonal,Therefore, the grammetry networks. Our approach uses the estimation reduction of the point coordinatescan be done on a "point model {Equation4a). Instead of obtaining the updated upper by point" basis, leaving the structure of the N* and No ma- triangular matrix (Equation 15) by means of Gauss factoriza- trices unchanged,i.e., producing no new fill-ins in those mation of the normal equations, it applies Givens transforma- trices. This particular feature,based on the structure of N*, tions directly to the upper triangular matrix U of the (Equation is the key to a successfulapplication of the Triangular Factor previous stage.At stagek-t the reduced system 9) Update technique in on-line triangulation. takes the form Assuming a sequentialprocess and interpreting Equation F,'f 4a as the statusof the measurementsystem at stagek-1 of r^r the process,we get the following system if one or more im- 'Ltl='-'Lil: u age coordinate obsewationsare added, including new parameters x1pand t1r,7: Adding one observationequation, including a set of new -e=A,x*Az parametersy, to this systemresults in stagek and gives t-l; P (with P,*1= I) PEER.REVIEWED ARIICTE

GP1,,[A1^,,: ,GP,t",= (27) (16) [H] [$;] Hence, \3 : O./r can easily be derived from ttre lower por- l;:ltil ['l tion d, of the transformed right hand side of the (.k - r)6 in which stage of the systom. The sequential updating of ,0 can be y is the new parameter vector of length p, achioved by simply adding O to d and updating O with Giv- transformations(Gentleman, afu is the row vector with the coefficients of the new ob- ens 1973):i.e., servation equation,and 16,is the right hand side of the new observation equation. (22) Applying a series of orthogonal Givens transformations "[;] "[*] G=G"G,-, . . . G, (n is the total number of systempammeters)(17) [r] [$] For blunder detection, which is particularly important in to Equation 16 results in automatedmeasurement systems, Bairda's data snooping has become standard. The computation of the related test criteria -v,\o",, : \n-p u, w,: (i 1,. . . , ni requires the computation of both )p : (18a)the residual vector v and the diagonal elements of the Q-- Itl matrix. In Gruen (1985a) it has been how )1 shown t}e full Q*- "[g] lal ri matrix can be efEciently computed or updated both with the Givens and the TFU approach. Irr practical operations, we clearly prefer the method of "unit observation vector" ,t. y 'l (Gruen, 1982),because in the processof on-line triangulation Ip= f'I (1Bb)only a relatively small number of selected residuals have to "l*l)1 li.l tt be tested at any given stage.An operational procedure for blunder detection and deletion of gross enoneous observations in the case of non-diagonal-dominant The updated solution vector can be found by back-sub- Q.P-mahices, and if the suspicion stitution into exists that more than one blunder is in the data at a given time, was suggestedin Gruen (1985b). Af- ter each deletion of an observation, the remaining residuals and the estimated variance factor are updated in a recursive =d. (1e) fashion and tested again in order to get rid of the influence '[il of the rejected observation. PracticalExample The sparsity pattems of both U and a[, can be exploited The software package OLTRIS(On-Line Triangulation System; advantageouslyin order to speed-upcomputations. see Kersten et al. (7992) for a description), which was devel- If a covariance matrix of the parameters has to be up- oped for the U.S. oon (Departmentof Defense),is used for dated, essentiallythe sameapproach can be followed as for the computation of the sequential estimation operations in the updated parameters,Another option is to derive it from the practical example of this section. the upper triangle U using Equation 1.4in Gruen (1985a). Figures 1, 2a, and 2b show our 3D laboratory-our testffeld Methods for the deletion of observationsand the addi- which servedas the test object to be imaged by (simu- tion and deletion of parametersare describedin Golub lated) robot, equipped with one cCDcamera. In fact, instead (rsos) and Lawson H"nrotr (Lg74).Some of these meth- of a robot, a "-nd human operator "ff.lmed" an image sequence ods fit nicely into the mechanizationof the Givens approach. with a |VC video camera cR-S778(S-VHS). Deletion of observationscan be handled by introducing these This "amateur" video camera (Figure 3) was intention- observation equations with negative weights into the stan- ally used instead of an industrial ccD c€unera,because of its dard format (Equation r0). Complex arithmetic is avoided in ease of operation (e.g.,viewfinder for optimal object coverage computations. and internal compact video cassettefor immediate data stor' itor the deletion of parameters,one simply cuts out the age of very long image sequences).Table 1 shows some of correspondingcolumns of the upper triangle U and trans- the technical speciffcations of this camera. forms the remaining matrix to upper triangular form with Givens matrices.The transformation vector d is nec- of also lmegeFrame Generatlon essary, The recording "flight" path of the video camera is illustrated The variance factor can be updated either through ex- in Figures 2a and 2b. The sequence of the sp testfield was plicit computation in Equation 3c after tho "new" residuals recorded in two strips, moving the "robot" parallel to the have been determined or through a sequential approach us- testfield at an approximate distance of 3,6 metres from the ing the Givens transformations.Lawson and Hanson (1974, wall. IiVhile recording the images, the auto focus was page 6) have shown that switched off and the camera was focused at in-ffnitv. With f,):(vrPY)trz:d;d, (20) the focal length of the camera fixed at 8.5 mm, the depth-of- field can be assumed sufEcient for sharp imaging of the ob- with d, being derived from the factorization of the observa- ject. Fifty-three secondsofthe sequencehave been chosen tion Equation 4a as for digitization. The imagery was digitized with a VideoPix

7A PEER.REYIETYED ARTICLE

of ccn camerasare describedby Beyer (1"992).The respective softwarehas also been used here' i,uii;s In addition to the test sequence,images were acquired :: gn ::'j for calibrating the fVC. The testfield was imaged from four different camera positions. The pixel coordinates of the test-field targets were determined by least-squarestemplate matching (rsrut), while reference coordinates for tle targets were obtained by theodolite measurements'Measuring some well-distributed points in the four images yielded sufEciently precise approximations for the exterior orientation of the lour imagesby resectionin space,Using these data and the known object point coordinates, approximate image_coo_rdi- nates could be computed. These were used as initial values for automatic Ieast squarestemplate matching of t30 points in each image. In thia test, LSTMwas capable of measuring seven targetsper second including screendisplay with an averagepiecision of 0.33 pm (ri33 pixels) in x and 0.29 pm (1/35 pixels) in y. The observationswere processedin a bundle adjustment Figure1. Three{imensionaltest fieldused for on-linetri- witl self-calibration. The measurements and adjustment were angulation. performed in DEDIP(Development Environment for_DiSitaI - Photogrammetry;Beyer, 19s7),which is a part of the Digital PhotogrammetricStation DIPSIt (Gruen and Beyer, 1990)' The resultJ from the bundle adjustmentand the comparison with framegrabberon a sPARcstation1+ (Sun Microsystems).The check points, as an independent verification of the accuracy, generatedimage frames were pre-processedwith a low-pass are shown in Table 2. Version 1 summarizesthe results of filter (3 by 3 average).The effective size of each digitized im- the calibration with a minimal control datum (three control agewas 72O(H) by 575 (V) pixels. Altogether, 90 image points on the wall). In version 2, eight woll- distributed con' frames were generatod, giving a rate of 1.8 images per second i.ro} points on the wall and test-field frame were used in the of the sequenceor one digitized image every 0.6 seconds. adiuitrnent. The empirical accuracy measures (Vx, Vz), Due to blurring effects caused by image motion, two image Vy, wliich are computed after a spatial similarity transformation frames were left out of the digitized sequence.Figure 4 onto their check-point coordinates, show that an accr:racy of shows three frames of the complete sequence (enhanced with better than one millimetre was obtained. A more accurate a Wallis filter for illustration). The original visual quality of calibration of this video camera with a larger number of im- the frames is not very good. Radiometric and thus geometric age frames and a different camera configuration was per- distortions due to motion blur, analogvideo cassettestorage, formed by Beyer et al. (7992). and frame grabbing with plt line synchronization are visible An accuracy on the order of thoth of the pixel spacing if imaged at larger scales. in image spacecould be achieved.The cameraconstant was determined as c : 8.62 mm, and the pixel spacingas 9.1 pm CameraCallbratlon (H) by 8.3 pm (V). The curve of radial distortion of theJVC Before measuringimage coordinatesand processingdata in is illustrated in Figure 5, The 6.a- by 4.8-mm2sensor of the oLTRis,the video camera was calibrated. In the calibration, is affected by a maximum distortion of. -zt pm at the additional parametersincluding parametersof interior orien- fVC sensorcorners. tation, x-scalefactor, shear,and radial and decenteringdis- tortion were determined. Investigations into the calibration

f-. i.<--- L

fi U I (a)

Figure2. (a) Dimensionsof the test fieldand camerapath (planview).(b) Dimensionsof the testfieldand illustrationof the camerapath usedfor acquiringthe test sequence.

PE&RS PEER.REYIEIYED

.T: (o a f,

Figure3. JVCvideo camera cR-s77E. c.) 3

2(U 0n-LlneTdangulatlon In oLTRIS,the image sequencewas triangulated to demon- (o strate the performance and capability of sequential adjustment for point positioning purposes.As mentioned earlier, = the triangulation was processedwithout self-calibration.The c) c image coordinatesfor the object points in the 88 imageswere (o c determined in a similar fashion as describedabove for the c camera calibration. Known data at the start of the triangula- () tion included the station orientation data of the fust image 0) C (introduced as initial values) and ffve distributed object c) points of the test ffeld which deffned the datum. After in- = cluding a new frame into the triangulation process, at least c) three points have to be measuredto compute the approxi- 0) mate values of the exterior orientation of the "current" camera position. These orientation values of each consecutive c) imale in the sequencewere computed by rcsection in space using the orientation data of the preceding image as initial o values. In each image,between 79 and 146 points were measured.A total of 166 different object points in the test N field were used. In total, 20,860 observations( object and (o 20,362image point coordinates)were processed,with a max- rr, imum number of 1026 unknowns to be determined in the bundle adiustment.The oath of the video camerafor the test /.i r sequenceis plotted in Figure 6. The lower line represents the estimatedpath (i.e., exterior orientation of ttre 88 images) q) as determined by OLTRIS(with sequential estimation and si- f multaneous adjustmentinbetween). The upper line indicates c the "path" as estimatedin a (simultaneous)bundle adjust- a ment with self calibration in DEDIP.The mean of the differ- E encesbetween the two paths is 4.5 mm in the x-direction, 0) 0) .c TABLE1. RELEVANTTECHNTCAL oFrHE JVC VroEo Cauenn 3i:gi;i,*r + c) IVC video camera GR.S77E = .9! r Super VHS Systemfor record and play mode High resolution '/.inch CCD Chip (420,000pixels) Focal length 8.5-68 mm, 8X zoom lens Auto focus Variable electronicshutter 7150, 7125o, 1/500, 1/1000 sec Weight r.z kg PEER.NEYIEWED ARTIGIE

TABLE2. REsuLTsoF THEBUNDLE ADJUSTMENT wrrH SETFCALTBRATToNFoRTHE VrDEo CnvEne CnLteRArtoN Precision from adjustrnent Accuracy from check pointsr

Object space [mm] lmage space [pm] Object space [mm] Irnage space [pm] d Ver Im AP Co ch lpml UY cz c.- Fv Fr lLy 149372581 0.82 0.91 o.82 7.23 o.7 0.4 0,5s 0.s8 0.85 1.5 1.5 2498675980.84 o.29 o.27 o.62 o.7 0.5 0.39 0,31 0.75 0.9 0.7 Ver Im ,.....,....,...Numberof images 4 .,.....,.,.,.,..Standarddeviation ofmeasured image coordinates a posteriori AP ...... Numberof additionalparameters owz ,., ,..., ., ,. .Theoreticalprecision in objectspace Co ...... Numberof controlpoints crn ...... ,.,,....Theoreticalprecision in imagespace ch ...... Numberof checkpoints Vwz ,..,,...,,,..Empiricalaccuracy in objectspace r ...... Redundancv Fry ...... ,..,.....Empiricalaccuracy in imagespace lValues obtained after a spatial similarity fuansformation of all check point coordinates onto their reference values.

?D 2

.9 -20 E

240 E -so -60

E *"0,u,,..1 -

Figure5. Radiallens distortionof the JVCvideo camera. o 10 20 30 40 50 60 70 *",,tJ, **.3"0 60 2726 5178 1246 e431 12056 '4le8 'toK**Jff:3"'*""fff

282 5r0 570 624 6e3 ',7e2 852 nou ",-r?3!_*,jl3u Figure7. cPu-timesfor the inclusionof one additionalim- age point into the sequence(sequential estimation in oL- TR|Son a SPARcstation1+, Sun Microsystems). 1200

1100 4000 6000 6500 including the observations of one additional image point are Z Imm] 400 illustrated in Figure 7. The plotted line shows the increase of CPUtime consumption, depending on the number of frames, 4300 observations, and unknowns, respectively. The computadon time measuredwas between 0.01 secondsper additional im- 1200 age point measurement at the start phase of the triangulation 4100 and 1.54 secondsat the stageof the last frame of tlle sequence.In comparisonwith these results,this speed could 4000 not be achieved with a simultaneous adjustment. Here, the 10 20 30 40 50 50 ?0 8o 90 Number of fmes normal equation system is relinearized, if the updating of the Figure6. Pathof the video camera. solution vector is requested after adding an image point measruement into the nounals. For that, forming and solving the normal equations during the triangulation takes 3 seconds per iteration at the stage of ten introduced image frames,including 2726 obsewations,and approximately 20 -11 mm in the y-direction, and L2 mm in the z-direction. secondsat the stageof 40 frames (9432 observations).This is This difference can be attributed to the absence of systematic factor 70 worse tle sequential mode error compensationin the sequentially estimatedversion. approximately by a than video real-time. The important comparison to be made here betweon the and is far away from two adjustment techniques, simultaneous and sequential, re- lates to their respective computation times (ceu) for updating Concluslons the normal equation system and calculating the solution vec- Our investigations have shown that sequential estimation in tor. In OLTRIS,it is possibleto perform sequentialupdate a general point positioning and camera orientation module with Givens transformations and simultaneous adjustment fbundle adjustment) using Givens transformations can result with Cholesky factorization and back-substitution. Comput- in very short response times for system updating, In our ex- ing times (cpu) for the updating of the solution vector when ample, the insertion of one additional image point required

PE&RS PEER.REVIEWED ARTICTE

0.01 secondsat the stageof the ffrst CCDframe and 1.54 sec- Roots, /ourna.l of the Institute of Mathematical Applications, onds at 88 frames on a Sun sPARcstation1+. The simultane- (72).32s-3s6. ous solution required by a factor 70 higher computing times. Golub, G.H., 1969. Matrix Decompositions and Statistical Calcula- Thus, within this computer environment, an image point in- tions, Stafi'sfical Calculations, (R.C. Milton and J.A. Nelder, edi- sertion (and deletion) at video rate (0.02 sec) can be achieved tors) Academic Press,New York, pp. 365-397, at a svstem size of ten frames. This excellent computational Gruen, A., 7582. An Optim"m Algorithm for On-Line Triangulation, perfoimance makes the procedure of sequential uidating of Int. Arch. of Photogmmmetry, Y ol. 24-III, Helsinki, Finland. bundle systems by Givens transformations particularly usefuI 1985a. Algorit}mic Aspects in On-Line Triangulation, Photo- in time-constrained machine vision and robot vision applica- grammetric En$neering & Remote Sensing,51(4):419-{36, tions. From a systempoint of view, however, object space 1985b. Data ProcessingMethods for Amateur Photographs, feature positioning may be only a minor portion of the over- Photogrammetric Record, 11 (65):567-5 79, - all computing time budget. Becauseimage analysis and image Gruen, A., and H.A. Beyer, 1990, DIPS U Tuming a Standard understanding operations can easily chew up a large amount Computer Workstation into a Digital Photogrammebic Station, Int. Arch. of Photogrammetry, 29(Part nl:247-255, and,Zeit- of computing time, it should be worthwhile to investigate pos- schrifi, P hotogrcmmetrie und F ernerkundun& 199 1 (1 :2-10. sible sequential formulations of related algorithms. filr ) Holm, K.R., 1989. Test of Algorithms for Sequential Adjustment iu As a by-product of our investigationswe could show On-Line Phototriangulanon, Photogrammetria, 43:143-156. that, even wi& an "amateur" TV video camera with an inte- Kersten, Th., A. Gruen, and K.R. Holm, 1992. Point Posi- grated analog storage device, a fairly good accuracy (1/106 of On-Hne tioning with Single Frame Camerc Dofo, DoD Final Tecbnical a pixel from sD check points) can be achieved. We believe Report No. 7, Swiss Federal lnstitute ofTechnology, IGP-Be- that even better accuraciesare possibleif emphasisis put on richt, No. 197, ETH-Zurich, Switzerland. a more sophisticated procedure for systematic error compen- Lawson, Ch. L., and R.|. Hanson, 7974. Solving Least SquaresProb- sation. This opens interestingperspectives for the use of Tv ,lems,Prentice-Hall, Englewood CIiffs, N,j. video camerasin a great variety of measurement applications. Matthies, L.M., R. Szeliski,and T. Kanade,1989. Kalman Filter- based Algorithms for Estimating Depth from Irnage Sequences, Acknowledgment Int. lournal of Computer Vision, 3(3):2O9-238. The software package OLTRISwas developed in the project Runge, A., 1987. The Use of Givens Transfomration in On-Line Trian- "On-line point positioning with single frame cameradata," gulation, Proceedings, ISPRS Conference on "Fast Processing of which was sponsoredby the U.S. Government,Department Photogmmmetic Doto", Iaterlaken, Switzerland, 2-4 fune. of Defense,represented through its European ResearchOfftce Wyatt, 4.H., 7982. On-Line Triangulation - An Algorithmic Ap- of the U.S. Army in London. This support is gratefully ac- proach, Master Thesis, Department of Geodetic Sciences and knowledged. Surveying, The Ohio State University, Columbus, Ohio. Zhang,2., and O. Faugeras,1990. Motion Tracking in a Sequenceof Stereo Frames, Proceedings, 9th European Conference of Artifi- References cial Intelligence, Stockholm, Sweden, August, pp, 747-752. Baltsavias,E,, A. Gruen, and M. Meister, 1991.DOW - A Systemfor (Received17 February1993; accepted 3 August 1993) Generation of Digital Orthophotos from Aerial and SPOT Im- ages,presented paper, ACSM/ASPRSAnnual Convention,25-29 Armin Gruen March, Baltimore, Maryland. Armin Gruen is Professor for Photogrammetry Beyer, H.A., 7987. Einige grundlegende Designfmgen ftr die Ent- (since 1984) and currently Head of the Institute wicUungsumgebung filr Digitale Photogrammetrie auf den Sun of Geodesy and Photogrammefy, ETH Ziirich, Workstations (DEDIP,Development Environment for Digital Pho- Switzerland. He received the Dipl.-Ing. degree togrammetry), intemal report, ETH-Zurich, Switzerland. in Geodetic Science in 1968 and the doctorate 7992. Geometric and Radiometric Analysis of a CCD-Camera degreein Photogrammetry in 1974, both from the Technical Photogrammetric System,Dissertation No. Based Close-Range University Munich, Germany. He has held an appointment as 9701,ETH-Zurich. Associate Professorat the Deparbnent of Geodetic Science,the Beyer,H., Th. Kersten,and A, Streilein, 1992.Metric Accuracy Per- Ohio State University, (1981-1984). Gruen paper presented Columbus, Ohio Dr. formanceof Solid-StateCamera Systems, at the (1984-88), SPIEOE/Technology'92 - Videometricsconference, 15-20 No- has served the ISPRSas Working Group Chairman vember, 1992,Boston, Massachusetts. Published in SPIEVol. President of Commission V (1988-92),and is currently Mem- I82O, Videometrics, ber of Council as SecondVice President(1992-96). Blais, f.A.R., 1983.Linear Least-SquaresComputations Using Givens Transformations, Tie Canadian Suw eyor, 37(a) :225-23 3, Thomas P. Kersten Boullion, T., and P. Odell, 7971. Generalized Inverse Matrices,Wiley Thomas Kersten received the Dipl.-Ing. degree Interscience, Wiley ald Sons, New York. in Geodetic Science in 1988 from t}e Universitv Deriche, R., and O, Faugeras,799o. Tracking Line Segments,Pro- of Hannover, Germany. Since 1989, he has beeri ceedings of the First European Conference of Computer Vision, a research and teaching assistant at the Institute 'ti:s'i'rrrrillvr Antibes, France, April, pp. 259-268. of Geodesy and Photogrammetry, Swiss Federal Edmundson, K.L., 1991. On-Line Triangulation of Sequential Stereo- Institute of Technology, ETH Zurich. From 1989 to 1991, Mr. Pairs Using Givens Transformations without Sguare .Roots,Mas- Kersten did research in on-line triangulation for photogram- ter Thesis,Department of GeodeticSciences and Suweying, The metric point positioning in aerial applications. His current Ohio StateUniversity, Columbus,Ohio. researchinterests include, among others,the use of on-line Gentleman,W.M., 1973,Least SquaresComputation without Square triangulation algorithms in robot vision applications,