A Statistical Examination of Image Stitching Software Packages for Use with Unmanned Aerial Systems

Home , Image Composite Editor, Orthophoto

A Statistical Examination of Image Stitching Software Packages For Use With Unmanned Aerial Systems

John W. Gross and Benjamin W. Heumann

Abstract There is growing demand for the collection of ultra-high spatial orthomosaic. This issue is typically addressed using aerial resolution imagery, such as is collected using unmanned aerial photogrammetric techniques. systems (UAS). Traditional methods of aerial photogrammetry One of the more conventional ways to handle the creation are often difficult or time consuming to utilize due to the lack of orthomosaics in aerial photogrammetry is through the of sufficiently accurate ancillary information. The goal of this use of automatic aerial triangulation (AAT) and bundle block study was to compare geometric accuracy, visual quality, and adjustment (BBA). In this method, software is able to utilize price of three commonly available mosaicking software pack- interior orientation (IO) information provided by the camera, a ages which offer a highly automated alternative to traditional global positioning system (GPS), and an inertial measurement methods: Photoscan Pro, Pix4D Pro Mapper, and Microsoft Im- unit (IMU) to match individual images together then adjust age Composite Editor (ICE). A total of 223 images with a spatial those blocks of images to match the real world (for a more resolution of 1.26 cm were collected by a UAS along with 70 thorough review of AAT and BBA readers should refer to Wolf ground control points. Microsoft Image Composite Editor had and Dewitt , 2000). The accuracy and quality of these proce- significantly fewer visual errors (Chi Square, p < .001), but it dures are highly dependent on the ability to provide the soft- had the poorest geometric accuracy with a RMSE of 34.7 cm ware with accurate information (Barazzetti et al., 2010; Turner (Tukey-Kramer, p < 0.05). Photoscan had the most visual errors et al., 2012). This is not a problem when using sophisticated (Chi Square, p < 0.001), and a RMSE of 10.9 cm. Pix4D had survey grade metric cameras for which the IOs have been the best geometric accuracy with a RMSE of 7.7 cm, however accurately calculated, and highly accurate GPS and IMU data this was not found to be statistically different from Photoscan which is typically available on manned aircraft platforms. (Tukey-Kramer, p > 0.05). In terms of price, MicrosoftDelivered Image by IngentaDue to the small size and limited power capacity of many Composite Editor was the least expensiveIP: 192.168.39.211 while Pix4D was On: the Fri, commercially24 Sep 2021 available 18:37:04 sUAS, the weight and size of mounted most expensive, althoughCopyright: specific pricing American varies Societydepending for Photogrammetryon sensors, or any and payload, Remote is severelySensing restricted. This often trans- the type of licensing needed. These results suggest that unless lates into the use of consumer grade digital cameras, GPS, and high geometric accuracy or 3D images are required, ICE is the IMU, which typically have insufficient accuracy for conven- best option for most UAS photogrammetric applications. tional orthomosaic techniques (Laliberte et al., 2007; Laliberte et al., 2008; Turner et al., 2012). There can also be significant variability in rotational and angular camera position, degree Introduction of overlap, and illumination between images (Barazzetti et al., Small unmanned aerial systems (sUAS) are a novel comple- 2010). Such errors make the use of conventional photogram- ment to existing image acquisition platforms. Although they metric techniques difficult, and time consuming, to implement. cannot replace satellites and manned aircraft for all applica- Fortunately, recent advancements in the fields of photo- tions, UAS offer three key benefits. They: (a) represent a cost grammetry and computer vision have produced novel tech- effective option for the acquisition of imagery over study niques which offer the potential to not only handle these areas with limited spatial extents, (b) are capable of collecting issues, but to do so in a highly automated fashion. One of the imagery in regions which have traditionally been too danger- most notable techniques is structure from motion (SfM). SfM is ous or delicate for manned aircraft, and (c) fly at drastically of benefit because it does not requirea priori knowledge of any lower altitudes (less than100 m above ground level), which camera parameters or scene information, which complicates can translate into novel fine spatial resolutions on the order of the traditional methods (Choudhary and Narayanan, 2012; 1 cm, which is difficult if not impossible to replicate on any Westoby et al., 2012). SfM utilizes some form of scale invariant other platform (Anderson and Gaston, 2013). feature transform (SIFT) which uses a difference-of-Gaussian sUAS use has been well documented in numerous applica- function to identify “important” features in each image known tions including biological research, precision agriculture, and as keypoints (Lowe, 2004). These keypoints are then matched archeology (Knoth et al., 2013; Verhoeven 2011; Verhoeven in multiple images based on a minimization of Euclidian 2012; Zaman et al., 2011; Zhenkun et al., 2013). This growth distance function (Lindeberg, 2012). By tracking the keypoints in sUAS use will likely continue as government regulations from image to image, SfM is also able to accurately estimate a and safety practices adapt to meet application demands number of external camera parameters such as camera orienta- (Zweig et al., 2015). The use of such miniaturized platforms tion (Westoby et al., 2012). From this combination of infor- and sensors, however, leads to a number of challenges that mation the software is then able to project each pixel into an must be addressed, especially with regards to combining large numbers of images (100+) into a meaningful representa- Photogrammetric Engineering & Remote Sensing tion of the Earth’s surface, a process known as creating an Vol. 82, No. 6, June 2016, pp. 419–425. 0099-1112/16/419–425 Center for Geographic Information Science and Geography © 2016 American Society for Photogrammetry Department, Central Michigan University, Mt. Pleasant, MI and Remote Sensing 48859 ([email protected]). doi: 10.14358/PERS.82.6.419

PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING June 2016 419

05-16 June Peer Reviewed.indd 419 5/19/2016 5:30:00 PM accurate orthomosaic. These keypoints can also be viewed as a Pro, Pix4D, and ICE. The resulting mosaics were then compared dense point cloud which can be converted into a 3D represen- statistically based on geometric accuracy and visual quality. tation of the Earth’s surface, such as a digital surface model. Previous research has utilized a variety of homemade SfM Photoscan Pro scripts to varying degrees of success. For example, Laliberte et Created by the Russian-based company Agisoft in 2010, al. (2008) flew imagery over the Jornada Experimental Range in Photoscan Pro is a 3D model and image stitching software package. It utilizes an adapted form of SfM technology known southern New Mexico and used Autopano pro, a SIFT based software, to generate keypoints. These keypoints were included in a as the SIFT, proposed by Lowe (2004). At its core, this process custom script known as PreSync which also incorporated a 1 m uses feature points, which are simply geometrically similar digital orthoquad and a 10 m digital elevation model to adjust and distinct regions in an image, e.g., building corners or the top of light posts. These points are then tracked across and improve existing external orientation parameters (EOs). The multiple images creating a series of connections between original images as well as the updated EOs were then put into Leica Photogrammetric Suite to generate the mosaic. They were photographs (Verhoeven, 2011). In addition, this algorithm allows Photoscan Pro to automatically and accurately estimate able to obtain an overall RMSE of 47.9 cm which in part was due a large number of internal and external camera parameters to a lack of differential correction on their GPS unit. Turner et al. which previously had to be known and entered manually. (2012) created an automated technique using SIFT and SfM techniques to automate the mosaicking of imagery collected over Utilizing such algorithms, software packages such as Pho- two sites of an Antarctic moss bed. They were able to achieve toscan Pro are capable of matching images at the subpixel mean absolute total errors ranging from .103 m to 1.247 m. level (Woodget et al., 2014). In addition, Photoscan Pro is capable of utilizing GCP information directly in the program to A number of commercial SfM software packages have become available since 2010. Two of the most popular include increase geometric accuracy. Photoscan Professional (Photoscan Pro), and Pix4D Mapper The Photoscan Pro workflow can be broken down into four Pro (Pix4D). Both of these software have been successfully basic steps: image alignment, dense point cloud formation, used in current research (Vallet et al., 2011; Kung et al., mesh creation, and texture creation. Each of these steps is 2011a; Kung et al., 2011b; Verhoeven, 2011; Verhoeven et al., run independent of each other, and aside from the inclusion 2012; Woodget et al., 2014). of ground control points, they can be run with little to no The goal of this research was to compare commercially avail- user input. It is also important to note that these stages are all independent and can be saved separately for later use or able software packages for use with imagery acquired by sUAS. Specifically, this paper examines geometric accuracy, visual revision. All four steps can be run in a batch process, if the quality, and price as important factors for selecting software to parameters are known in advance. The use of batch processing decreases processing time, as well as, allows the software process digital aerial photographs from sUAS. Two commercially to be processed overnight without requiring user intervention. available SfM software packages, Photoscan and Pix4D were compared, as well as the freely available image stitching software Im- There are a number of parameters that must be defined, often with multiple options for each parameter. These parameters age Composite Editor, from the Microsoft Research Group.Delivered Previ- by Ingenta ous studies have primarily focused solely on geometric accuracy give the user control over a number of crucial factors that affect IP: 192.168.39.211 On: Fri,the overall24 Sep quality 2021 18:37:04of the final output, such as: the maximum in two or three dimensions (SonaCopyright: et al., 2014; American Turner etSociety al., 2014) for Photogrammetry and Remote Sensing with Photoscan Pro producing the best results. However, these number of tie-points to include in the point cloud, what type studies did not consider the impact of processing on image qual- of surface the imagery consists of, and how to handle any gaps ity which is important to photogrammetric applications. in the final model. Trial and error in combination with a care- ful review of the user’s manual was used to parameterize each step. A complete list of the parameters can be seen in Table 1.

Data Table 1. Parameters for Photoscan Pro A total of 969 images were collected on 26 September 2014 at Braeburn Marsh Preserve near Ann Arbor, Michigan, (~42° 16′ N, Step Parameter Setting ~84° 4′ W). Imagery was collected using a Cannon EOS 6D digital Accuracy High single lens reflex camera with a 50 mm fixed focal length lens at a Align photos Pair preselection Disabled flight height of 100 m above ground level. The camera was mount- Point limit 40000 ed on a Leptron Avenger airframe (Leptron, Golden, Colorado). Build dense Quality Low After the removal of low quality images, redundant images, and point cloud Depth filtering Mild turns in the flight lines (which reduce the accuracy of the final Surface type Height field product), only 223 images were retained for analysis, covering ap- Source data Dense cloud Build mesh proximately 6.3 hectares with a spatial resolution of 1.26 cm. Due Polygon count Medium to the low accuracy of the camera GPS (25 m), no GPS exchange- Interpolation Enabled able image file format EXIF( ) data was retained for the analysis. Mapping mode Adaptive orthophoto Ground control points (GCP) were established prior in the Build texture Blending mode Mosaic field season using a R8 GNSSRTK rover and a TCS3 hand held Texture size 4096 unit (Trimble Navigation Limited, Sunnyvale, California) with a recorded horizontal accuracy of <2 cm (95 percent confidence interval). Each point was marked with a metal pole and colored Pix4D Pix4D is an alternative orthomosaic software created in 2011 foam for easy recognition in the imagery. A total of 70 points by a Swiss company of the same name. The Pix4D workflow were collected. These ground control points align with field consists of three steps: initial processing, point cloud den- plots set up previously for other studies (unpublished data). sification, and DSM and orthomosaic generation. The user defined properties which guide the quality, accuracy, and Methods format of the final output are all handled through a processing options dialogue box which must be set up prior to any Overview processing steps. The options in this box are refined into five The 223 images and 53 GCPs were used to create an image mo- sections: initial processing, point cloud, DSM orthomosaic, saic; the remaining 17 GCP were used for validation The image additional outputs, and resources. A complete list of the pa- mosaic was created using all three software packages: Photoscan rameters can be seen in Table 2.

420 June 2016 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

05-16 June Peer Reviewed.indd 420 5/19/2016 5:30:01 PM able arameters for ix T 2. P P 4D n 22 XX− +−YY Step Parameter Setting ∑ i =1 ()( ii,,measured ) ( iimeasured ) (1) Processing Aerial nadir n Feature extraction 1 Initial processing Optimization Externals and all internals where n is the total number of samples, Xi and Yi are the X Output None selected and Y measurements of point i’s location in the image, and X and Y are the position of GCP i as measured in the Image scale 1/4 (Multiscale on) i,measured i,measured Point density Optimal field. To determine statistical significance a one way stacked Point cloud Minimum matches 3 analysis of variance (ANOVA) was conducted in MINITAB (ver- Point cloud filters None sion 14) with α = 0.05. ANOVA uses the variances of the variable provided for each group to determine if there are differences Noise filtering On DSM/orthomosaic Source data On between the groups. ANOVA was chosen because it is one of Polygon count Geotiff the most powerful statistical tests when comparing multiple groups and examining difference. It was also chosen because Resources Resources All avalible it is not overly sensitive to deviations from normality, and is more powerful than the Kruskal-Wallis test (McDonald, 2009). Table 3. Parameters for ICE The alpha value is the cutoff for determining significance, Step Parameter Setting although somewhat arbitrary, a value of .05 is typically used in scientific literature. Contingent on the results of theANOVA Panorama type Simple Import post hoc Tukey-Kramer test were also conducted for each Camera motion Auto-detect(planar) pairwise comparison. The Tukey Kramer tests compare each Stitch Roll 0 Degrees group pairwise to determine specific differences among all the Scale 100 groups. Tukey Kramer tests are one of the most commonly used Export File format JPEG comparison tests used with ANOVA. Tukey Kramer tests only Quality 100/High need to be done if the results from the ANOVA are significant. Visual Quality Assessment Before any processing is done or parameters are set, Pix4D To quantify the visual quality of the images, 100 points were highly recommends the inclusion of any GCP data that may be created using the “create random points” tool in ArcGIS available. This data was especially important for this research 10.2.2. A 2.5 m (5 m diameter) circular buffer was created since no GPS data was included in the Exchangeable Image around each point (Figure 1). File Format (EXIF) data for the images. GCP data also helps These locations were then assigned to one of four land reduce shift problems and other errors which may be present cover classes based on visual interpretation: reed canary grass in the final model. Each of theGCP s was located in five of the images to help ensure Delivered by Ingenta image quality. IP: 192.168.39.211 On: Fri, 24 Sep 2021 18:37:04 Copyright: American Society for Photogrammetry and Remote Sensing Image Composite Editor (ICE) ICE is an advanced panoramic image stitching freeware produced by Microsoft, and is typically used to create detailed panoramas from numerous individual images. Unlike SfM software packages, ICE does not create a separate 3D point cloud and cannot generate 3D images. For these reasons, it does not create the same outputs as Photoscan and Pix4D (i.e., no dense point clouds or meshes).Therefore, the number of steps and parameters are relatively few. In addition ICE currently has no way to include GCPs in the program; this must be completed using a separate program such as ArcGIS® or ENVI. ICE is run in four sequential steps: import, stitch, crop (optional), and export. A complete list of the parameters can be seen in Table 3.

Geometric Accuracy Assessment To quantify geometric accuracy, root mean square error (RMSE) was used. RMSE is the comparison of real world (ground truth) information to estimated (image derived) measurements. In the case of this research 17 GCPs (25 percent of the total GCPs) had their real world location compared to their estimated location in all orthomosaics. RMSE calculations were conducted using the following equation: Figure 1. Examples and distribution of the buffers used for validation of visual quality.

Photogrammetric Engineering & Remote Sensing June 2016 421

05-16 June Peer Reviewed.indd 421 5/16/2016 10:28:43 AM Figure 2. A portion of the study area enlarged to show visual quality issues. The left image, taken from ICE is shown for comparison. The image in the center, taken from pix4D, shows image blur. The right image, taken from Photoscan, shows image artifacts.

differences between two or more groups (i.e., more occurrences of image artifacts in one group than another). For this study the commonly applied alpha of .05 was used. In cases where Chi Square tests could not be used (typically when the number of observations per group is low) fisher’s exact tests were used. In this study, comparisons between vegetation classes for each software (i.e., Pix4D WV versus Pix4D TM) had sufficiently low numbers of observations to warrant the use of fisher’s exact tests. In order to deal with the issue of multiple comparisons (the increased likelihood of false positives due to chance when conducting multiple tests) Bonferroni Delivered by Ingenta corrections were applied as necessary IP: 192.168.39.211 On: Fri, 24 Sep 2021 18:37:04(Heumann, In Press; see McDonald, Copyright: American Society for Photogrammetry and Remote2014 for Sensinga more detailed discussion).

Results and Discussion Geometric Accuracy The results of this analysis can be viewed in Figure 3 and Table 4. Overall, ICE produced the least geometrically accurate image with an RMSE of 34.7 cm compared to Pix4D and Photoscan Pro (Tukey test p <0.05). Pix4D produced the lowest RMSE of 7.7 cm, however, this value was not found significantly different from Photoscan Pro’s RMSE of 10.9 Figure 3. Geometric error in the x and y direction for the 17 validation points. cm (Tukey test p >0.05) (Table 4). These results are readily apparent when the x Table 4. Tukey Test Results for Geometric Accuracy Assessment; and y errors are viewed graphically, as ANOVA p value = 0.001; Bold Values Indicate Significance ICE has a greater spread of errors in both the x and the y axis Lower boundary Center Upper boundary compared to Pix4D and Photoscan Pro which are clustered around the origin (Figure 3). Although they were not signifi- ICE vs Photoscan -0.187 -0.108 -0.030 cantly different, it should be noted that some of the increased ICE vs Pix4D -0.193 -0.114 -0.036 error of Photoscan Pro relative to Pix4D was likely due to the Pix4D vs Photoscan -0.084 -0.006 0.072 increased presence of image artifacts which were present in a number of the GCPs in Photoscan Pro (see the Visual Qual- (RCG), typha marsh (TM), wet-mesic prairie (WMP), or woody ity Section). Their presence made it more difficult to accu- vegetation (WV). A binary classifier (1 or 0) was used to denote rately determine the geometric center of the GCPs leading to the presence of image artifacts and image blur. Image artifacts decreased accuracy. It is also important to note that the goal are regions of the image where pixels have been misaligned of this study was to look at the products exactly as they were and result in distinct visible errors. Image blur occurs when created. Photoscan Pro and Pix4D are Orthomosaics, and have the quality of the output has a blurred appearance (Figure 2). already taken into account (at least to some degree) terrain Chi Square tests for independence were calculated between relief. No such corrections were applied to ICE, which may image groups (i.e., Pix4D versus ICE). Chi Square tests allow also help explain it increased geometric error. for the comparison of binary data to determine if there are

422 June 2016 Photogrammetric Engineering & Remote Sensing

05-16 June Peer Reviewed.indd 422 5/16/2016 10:28:43 AM Table 5. P Values for Each Class in Terms of Image Artifacts Compared between the Three Mosaics. Value in Parenthesis is the Target p Value for the Column after Bonferroni Corrections Bold Values are Significant. * Indicates Potential False Negative. Class ICE vs Photoscan vsPix4D (p=0.050) ICE vs Photoscan (p=0.017) ICE vs Pix4D (p=0.017) Photoscan vs Pix4D (p=0.017) Overall < 0.001 < 0.001 < 0.001 < 0.001 Reed canary grass < 0.001 < 0.001 < 0.001 < 0.001 Wet-mesic prairie < 0.001 < 0.001 0.306 0.001 Typha marsh < 0.001 < 0.001 0.019* 0.001 Woodland Vegetation < 0.001 < 0.001 < 0.001 < 0.001 Table 6. P Values for Each Class in Terms of Image Blur Compared between the Three Mosaics. Value in Parenthesis is the Target p Value for the Column after Bonferroni Corrections; Bold Values are Significant. NA Represents Comparisons that were not Calculated because Overall Comparison was not Significant or did not have Enough Data. Class ICE vs Photoscan vs Pix4D (p=0.050) ICE vs Photoscan (p=0.017) ICE vs Pix4D (p=0.017) Photoscan vs Pix4D (p=0.017) Overall < 0.001 0.259 0.005 < 0.001 Reed canary grass 0.040 0.040 0.585 0.011 Wet-mesic prairie 0.192 NA NA NA Typha marsh 0.081 NA NA NA Woodland Vegetation 0.001 0.397 0.001 0.007 Table 7. P Values for Each Class Comparison in Terms of Image Blur. Bold Values are Significant. * Indicates Potential False Negative. NA Represents Data that was not Calculated because Overall Comparison was not Significant. Class ICE Pix4D Photoscan Overall 0.065 < 0.001 < 0.001 Canary reed grass vs Wet-mesic prairie NA 0.399 1.000 Canary reed grass vs Typha marsh NA 0.550 1.000 Canary reed grass vs Woodland vegetation NA < 0.001 0.001 Wet-mesic prairie vs Typha marsh NA 1.000 1.000 Wet-mesic prairie vs Woodland vegetation NA 0.003 0.017* Typha marsh vs Woodland vegetation NA < 0.001 0.002

Delivered by Ingenta IP: 192.168.39.211 On: Fri, 24 Sep 2021 18:37:04 Copyright: American Society for Photogrammetry and Remote Sensing

Figure 4. Percent of each class with image artifacts. Figure 5. Percent of each class with image blur.

The accuracies achieved by Photoscan Pro and Pix4D are and 24.71 cm for Photoscan Pro and Pix4D, respectively, for comparable to values reported in previous literature using imagery acquired over a lettuce farm in Australia. It should be sUAS imagery. For example a study conducted in 2010 by noted that the all geometric errors are less than typical uncer- Laliberte et al (2010) found RMSE values of 11.95 cm, 20.17 tainty from handheld differential GPS. cm, and 16.69 cm for images collected over three study sites in southwestern Idaho utilizing an in-house software. Turner Visual Quality et al., 2012 reported mean absolute total errors of 11.49 cm The results of this analysis are shown in Figures 4 and 5 and Tables 5, 6, and 7. In terms of image artifacts, Photoscan Pro

Photogrammetric Engineering & Remote Sensing June 2016 423

05-16 June Peer Reviewed.indd 423 5/16/2016 10:28:43 AM had the most overall image artifacts with 89 percent of the Pricing sites showing some degree of artifact (Figure 4). This was sig- The pricing of software licensing ranged greatly from free nificantly more than Pix4D orICE with 29 percent and 0 per- to $8,700 USD. Note all prices are in US dollars and were cent, respectively, (Chi Square p <0.001) (Table 5). Pix4D also obtained using public websites in June 2015. Microsoft’s ICE produced significantly more than ICE (Chi Square p <0.001). It is freely available, which is appealing to both business and is likely that the cause for the lack of image artifacts produced educational users (http://research.microsoft.com/en-us/um/ by ICE is due to the different approach its algorithms take redmond/projects/ice/). However, Microsoft does not provide to create the mosaic; ICE maintains the original image pix- any formal support for this software, although the developers els rather than blending pixels from multiple images at the of this software do participate in the support forum. Agisoft’s expense of geometric accuracy and occasionally consistency Photoscan Pro is $3,499 USD for a stand-alone license with between images. education pricing available ($549 USD for a stand-alone When comparing the amount of image artifacts between license) and includes 12-months of e-mail support (http:// the software packages with respect to different vegetation www.agisoft.com/buy/online-store/). Photoscan Pro’s licenses types on the ground, Photoscan Pro was found to produce sig- are perpetual without time-limits. Pix4d Pro Mapper is the nificantly more artifacts in every vegetation category (Fisher’s most expensive of the software packages. A perpetual license exact p <0.001 for all comparisons) (Table 5). In fact, the only is $8,700 USD, 1-year license $3,500 USD, and one-month comparison found not be significant was wet-mesic prairie be- license is $350 USD. The perpetual license includes one year tween ICE and Pix4D (Fisher’s exact p = 0.306). The p value of of support with additional support costing $870 USD per year 0.019 found for typha marsh between ICE and Pix4D, although (https://pix4d.com/buy_rent/). It should be noted that Pix4d technically above the threshold of 0.017 was still considered Mapper Pro license can be installed on two computers to significant due the close proximity with regards to the target facilitate in-the-field rapid check as well as lab-based full pro- value and the conservative nature of Bonferroni corrections cessing. Pix4d also has perpetual education licenses ($1,990 (McDonald, 2009). USD) and non-commercial licenses ($4,990 USD). It should Although differences were found between the different also be noted that reduce pricing of Pix4D is available as part software packages when comparing image artifact presence of some UAS packages. with the vegetation types, no significant differences were found when comparing the vegetation types within a single software package (i.e., canary reed grass versus wet-mesic Conclusions prairie). This indicates that image artifacts, although more As sUAS become an increasingly popular platform for digital likely to occur in Photoscan Pro, are not affected by differenc- aerial photography, it is important to consider the tradeoffs es in the structure of the vegetation such as leaf size, height, between different mosaicking software programs. This re- or woody versus. herbaceous. search has presented three possible software packages well Pix4D showed the highest incident of overall image blur suited to sUAS digial aerial photography and offer a highly with 38 percent of the total sites showing some level of Deliveredblur automated by Ingenta approach: Photoscan Pro, Pix4D, and Microsoft (Figure 5). This was found to significantlyIP: 192.168.39.211higher than Photo -On: Fri,Image 24 Composite Sep 2021 Editor.18:37:04 A comparison was made based on scan Pro with 14 percent andCopyright: ICE with 20 American percent (Chi Society Square for Photogrammetrygeometric accuracy, and visual Remote quality, Sensing and pricing. A ranking p <0.001 and p = 0.05 respectively) (Table 6). Photoscan Pro summary of the software in each category can be seen in Table and ICE were not found to significantly different (Chi Square p 8. ICE produced the fewest visual errors, but had the worst = 0.259). It should be noted that there is some inherent blur in geometric accuracy. Pix4D and Photoscan Pro were statisti- the original imagery due to motion from the aircraft platform. cally similar in terms of geometric accuracy, however Pix4D Although significant differences were found when examin- had better visual quality. It is important to note that all three ing the sites as whole, a vegetation type comparison showed software packages struggled with tree canopies where wind- fewer differences. The only difference found between ICE and blown leaf movement led to increasing amounts of image Pix4D was woody vegetation (Fisher’s exact p = 0.001) (Table blur. Overall, this research suggests that, although there is no 6). Reed canary grass and woody vegetation were found to single best software option for optimizing all criteria, unless be significant between Photoscan Pro and Pix4D (Fisher’s sub-decimeter geometric accuracy is required for the given exact p = 0.011 and 0.007, respectively). This indicates that analysis, ICE provides the most cost effective option photo- the primary difference in performance between the software grammetric applications, although the lack of a DSM product packages is in the way they handle imagery acquired over may deter some users. reed canary grass and woody vegetation. Unlike image artifacts, which showed no significant dif- Table 8. Overall Comparison between the Software Packages. Numbers ference with regards to the type of vegetation present on the Represent the Software Order for that Category with 1 being the Highest. ground, image blur was shown to more likely occur with cer- Tie Values, indicated by “*”, were Awarded in Cases where No Statistical tain vegetation types. Although no significant difference was Difference was Observed. found between the vegetation types in ICE (overall comparison Category Photoscan Pro Pix4D ICE p = 0.065), Pix4D, and Photoscan Pro showed that woody vegetation preferentially produced more sites with image blur Geometric accuracy 1.5* 1.5* 3 (Table 7). Again, due to the conservative nature of Bonferroni Visual quality 3 2 1 corrections the p value of 0.017 found between wet-mesic Cost 2 3 1 prairie and woody vegetation in Photoscan Pro was consid- Ease of use 1 3 2 ered significant. This indicates that tree canopies created problems for both software programs. One possible explana- tion for this increased presence of image blur in regions with References woody vegetation is that those regions would receive a greater Agisoft Online Store, 2015. URL: http://www.agisoft.com/buy/online- amount of windblown movement, which causes leaves to store/ (last date accessed: 27 April 2016). change both location and orientation between subsequent im- Anderson, K., and Gaston, K.J., 2013. Lightweight unmanned aerial ages in a flight sequence. vehicles will revolutionize spatial ecology, Frontiers in Ecology and the Environment, 11(3):138-146.

424 June 2016 Photogrammetric Engineering & Remote Sensing

05-16 June Peer Reviewed.indd 424 5/16/2016 10:28:43 AM Barazzetti, L., F. Remondino, M. Scaioni, and R. Brumana, 2010. Fully Sona, G., L. Pinto, D. Pagliari, D. Passoni, and R. Gini, 2014. automatic UAV image-based sensor orientation. Proceedings of Experimental analysis of different software packages for the 2010 Canadian Geomatics Conference and Symposium of orientation and digital surface modelling from UAV images, Commission I. Earth Science Informatics, 7(2):97-107. Choudhary, S., and P. Narayanan, 2012. Visibility probability Turner, D., A. Lucieer, and L. Wallace, 2014. Direct georeferencing structure from sfm datasets and applications, Computer Vision– of ultrahigh-resolution UAV imagery, IEEE Transactions on ECCV 2012, Springer, pp. 130-143. Geoscience and Remote Sensing, 52(5):2738-2745. Heumann, B.W., 2015.The multiple comparison problem in empirical Turner, D., A. Lucieer, and C. Watson, 2012. An automated technique remote sensing, Photogrammetric Engineering and Remote for generating georectified mosaics from ultra-high resolution Sensing, 81(12):921-926.. unmanned aerial vehicle (UAV) imagery based on structure from Knoth, C., B. Klein, T. Prinz, and T. Kleinebecker, 2013. Unmanned motion (SfM) point clouds, Remote Sensing, 4(5), 1392-1410. aerial vehicles as innovative remote sensing platforms for high- Vallet, J., F. Panissod, C. Strecha, and M. Tracol, 2011. resolution infrared imagery to support restoration monitoring in Photogrammetric performance of an ultra light weight swinglet cut-over bogs, Applied Vegetation Science, 16(3):509-517. UAV, Proceedings of UAV-g. Küng, O., C. Strecha, A. Beyeler, J.-C Zufferey, D. Floreano, P. Verhoeven, G., 2011. Taking computer vision aloft - Archaeological Fua, P.,and F. Gervaix, 2011. The accuracy of automatic three-dimensional reconstructions from aerial photographs with photogrammetric techniques on ultra-light UAV imagery, photoscan, Archaeological Prospection, 18(1):67-73. Proceedings of the UAV-g 2011-Unmanned Aerial Vehicle in Verhoeven, G., M. Doneus, C. Briese, and F. Vermeulen, 2012. Geomatics. Mapping by matching: a computer vision-based approach Küng, O., C. Strecha, P. Fua, D. Gurdan, M. Achtelik, K.-M. Doth, and to fast and accurate georeferencing of archaeological aerial J. Stumpf, 2011. photographs, Journal of Archaeological Science, 39(7):2060- Simplified building models extraction from ultra-light UAV imagery, 2070. ISPRS-International Archives of the Photogrammetry, Remote Westoby, M., J. Brasington, N. Glasser, M. Hambrey, and J. Reynolds, Sensing and Spatial Information Sciences, 3822, 217. 2012. Structure-from-motion photogrammetry: A low-cost, Laliberte, A.S., A. Rango, and J. Herrick, 2007. Unmanned aerial effective tool for geoscience applications, Geomorphology, 179, vehicles for rangeland mapping and monitoring: A comparison 300-314. of two systems, Proceedings of the ASPRS Annual Conference, Wolf, P.R., and B.A. Dewitt, 2000. Elements of Photogrammetry: With Baltimore, Maryland. applications in GIS, Vol. 3, McGraw-Hill, New York. Laliberte, A.S., C. Winters, and A. Rango, 2008. A procedure for Woodget, A., P. Carbonneau, F. Visser, and I. Maddock, 2014. orthorectification of sub-decimeter resolution imagery obtained Quantifying submerged fluvial topography using hyperspatial with an unmanned aerial vehicle (UAV), Proceedings of the resolution UAS imagery and structure from motion ASPRS Annual Conference. photogrammetry, Earth Surface Processes and Landforms, Laliberte, A.S., J.E. Herrick, A. Rango, and C. Winters, 2010. 40(1):47-64. Acquisition, orthorectification, and object-based classification Zaman, B., A.M. Jensen, A. and M. McKee, 2011. Use of high- of unmanned aerial vehicle (UAV) imagery for rangeland resolution multispectral imagery acquired with an autonomous monitoring, Photogrammetric Engineering & Remote SensingDelivered, by Ingentaunmanned aerial vehicle to quantify the spread of an invasive 76(6):661-672. IP: 192.168.39.211 On: Fri, 24 Sepwetlands 2021 species. 18:37:04 Proceedings of the Geoscience and Remote Lindeberg, T., 2012. Scale invariantCopyright: feature American transform, Society Scholarpedia, for Photogrammetry Sensing Symposium and Remote (IGARSS) Sensing, IEEE International. 7(5):10491. Zhenkun, T., F. Yingying, and L. Suhong, 2013. Rapid crops Lowe, D.G., 2004. Distinctive image features from scale-invariant key- classification based on UAV low-altitude remote sensing, points, International Journal of Computer Vision, 60(2):91-110. Transactions of the Chinese Society of Agricultural Engineering, 2013(7). McDonald, J.H., 2009. Handbook of Biological Statistics, Volume 2, Sparky House Publishing, Baltimore, Maryland. Zweig, C.L., M.A. Burgess, H.F. Percival, and W.M. Kitchens, 2015. Use of unmanned Aircraft systems to delineate fine-scale Microsoft Image Composite Editor, 2016. URL: http://research. wetland vegetation communities, , 35(2):303-309. microsoft.com/en-us/um/redmond/projects/ice/ (last date Wetlands accessed: 27 April 2016). Pix4d Buy or Rent, 2016. URL: https://pix4d.com/buy_rent/ (last date (Received 01 July 2015; accepted 05 November 2015; final accessed: 27 April 2016). version 13 January 2016)

Photogrammetric Engineering & Remote Sensing June 2016 425

05-16 June Peer Reviewed.indd 425 5/16/2016 10:28:43 AM