Introduction
This benchmark compares the performances of the three main photo-based 3Dmodeling software. Five projects related to different kind of applications were led on the same machine and this document provides the results.
1 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
About this document
Information in this document is subject to change without notice and is provided «as is» with no warranty. Acute3D makes no warranty of any kind with regards to this material, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Acute3D shall not be liable for errors contained herein or for any direct, indirect, special, incidental or consequential damages in connection with the use of this material.
2 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
Table of contents
INTRODUCTION ...... 1 ABOUT THIS DOCUMENT ...... 2 1- APPLICATION 1: CITIES - GRAZ ...... 4
1.1. COMPUTATION SYSTEM ...... 4 1.2. DATASET ...... 4 1.3. PRELIMINARY REMARKS ...... 4 1.4. AEROTRIANGULATION ...... 5 1.5. RECONSTRUCTION ...... 5 1.6. RESULTS ...... 6 2- APPLICATION 2: BUILDINGS - FLEURAC ...... 9
2.1. COMPUTATION SYSTEM ...... 9 2.2. DATASET ...... 9 2.3. PRELIMINARY REMARKS ...... 9 2.4. AEROTRIANGULATION ...... 10 2.5. RECONSTRUCTION ...... 10 2.6. RESULTS ...... 11 3- APPLICATION 3: MINING QUARRY ...... 13
3.1. COMPUTATION SYSTEM ...... 13 3.2. DATASET ...... 13 3.3. PRELIMINARY REMARKS ...... 13 3.4. AEROTRIANGULATION ...... 14 3.5. RECONSTRUCTION ...... 14 3.6. RESULTS ...... 15 4- APPLICATION 4: STREET - TOKYO ...... 17
4.1. COMPUTATION SYSTEM ...... 17 4.2. DATASET ...... 17 4.3. PRELIMINARY REMARKS ...... 17 4.4. AEROTRIANGULATION ...... 18 4.5. RECONSTRUCTION ...... 18 4.6. RESULTS ...... 19 5- APPLICATION 5: TOMBSTONE ...... 21
5.1. COMPUTATION SYSTEM ...... 21 5.2. DATASET ...... 21 5.3. PRELIMINARY REMARKS ...... 21 5.4. AEROTRIANGULATION ...... 22 5.5. RECONSTRUCTION ...... 22 5.6. RESULTS ...... 23 6- SYNTHESIS ...... 25
6.1. PIX4D ...... 25 5.2.1. Advantages ...... 25 5.2.2. Drawbacks ...... 25 6.2. AGISOFT PHOTOSCAN...... 26 5.3.1. Advantages ...... 26 5.3.2. Drawbacks ...... 26
3 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
1- Application 1: Cities - Graz
1.1. Computation System
OS : windows 7 professional Processor : Intel core I7-4930k CPU @ 3.40 GHz (8 processors / 16 threads) RAM : 64 Go GPU : Nvidia GeForce GTX 980
1.2. Dataset
The dataset is composed of 203 pictures over the city of Graz (Austria). The dataset was acquired from a Microsoft UltraCam Osprey mounted on an airplane and is composed of 39 nadir and 164 oblique photos. Position and orientation information are available for each photo of the dataset. The multi-cameras system is composed of:
1 Nadir camera o Sensor size : 70.044*45.084 mm o focal length : 80 mm o Photo dimensions : 11674*7514 Pix o boresight angle : vertical o orientation : landscape (according to flight direction).
2 oblique cameras oriented forward and backward o Sensor size : 53.4*39.9 mm o focal length : 120 mm o Photo dimensions : 8900*6650 Pix o boresight angle : 45° from vertical o orientation : landscape
2 oblique cameras oriented left and right o Sensor size : 53.4*39.9 mm o focal length : 120 mm o Photo dimensions : 8900*6650 Pix o boresight angle : 45° from vertical o orientation : portrait
1.3. Preliminary remarks
This dataset is the biggest of this Benchmark. The input data size is 2.27 Go. This example can be considered as a reference for the following applications: City-modeling Urban Planning DSM and DEM Creation on wide area
4 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
1.4. Aerotriangulation
Smart 3D Capture Pix4D Agisoft Photoscan
Pictures aligned 203/203 190/203 196/203
Max RAM used 0.97 Go 1.3 Go 3.4 Go
Computation time 0h 50min 28sec 4h 40min 00sec 22min 00sec
1.5. Reconstruction
Smart 3D Capture Pix4D Agisoft Photoscan
Max RAM used 20.5 Go 28.6 41 Go
Computation time 27h 46min 3h 57min 33h42min
Number of triangles 21 801 179 4 000 000 X
Size of generated file 1.1 Go X 1.2 Go
* About this project, we couldn't lead a successful process with Pix4D default settings The initial step (Aerotriangulation) crashed. We needed to use the "Split into Subprojects" function to separate our initial block into five parts before merging them. That's why the AT processing time is important. On this dataset, we also faced the rental-edition input-size limit of 42MPx. We led the whole project on the same machine but using the Discovery edition. So that, we couldn't produce OBJ file and that's why the size of output file isn't available.
** To run this project on Smart3DCapture and Photoscan, we used the tiling feature in order to avoid memory crashes
***The tiled export format from Photoscan does not provide any information about the number of triangles
5 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
1.6. Results
6 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
7 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
8 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
2- Application 2: Buildings - Fleurac
2.1. Computation System
OS : windows 7 professional Processor : Intel core I7-4930k CPU @ 3.40 GHz (8 processors / 16 threads) RAM : 64 Go GPU : Nvidia GeForce GTX 980
2.2. Dataset
The dataset is composed of 90 pictures looking at a castle in Fleurac (Dordogne, France). The pictures are set circularly around the scene and were shot from an aircraft. We have camera position information in the EXIF files for each images.
Camera: Nikon D800 Pictures dimensions : 4858x3242 Focal length : for 68/90 images, focal length is 105mm for 22/90 images, focal length is 195mm Sensor size : 36 mm format: JPG
2.3. Preliminary remarks
The input data size is 787Mo. This example can be considered as a reference for the following applications: Building modeling Cultural Heritage Infrastructures
9 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
2.4. Aerotriangulation
Smart 3D Capture Pix4D Agisoft Photoscan
Pictures aligned 90/90 90/90 90/90
Max RAM used 0.47 Go 1.10 Go 1.2 Go
Computation time 9min 56sec 16min 59sec 5m36sec
2.5. Reconstruction
Smart 3D Capture Pix4D Agisoft Photoscan
Max RAM used 9.3 Go 3.8 Go 16.6 Go
Computation time 2h 19min 5h 08min 6h 22min
Number of triangles 1 689 860 5 000 000 4 841 945
Size of generated file 105 Mo 665 Mo 376 Mo
10 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
2.6. Results
11 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
12 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
3- Application 3: Mining Quarry
3.1. Computation System
OS : windows 7 professional Processor : Intel core I7-4930k CPU @ 3.40 GHz (8 processors / 16 threads) RAM : 64 Go GPU : Nvidia GeForce GTX 980
3.2. Dataset
The dataset is composed of 127 nadir photos over a quarry. The photos were shot from an UAV. Camera position information is available in the EXIF files and 7 Ground Control points for this site.
Camera: Canon IXUS220HS Pictures dimensions : 4000*3000 Focal length : 4.4 mm Sensor size : 6.2 x 4.65 mm Format: JPG 7 ground control points (WGS84)
3.3. Preliminary remarks
The input data size is 450Mo. This case can be considered as a reference for the following applications: Land Surveying Mining Volume workouts Worksite management
13 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
3.4. Aerotriangulation
Smart 3D Capture Pix4D Agisoft Photoscan
Pictures aligned 127/127 127/127 127/127
Max RAM used 0.37 Go 1.06 Go 1.3 Go
Computation time 6min 13sec 6min 32sec 12 min 10s
3.5. Reconstruction
Smart 3D Capture Pix4D Agisoft Photoscan
Max RAM used 8.35 Go 4.08 Go 30 Go
Computation time 1h 47min 0h 26min 2h 04min
Number of triangles 1 974 670 1 000 000 7 400 000
Size of generated file 123Mo 135 Mo 594 Mo
14 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
3.6. Results
15 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
16 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
4- Application 4: Street - Tokyo
4.1. Computation System
OS : windows 7 professional Processor : Intel core I7-4930k CPU @ 3.40 GHz (8 processors / 16 threads) RAM : 64 Go GPU : Nvidia GeForce GTX 980
4.2. Dataset
The dataset is composed of 250 pictures shot from a car and looking around a street in Tokyo
Camera: Point Grey Research Ladybug Pictures dimensions : 1232x1616 Focal length : 3.3 mm Sensor size : 7.11 mm format: JPG
4.3. Preliminary remarks
The input data size is 128Mo. This case can be considered as a reference for the following applications: Street Modeling 3D mobile mapping
17 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
4.4. Aerotriangulation
Smart 3D Capture Pix4D Agisoft Photoscan
Pictures aligned 241/250 250/250 250/250
Max RAM used 0.22 Go 0.49 0.8 Go
Computation time 6min 53sec 9min 33sec 32 min 12sec
4.5. Reconstruction
Smart 3D Capture Pix4D Agisoft Photoscan
Max RAM used 2.17 Go 2.29 Go 3.1 Go
Computation time 42min 27sec 18 min 37 sec 38 min 23sec
Number of triangles 532 385 1 000 000 1 450 000
Size of generated file 47.8Mo 129 Mo 109 Mo
18 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
4.6. Results
19 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
20 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
5- Application 5: Tombstone
5.1. Computation System
OS : windows 7 professional Processor : Intel core I7-4930k CPU @ 3.40 GHz (8 processors / 16 threads) RAM : 64 Go GPU : Nvidia GeForce GTX 980
5.2. Dataset
The dataset is composed of 89 terrestrial pictures shot looking around a tombstone in a cemetery
Camera: Canon EOS 30D Pictures dimensions : 2336x3504 Focal length : 50 mm Sensor size : 22.5 mm format: TIF
5.3. Preliminary remarks
The input data size is 2.09Go. This case can be considered as a reference for the following applications: - Cultural heritage - Archeology
21 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
5.4. Aerotriangulation
Smart 3D Capture Pix4D Agisoft Photoscan
Pictures aligned 79/89 84/89 89/89
Max RAM used 0.25 Go 0.54 Go 1.20 Go
Computation time 3min 50sec 4min 24sec 4min 28sec
5.5. Reconstruction
Smart 3D Capture Pix4D Agisoft Photoscan
Max RAM used 3.75 Go 2.29 Go 10.1 Go
Computation time 36min 20sec 12 min 01 sec 1h 02min 00sec
Number of triangles 695 140 1 000 000 2 700 000
Size of generated file 50.3 Mo 129 Mo 200 Mo
22 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
5.6. Results
23 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
24 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
6- Synthesis
6.1. Pix4d
5.2.1. Advantages - Fast computation steps. The use of GPU in step 1 is really time saving (around 55% with a GTX980 for our tests). - The quality reports are really complete and give a good idea on how a process worked. - Even if they do not appear in this benchmark, the point-classification features (automatic classification and pointcloud edition) are interesting to extract some relevant parts of the model (Buildings, DTM, ...). - The volume, perimeter and distance measurement tools are quite efficient and easy to use. Interoperability between these tools and Topography/GIS software looks quite efficient. - The user interface is intuitive, mainly for importing GCP or position/orientation info. Their automatic-pointing tool is really interesting and time-saving. - Pix4D provides an easy-to-use animation tool which allows the user to export videos from his work - The ability to automatically split a large georeferenced dataset is quite interesting and allows to deal with big data for initial processing (Aerotriangulation). - The licensing system is really efficient. The possibility to work on field with a portable license is interesting.
5.2.2. Drawbacks - The production of 3D-textured mesh is a new feature and it's not efficient yet. The number of triangles and texture size are defined-by-user parameters and they can't be automatically adjusted according to your model size or to your point cloud density. - The edges of the 3D-mesh are not as sharp as they can be in Smart3DCapture. Some small details cannot be reconstructed even with higher settings (See the Tombstone example). - Even if you can split your project in several parts and run them on separate machines, Pix4D doesn't' have real network computing feature. There is no way to run a same project on different engines from a unique computer. - A process (AT or point cloud densification) automatically overwrites your previous work. If you run an AT over an already aerotriangulated dataset, your first result is automatically lost. It forces you to save your work as a new project every time you reset your AT or dense point cloud computation if you want to keep intermediate results. - Impossibility to automatically use masks in order to hide some irrelevant parts of images. The annotation tool allows to discard some part of images during reconstruction step but you must do it manually by selecting these irrelevant areas in each photos. - No way to import Image-Path when importing a geolocation file. Which means you can't work on pictures with the same name but contained in different folders. - No way to import your whole data (images, cameras and GCP) in a single block-file. - The size of input pictures is limited to 42MP in the rental edition and you need to ask for an upgrade if you want to work with that kind of images. That's not mentioned when you purchase your PRO License.
25 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines
6.2. Agisoft Photoscan
5.3.1. Advantages - The tool for mask creation is really convenient. - Almost all the processing steps are parallelizable (Keypoint detection, photo matching, dense point cloud, tile export), meaning all the processing steps can be fasten using several computers. - Alignment between chunks: if a dataset has to be split in several chunks because of memory limitation, Photoscan is able to adjust the different chunks to get a seamless model at the junction. - Classification tool: the classification tool allows to remove quite efficiently all the building and vegetation in order to create a true DTM. - Volume calculation: Photoscan has a tool to calculate volume. Anyway, the process to do this calculation is not really convenient and not really flexible. You cannot specify a special/custom reference surface. - Orthophotos can be generated along different axis (not only Z).
5.3.2. Drawbacks - The RAM consumption at all steps of the process is really important. This means you will need much more RAM than Smart3DCapture too process similar size projects. - 3D geometry not as sharp and detailed as Acute's. The building edges and small details are not as good. the model seems to be smoothed. - Photogroup and camera calibration per photogroup is tedious to setup (quite painful for multi-cameras systems). - After importing the metadata, there is no visual confirmation of the orientation of the cameras. - No smart bounding box setup (from 3D polygon for example, only manual) - The tiled 3D mesh export (.tls) is created from the dense point cloud and only convertible in LodTree format. To get OBJ, you will have to export the mesh again, and you will face memory crash for large project. - No touchup process for large tiled production - Impossible to process a large area in standard 3D format (as OBJ). - Creates very large 3D models with huge amount of triangles. - No link between the tile export and the standard 3D model export (Smart3DCapture has the "reference 3D model") - Tile export is handle as one "job", so if it fails, the export is lost and has to be started from scratch again.
26 Acute3D S.A.S. WTC 2 06560 Valbonne Sophia Antipolis 120 route des Macarons France www.acute3D.com MMacaronsDolines