POLISH MARITIME RESEARCH Special Issue 2017 S1 (93) 2017 Vol. 24; pp. 174-181 10.1515/pomr-2017-0036 INTERNET PHOTOGRAMMETRY FOR INSPECTION OF SEAPORTS Zygmunt Paszotta1 Malgorzata Szumilo1 Jakub Szulwic2 1 Univeristy of Wamia and Mazury in Olsztyn, Poland 2 Gdansk University of Technology, Poland ABSTRACT This paper intends to point out the possibility of using Internet photogrammetry to construct 3D models from the images obtained by means of UAVs (Unmanned Aerial Vehicles). The solutions may be useful for the inspection of ports as to the content of cargo, transport safety or the assessment of the technical infrastructure of port and quays. The solution can be a complement to measurements made by using laser scanning and traditional surveying methods. In this paper the authors recommend a solution useful for creating 3D models from images acquired by the UAV using non-metric images from digital cameras. The developed algorithms, created and presented software allows to generate 3D models through the Internet in two modes: anaglyph and display in shutter systems. The problem of 3D image generation in photogrammetry is solved by using epipolar images. The appropriate method was presented by Kreiling in 1976. However, it applies to photogrammetric images for which the internal orientation is known. In the case of digital images obtained with non-metric cameras it is required to use another solution based on the fundamental matrix concept, introduced by Luong in 1992. In order to determine the matrix which defines the relationship between left and right digital image it is required to have at least eight homologous points. To determine the solution it is necessary to use the SVD (singular value decomposition). By using the fundamental matrix the epipolar lines are determined, which makes the correct orientation of images making stereo pairs, possible. The appropriate mathematical bases and illustrations are included in the publication. Keywords: inspection of wharf, security of seaports, registration of cargo ships, UAV (unmanned aerial vehicles) INTRODUCTION spatial observations. A tool like a photogrammetry may be useful for evaluating and measuring tanks, silos and special During recent years the use of unmanned aerial vehicles constructions in the ports [1, 7, 12-15, 17, 28, 30]. The main (UAVs) has increased significantly. UAVs with the capability limitation of the method is range of UAV and availability of of photogrammetric data acquisition opens various new satellite positioning system [11]. applications in aerial and terrestrial photogrammetry There are also used groups of drones which take images and also introduces low-cost alternatives to the classical synchronously, can analyze filling ?, moving the autonomous photogrammetry [25, 26]. An extensive overview of the objects and inspect moving and vibrating parts of vessels. evolution and the state-of-the-art of photogrammetry by Emergency service might be another area of the potential using UAV is given by Eisenbeiss and Unger [4, 31]. The application where rescue teams might need quick and portable authors describe early developments and present a review access to UAV data. The examining stereoscopic model for the on UAVs for photogrammetric topics like flight planning, purpose of identifying objects and judging their significance image acquisition and orientation and data processing. In is much easier than plane image interpretation [21, 22]. this paper the authors focus on generating 3D images from In traditional photogrammetry (which is based on images taken from UAVs through the Internet. photogrammetric images) generating stereoscopic models The solution is based on the website photogrammetry and is a public process. The method of analogue spatial can be used for inspection of ports, wharves, coasts, and ships. visualization in stereoscopic entertainment mediums such Construction of stereoscopic model, optionally enriched as Kaiserpanorama (consisted of sets of stereo slides and with scale imaging, allows for a correct identification and a multi-station viewing apparatus) has been known since the 174 POLISH MARITIME RESEARCH, No S1/2017 mid 19th century. In the literature source [20] four conditions THEORETICAL FUNDAMENTALS are mentioned for the analogue way of obtaining the 3D effect. They refer to: The concept of a fundamental matrix is introduced, – parallactic angle, assuming that homologous points lie on the epipolar plane – the difference of scales of the left and right image, as the starting point. – separation of transmission channels for left and right According to Fig. 1, the following equation must be images (sending the “right eye” image to the right eye satisfied: and the “left eye” image to the left eye), – watching pictures on the epipolar plane. r "⋅(b ×r') = 0 (1) When all the above mentioned conditions are met, to t obtain 3D images from non-metric images will be possible. where: The first condition is satisfied if the images acquired during the photogrammetric flight have appropriate forward overlap. ijk ª0 − brzy'+bryz'º ª 0 − bbz y ºªrx 'º « » « »« » The images taken when the UAVs have only turned around, b × r'= bbx y bz = « brzx'+0 − brxz' » = « bz 0 − bx »«ry '» = Br' (2) « » « »« » are improper. In this case the base which represents the rx ' ry ' rz ' ¬− bryx'+brxy'−0¼ ¬− bby x 0 ¼¬rz '¼ distance between the projection centres of both images is too small. Moreover, these are not overlapping images. If Q is the rotation matrix for vector r" to the object The second condition (scale stability) shouldbe guaranteed coordinate system (O1,x,y,z), i.e.: for in-flight plane. The next condition depends on the selection of 3D display techniques. In this paper the anaglyph r "= Qr", (3) method and active shutter 3D systems are implemented. The t theoretical basis of the fourth condition is described in detail then the scalar product is given as : in this work beneath. T T T A breakthrough in the process of 3D image generation was ()rt " Br'= (r") Q Br'= 0 . (4) made in 1976 when Kreiling developed a method of generating epipolar images [3, 9, 27, 33]. However, it is possible to project Let the two cameras be characterized by two calibration stereogram images onto the common plane if the camera matrices A1 and A 2 (containing only the principal distance): constants and the elements of relative orientation are known. Generating and superimposing component digital images is rp '= Ar1 ' rp"= Ar2 " . (5) also a public process. Many functions and actions which perform measurements and computations, must be programmed. It appears that it is possible to generate 3D images (the pair of images which relative orientation is known and which could be visualized as a stereoscopic model) with the use of the idea of anaglyphic images on the Internet, interactively. It is possible to reach such solution on an ordinary computer, but the quality of the so obtained three-dimensional image is not sufficient. The generation and visualization of 3D images by using shutter glasses is a more advanced approach. Both of the mentioned solutions work as an Internet application in JAVA and use client-server technology, which in practical terms means communication between applets and the servlet. This paper presents the theoretical fundamentals for the adopted solution, together with a plan for its implementation. Since a few years ago (from about 2001) up to now these authors have developed interactive Internet photogrammetric applications [22, 23]. The described research is a summary of work performed on non-metric images. Due to the use of JAVA software , the presented solution Fig. 1. Geometrical relationships on a stereogram might be used in most of the contemporary web browsers. The authors propose a solution which is available through web browsers as a portable tool and convenient alternative to therefore: standalone desktop applications. The use of Internet solution T T T −1 allows to make advantage of all the merits of this technology, (rp") A2 Q B A1 rp '= 0 (6) such as universal access to the data, simultaneous use by multiple users. POLISH MARITIME RESEARCH, No S1/2017 175 T T −1 Let F = A2 Q B A1 to be called a fundamental matrix. It number of non-zero elements on the diagonal of diagonal helps to determine epipolar lines. For the point with the matrix V. In order to obtain rank (M)=8, or rather rank coordinates rp" on the right image, the epipolar line on the left (M’)=8, the least singular value should be equal to zero. image is described by the following equation (in homogeneous If our measurements of the coordinates of homologous points coordinates): were correct, there should be a sharp jump between the eighth and the ninth singular value. The corrected matrices V and rp"⋅u"= 0 , where u"= Frp ' (7) M will be marked V’ and M’, respectively , where : T T −1 T The shape of the matrix F = A2 Q B A1 is 3×3. There are M'= S'V'D' (11) nine coefficients to determine. One of them is connected with scale factor and hence it could be reduced, which makes that only eight coefficients have to be calculated. To determine them, at least eight homologous points are needed. This algorithm was introduced by Longuet-Higgins [16, 24]. Therefore Eq. (6) can be written as follows : T (rp ") F rp '= 0 (8) All the epipolar lines on a given image plane, either left or right, go through one epipole. They make up a pencil of lines which meet in the epilpole (epipolar pencil). If they are marked e p ' and e p", respectively, then the following relationships are satisfied [6]: Fe '= 0 e "F = 0 (9) p , p IMPLEMENTATION OF THE APPLICATION Fig. 2. Activity diagram in the process of generating anaglyphic 3D images Implementation of the method in question requires the proper software and execution of various operations both on the part of the client and of the server.