Multi-Temporal Insar Parallel Processing for Sentinel-1 Large-Scale Surface Deformation Mapping

Article Multi-Temporal InSAR Parallel Processing for Sentinel-1 Large-Scale Surface Deformation Mapping

Wei Duan 1,2, Hong Zhang 1,* , Chao Wang 1,2 and Yixian Tang 1

1 Key Laboratory of Digital Earth Science, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China; [email protected] (W.D.); [email protected] (C.W.); [email protected] (Y.T.) 2 College of Resources and Environment, China University of Chinese Academy of Science, Beijing 100049, China * Correspondence: [email protected]

Received: 6 October 2020; Accepted: 11 November 2020; Published: 14 November 2020

Abstract: Interferometric synthetic aperture radar (InSAR) has achieved great success in various geodetic applications, and its potential for ground deformation measurements on the large scale has attracted increasingly more attention in recent years. The increasing number of synthetic aperture radar (SAR) satellite systems have steadily provided a large amount of SAR data. Among these systems, the Sentinel-1 mission can be considered a milestone in the development of InSAR techniques, offering new opportunities to monitor global surface deformation with high precision, due to its wide coverage, short revisit time, and free access. However, conventional InSAR techniques have encountered great challenges in large-scale InSAR processing over wide areas because of the large computational burden and complexity. In this work, we present a novel parallel computing-based coherent scatterer InSAR (P-CSInSAR) method for automatic and efficient generation of deformation results from Sentinel-1 raw data. To achieve high parallelization performance for the overall InSAR processing chain, parallelization strategies at different levels have been adopted in the P-CSInSAR method, which has been fully addressed in this work. To evaluate the efficiency and accuracy of the proposed method, P-CSInSAR has been tested on the North China Plain regions with three adjacent frames of Sentinel-1 images, and the deformation results have been validated by GPS measurements. The experimental results confirm the effectiveness of the proposed parallel computing-based P-CSInSAR method. The proposed method can also play an important role in exploiting Sentinel-1 InSAR big data for disaster prevention and reduction.

Keywords: Sentinel-1; InSAR; big data; parallel computing; deformation mapping

1. Introduction Space-borne interferometric synthetic aperture radar has the ability to detect subtle surface deformations with wide coverage and high resolution; thus, it has been widely used in a range of geophysical applications, such as earthquakes, landslides, subsidence, and other natural or man-made hazards [1]. To overcome the limitations of temporal–spatial decorrelation and atmospheric eﬀects in conventional interferometric synthetic aperture radar (InSAR) processing, multi-temporal InSAR (MT-InSAR) techniques, especially permanent scatterer interferometry (PSI) [2,3] and the small baseline method (SBAS) [4], have been developed by using the time series analysis of a stack of interferograms. These advanced processing algorithms [5] have achieved great success in obtaining ground deformation with high precision on the regional scale [6–9], and have made InSAR one of the most important remote sensing tools for risk prevention and monitoring.

Remote Sens. 2020, 12, 3749; doi:10.3390/rs12223749 www.mdpi.com/journal/remotesensing Remote Sens. 2020, 12, 3749 2 of 20

With the rapid development of InSAR techniques, a large number of SAR satellite systems have been launched over the past few decades, including ERS-1/2, ENVISAT ASAR, Radarsat-2, TerraSAR-X, Cosmos-skyMed, ALOS-2/PALSAR-2, and Sentinel-1. The fast growth of SAR achieved with different bands and resolutions provided by these sensors has greatly facilitated InSAR data source accessibility. Among them, the C-band Sentinel-1 mission with medium resolution was originally designed for large-scale interferometric applications. The Sentinel-1 SAR data with Interferometric Wide Swath (IW) acquisition mode is characterized by its 250 km-wide swath, global coverage, and short revisit time (6 or 12 days) [10–12]. Moreover, the European Space Agency (ESA) has adopted a complete, free, and open-access policy for all Sentinel-1 datasets [13]. Thus, these abundant SAR images have greatly promoted the InSAR ground deformation monitoring from the regional to national scale. Many researchers have exploited the large amount of InSAR data for wide areas and long time series investigations in recent years, and these successful applications have shown that the potential of InSAR large-scale deformation monitoring is promising. In 2015, funded by the Italian Ministry of the Environment, Costantini et al. [14] first carried out national-scale InSAR research and processed all the ERS and ENVISAT SAR scenes to generate deformation results for the entire Italian territory. Haghighi et al. [15] studied large-scale Sentinel-1 interferometry over Germany and assessed the perspective of national-scale deformation analysis. Shamshiri et al. [16] generated large-scale Sentinel-1 interferograms over Norway by concatenating consecutive frames along the specific orbit and studied the tropospheric correction for the interferograms by using nationwide Global Navigation Satellite System (GNSS) measurements. Raspini et al. [17] developed an automatic Sentinel-1 InSAR processing system for deformation generation and risk evaluation, and the software system was applied in disaster monitoring for the Tuscany region. Many open-source software packages have been widely used for Sentinel-1 InSAR processing, including SNAP [18], GMTSAR [19], STAMPS [20], GIANT [21], MintPy [22], etc. Some researchers have also developed their own InSAR processing systems based on the combination of these software packages [23]. However, effectively exploiting the massive volume of Sentinel-1 data over wide areas has posed a great challenge for these InSAR tools. Currently, Sentinel-1 provides approximately 10 TB of SAR products per day [24], and time series SAR images are more than 1 TB in size per scene. Moreover, the total volume of national-scale SAR archives has reached the PB level. The surge in InSAR data volume has caused not only a demand for more storage devices and computing resources, but also a drastic increase in processing burden and complexity. In this sense, the InSAR data nowadays can be regarded as remote sensing big data. To address the InSAR big data problem, modern advanced high-performance computing (HPC) infrastructure, such as cloud and cluster computing, has been employed to improve the efficiency of InSAR processing [25–28]. As conventional InSAR algorithms were originally developed for sequential processing, some parallel computing-based methods, such as the Parallel Small BAseline Subset (P-SBAS) [29,30], have been proposed to better-exploit the cloud computing environment. The results have demonstrated the feasibility and effectiveness of parallel computing in InSAR large-scale deformation monitoring. China has experienced strong subsidence and natural hazards over a long time period, and a fast and effective geodetic tool for mapping national ground deformation is of great significance for government planning and risk management. Meanwhile, the lack of a high-performance computing environment-based MT-InSAR processing platform has greatly limited large-scale InSAR applications in China. In this work, we focus on an algorithmic parallelism design for the Sentinel-1 MT-InSAR method to take full advantage of parallel processing resources in cloud computing or distributed computing infrastructure, and a novel parallel-based MT-InSAR method, namely parallel computing-based coherent scatterer InSAR (P-CSInSAR), has been proposed in this work. Based on the different data and algorithmic structures used during the processing stages, parallelization strategies with multiple levels have been adopted in the workflow to optimize parallel processing efficiency. In addition, the processing chain of P-CSInSAR is highly automatic and can perform the task from raw data conversion to displacement time series generation without any human interaction. The performance Remote Sens. 2020, 12, 3749 3 of 20 of the P-CSInSAR method has been tested on a real Sentinel-1 dataset over the North China Plain, acquired from September 2018 to March 2020. The accuracy of the estimated deformation results has been validated by GPS measurements, and the parallel computing efficiency of the proposed method has been evaluated. By developing the high-performance InSAR big data processing platform for the Sentinel-1 SAR dataset, we can realize the capability of routinely monitoring the widespread subsidence in the North China Plain and other city clusters in China, which will be very helpful for local governments in urban planning and policy making. This paper is organized as follows. In Section2, we describe the workflow of the proposed P-CSInSAR method and its parallelization strategies. In Section3, the P-CSInSAR method is applied in a real case on the North China Plain, and the performance and accuracy of the proposed method are addressed. Finally, in Section4, the conclusion and future developments of the present work are discussed.

2. Methodology Developing an efficient parallel computational model for the Sentinel-1 MT-InSAR method is very challenging, as several aspects, such as load balancing, communication, and data dependencies, should be properly taken into account throughout the processing chain. The MT-InSAR method requires several processing steps, and the algorithms are faced upward with different data structures during the processing stages. Moreover, the sequential implementation of the MT-InSAR method has a strong dependency between different algorithm steps, and massive input/output (I/O) operations are performed during the processing. Therefore, to obtain a parallel solution for the MT-InSAR method with high efficiency and scalability, we should carefully design the parallel algorithm and organize the sequence of the MT-InSAR processing steps to minimize disk access, minimize interprocess communication, and maximize the usage of the CPU. Through an analysis of the workflow and features of the Sentinel-1 MT-InSAR method, the parallelizability for each algorithm step has been studied. For specific algorithms deemed suitable for parallelization, the appropriate parallel strategies are selected. For instance, Sentienl-1 burst data are separated as independent images; thus, the processing for burst images can be naturally parallelized on the burst level. For those algorithms that cannot be directly implemented in parallel form, we should exploit the potential of data independency in the processing, and decompose the algorithm into the parallelizable and nonparallelizable parts; thus, the former will be executed in parallel. For example, in the time series analysis, image data can be divided into different blocks, while the calculation for these blocks is the parallelizable part, and the results fusion for blocks is the nonparallelizable part. By adopting the parallelism at different levels during the course of the processing, we can maximize the parallelization for the overall processing chain. Based on above considerations, the framework of the proposed P-CSInSAR algorithm is mainly divided into three parts: Sentinel-1 Terrain Observation with Progressive Scans (TOPS)-mode image processing, Sentinel-1 interferometry, and time series analysis, as shown in Figure1. We first provide an overview of the P-CSInSAR algorithm, and the detailed description is provided in Sections 2.1–2.3. The algorithm starts with preparation of the Sentinel-1 raw images, Digital Elevation Model (DEM), and the precise orbit files. After Sentinel-1 TOPS-mode image processing, a series of coregistrated SAR images and their parameter files are produced. Then, during Sentinel-1 interferometry, multilook interferograms with small baselines are generated, and the flat-Earth and topography phase are removed. Finally, through time series analysis, the deformation rate and displacement time series are obtained, and the output files can be exported into text, GeoTIFF, or shapefile formats. A large amount of intermediate results will save to the disk storage, and the program will delete some unused files if necessary. Moreover, the processing pipeline is written in Python and Bash scripts with predefined parameters, and can automatically generate deformation results from the raw Sentinel-1 data without any extra interaction work. Remote Sens. 2020, 12, 3749 4 of 20 Remote Sens. 2020, 12, x FOR PEER REVIEW 4 of 19

DDaattaa pprreeppaarraattiioonn Sentinel-1 PPrreecciissee DDEEMM raw data oorrbbiitt ffiillee

SSeennttiinneell--11 TTOOPPSS mmooddee Sentinel-1 iimmaaggee pprroocceessssiinngg Sentinel-1 Time series analysis iinntteerrffeerroommeettrryy Time series analysis Data unpacking, Data unpacking, CCoohheerreenntt ssccaatttteerreerr importing, and IInntteerrffeerrooggrraamm ppaaiirr CC importing, and sseelleeccttiioonn ccoonnvveerrssiioonn sseelleeccttiioonn

Network connection TTOOPPSS sspplliitt aanndd DDee-- Network connection AA CC Interferogram generation E and deformation rraammpp Interferogram generation E and deformation eessttiimmaattiioonn GGeeoommeettrriicc AA FFllaatt--EEaarrtthh aanndd ttooppooggrraapphhyy ccoorreeggiissttrraattiioonn CC NNeettwwoorrkk ccaallccuullaattiioonn pphhaassee eessttiimmaattiioonn EE aanndd iinntteeggrraattiioonn A ESD fine coregistration A ESD fine coregistration DDiiffffeerreennttiiaall iinntteerrffeerrooggrraamm CC ggeenneerraattiioonn FF RReessiidduuaall pphhaassee uunnwwrraappppiinngg AA RRee--rraammpp D Multilook filtering D Multilook filtering DD AAttmmoosspphheerriicc ffiilltteerriinngg BBuurrsstt aanndd SSuubb--sswwaatthh BB mmeerrggiinngg DDeeffoorrmmaattiioonn mmaapp aanndd AAvveerraaggee Coregistrated ddiissppllaaccmmeenntt ttiimmee sseerriieess Int ccoohheerreennccee SAR images mmaapp

AA :: BBuurrsstt--bbaasseedd ppaarraalllleelliissmm BB :: SSuubbsswwaatthh--bbaasseedd ppaarraalllleelliissmm CC :: PPiixxeell--bbaasseedd ppaarraalllleelliissmm DD :: WWiinnddooww--bbaasseedd ppaarraalllleelliissmm EE :: BBlloocckk--bbaasseedd ppaarraalllleelliissmm FF :: IImmaaggee--bbaasseedd ppaarraalllleelliissmm

FigureFigure 1. The 1. The workflow workflow of the of proposed the proposed parallel parallel computing-based computing-based coherent coherent scatterer scatterer InSAR (P-CSInSAR)InSAR (P- algorithm.CSInSAR The) algorithm. different The letters different represent letters the represent adopted the parallelization adopted parallelization strategies strategies at different at different levels. levels. 2.1. Sentinel-1 TOPS-Mode Image Processing 2.1. Sentinel-1 TOPS-Mode Image Processing 2.1.1. Algorithm Steps for Sentinel-1 TOPS-Mode Image Processing 2.1.1.In thisAlgorithm part, the Steps processing for Sentinel steps-1 mainly TOPS- includeMode Image image Processing coregistration and mosaicking. Unlike other SAR images,In this Sentinel-1part, the processing SLC (single steps look mainly complex) include data image in IWcoregistration mode use and the mosaicking. Terrain Observation Unlike withother Progressive SAR images Scans, Sentinel (TOPS)-1 SLC to (single achieve look wide-swath complex) data coverage in IW mode [31]. use A the TOPS-mode Terrain Observation SAR image consistswith Progressiv of three individuale Scans (TOPS) subswath to achieve files (IW1,wide- IW2,swath and coverage IW3), [31] where. A eachTOPS subswath-mode SAR contains image a seriesconsists of burst of three files individual (at least nine). subswath There files are (IW1, overlapping IW2, and areas IW3) between, where each the consecutive subswath contains bursts anda subswaths,series of burst and there files are(at least invalid nine). data There in the are black overlapping regions of areas burst between images. the Due consecutive to the azimuth bursts antenna and steering,subswaths, Sentinel-1 and there SLC aredata in havevalid adata stricter in the requirement black regions for image of burst coregistration, images. Due which to the demandsazimuth a coregistrationantenna steering, accuracy Sentinel of 0.001-1 SLC samples, data ha andve a a stricter small mis-coregistration requirement for image will introducecoregistration, an unwanted which phasedemand ramps a into coregistration interferograms accuracy among of 0.001 adjacent samples, bursts. and a small mis-coregistration will introduce an unwanted phase ramp into interferograms among adjacent bursts. Assuming N Sentinel-1 SAR images of the same scene acquired at times t1, t2, ...... , tN, these SLC imagesAssuming are first N Sentinel unpacked-1 SAR from images the raw of zipthe filessame and scene subjected acquired to at data time formats 푡1, 푡2, conversion...... , 푡푁, these Then, theSLC precise images orbit are files first are unpacked applied from to update the raw the zip orbit files parameters, and subjected which to data are format then used conversion in the geometric. Then, coregistration.the precise orbit Through files are TOPS applied split to update processing, the orbit all parameters burst images, which are are extracted then used from in the each geometric subswath coregistration. Through TOPS split processing, all burst images are extracted from each subswath image file and stored into separate burst image files. Sentinel-1 image coregistration is then performed image file and stored into separate burst image files. Sentinel-1 image coregistration is then on these burst images. However, due to the azimuth antenna steering, the deramping phase must be performed on these burst images. However, due to the azimuth antenna steering, the deramping removed from the burst data prior to image resampling operation, while after image coregistration, phase must be removed from the burst data prior to image resampling operation, while after image the coregistrated burst images are compensated for the deramping phase. Thus, the deramping phase coregistration, the coregistrated burst images are compensated for the deramping phase. Thus, the is estimatedderamping by phase the Doppleris estimated centroid by the frequency, Doppler centroid and then frequency, stored in and the diskthen tostored be used in the in disk the derampto be step and reramp step. SAR image coregistration transforms all the slave images to the master reference

RemoteRemote Sens.Sens. 20202020,, 1212,, 3749x FOR PEER REVIEW 55 ofof 2019

used in the deramp step and reramp step. SAR image coregistration transforms all the slave images geometry,to the master and the reference good coregistration geometry, and is vital the goodto generating coregistration correct is interferograms, vital to generat especiallying correct for Sentinel-1interferograms, images. especially To achieve for high Sentinel coregistration-1 images. accuracy, To achieve a two-step high coregistration method, the accuracy, initial coregistration a two-step andmethod fine, coregistration, the initial coregistration is widely used and in Sentinel-1 fine coregistration, image processing. is widely used in Sentinel-1 image processingGeometric. coregistration is used as the first-order coregistration by the use of precise orbit state vectorsGeometric and an externalcoregistration DEM. is This used method as the transformsfirst-order coregistration the pixel from by SAR the coordinatesuse of precise to orbit Cartesian state referencevectors and coordinates an external based DEM on. This the range-Doppler method transforms (RD) equations,the pixel from and theSAR range coordinates and azimuth to Cartesian offsets betweenreference the coordinates master and based slave on images the range can be-Doppler estimated. (RD) Then, equations slave image, and the resampling range and and azimuth interpolation offsets arebetween performed the bymaster using and the sinc slave kernel. image Geometrics can be coregistration estimated. Then does not, slave rely on image SAR image resampling coherence, and andinterpolation its accuracy are performed greatly benefits by using from the the sinc precise kernel. orbit Geometric file of Sentinel-1.coregistration The does processing not rely on step SAR is performedimage coherence, on all the and burst its accuracy data for greatly each master–slave benefits from image the precisepair at full orbit spatial file of resolution; Sentinel-1.thus, The theprocessing process step requires is performed intensive memoryon all the and burst I/O data access, for andeach is master one of– theslave most image time-consuming pair at full spatial steps duringresolution Sentinel-1; thus, the image process TOPS-mode requires processing.intensive memory and I/O access, and is one of the most time- consuminAfterg the steps initial during coregistration, Sentinel-1 image the enhanced TOPS-mode spectral processing diversity. (ESD) method [32] is applied to furtherAfter correct the initial the residual coregistration azimuth, coregistrationthe enhanced errors.spectral The diversity algorithm (ESD) exploits method the [32] phase is applied difference to infurther the overlapping correct the residual area among azimuth adjacent coregistration bursts, and errors uses. the The di algorithmfference phase exploit tos retrievethe phase the difference azimuth oinff set.the overlapping First, the ESD area phase among is calculated adjacent bybursts the, doubleand use dis fftheerence difference for every phase overlapping to retrieve area, the azimuth and the ESDoffset. phase First, for the Burst ESD iphaseand j canis calculated be given by as follows:the double difference for every overlapping area, and the ESD phase for Burst i and j can be given as followsn : o = ( ) ϕESD,bibj arg misi∗ mi+1si∗+1 ∗ (1) ∗ ∗ ∗ 휑퐸푆퐷,푏푖푏푗 = 푎푟푔{ (푚푖푠푖 )(푚푖+1푠푖+1) } (1) where m and s refer to the master and slave images, respectively. The relation between the ESD phase where m and s refer to the master and slave images, respectively. The relation between the ESD phase and azimuth offset ∆a can be given as follows: and azimuth offset 훥푎 can be given as follows: ∆훥푎a ϕ휑ESD ==22π휋훥∆푓fd (2) 퐸푆퐷 푑 fa (2) 푓푎 wherewhere ∆훥f푓d푑is is the the Doppler Doppler frequency frequency di differencefference in in the the overlap overlap areas areas between between two two bursts, bursts, and andfa is 푓푎 the is azimuththe azimuth sampling sampling frequency. frequency. Thus, Thus we can, we obtain can obtain the azimuth the azimuth offset offset∆a based 훥푎 onbased Equations on Equations (1) and (2).(1) Theand ESD(2). The algorithm ESD algorithm is implemented is implemented on all the on master–slave all the master pairs.–slave To reducepairs. T theo reduce potential the temporalpotential decorrelationtemporal decorrelation effect of some effect pairs, of some a joint pairs time, a series joint coregistrationtime series coregistration for redundant for image redundant pairs [image33] is usedpairs to[3 obtain3] is used more to accurateobtain more offsets. accurate In this offset work,s the. In followingthis work, pairs the follow are selecteding pairs for ESDare selected calculation, for asESD shown calculation in Figure, as2 shown. in Figure 2.

Figure 2. The selected image pairs for enhanced spectral diversity (ESD) calculation. Figure 2. The selected image pairs for enhanced spectral diversity (ESD) calculation. The ESD algorithm is implemented on the above-selected pairs, and the estimated azimuth offsets Theh ESD algorithm is implemented oni the above-selected pairs, and the estimated azimuth DAz = DAz0 1, DAz0 2, ... , DAz(N 2) (N 1) are obtained. Thus, the azimuth offsets relative to the offsets 푫푨풛 =− [퐷퐴푧 −, 퐷퐴푧 , . . . , 퐷퐴− −푧 − ] are obtained. Thus, the azimuth offsets relative to master image can be0 calculated−1 0−2 by using(푁 the−2)− least(푁−1) squares method: the master image can be calculated by using the least squares method: 1 Az푨풛== ((A푨T푇A푨))−−1A푨T푇 ⋅DAz푫푨풛 (3(3)) · where 푨 is the system matrix defining the relation between the different offsets. By using this where A is the system matrix defining the relation between the different offsets. By using this redundant redundant offset calculation approach, we can obtain more accurate and reliable coregistrated images. offset calculation approach, we can obtain more accurate and reliable coregistrated images. After image coregistration and reramp processing, these burst images will mosaic together in After image coregistration and reramp processing, these burst images will mosaic together in the burst and subswath merging step. Then, the whole coregistrated time series of SLC images are the burst and subswath merging step. Then, the whole coregistrated time series of SLC images are obtained, which can be used in the following interferometry steps. The burst and subswath merging obtained, which can be used in the following interferometry steps. The burst and subswath merging step is the most memory-intensive step because the entire image data with the original resolution step is the most memory-intensive step because the entire image data with the original resolution must must be read into memory. be read into memory.

Remote Sens. 2020, 12, x FOR PEER REVIEW 6 of 19 Remote Sens. 2020, 12, 3749 6 of 20 2.1.2. Parallelization Strategies for Sentinel-1 TOPS-Mode Image Processing Although the Sentinel-1 TOPS-mode has caused some problems to the SAR image coregistration, 2.1.2. Parallelization Strategies for Sentinel-1 TOPS-Mode Image Processing its unique subswath and burst structure has provided good opportunities for parallelizing the processingAlthough algorithm the Sentinel-1s. The parallelization TOPS-mode has strateg causedies somefor this problems step are to straightforward the SAR image,coregistration, and the data- itscenter unique parallel subswath approach and burst is adopted structure, whereby has provided a series good of process opportunities threads for apply parallelizing the same the operation processing on algorithms.the different The data parallelization. strategies for this step are straightforward, and the data-center parallel approachFrom is the adopted, previous whereby algorithm a series description of process, threads the burst apply and the subswath same operation data are on the separated different as data. the independentFrom the image previous files algorithmafter the TOPS description, split step. the T bursthen, d anderamp, subswath geometric data coregistration, are separated ESD as thecoregistration independent, and image reramp files algorithms after the are TOPS all splitimplemented step. Then, on the deramp,se burst geometric images, while coregistration, burst and ESDsubswath coregistration, merging and is performedreramp algorithms on the arethree all implementedsubswath image on theses. Moreover, burst images, both while the burst burst and and subswathsubswath mergingfiles are isalmost performed the same on the in threesize. Thus subswath, the algorithms images. Moreover, during this both processing the burst and can subswath be easily filesparallelized are almost without the same having in size. to consider Thus, the the algorithms problem of during data dependency this processing or load can beunbalancing easily parallelized, leading withoutto these havingalgorithms to consider being execut the problemed in parallel of data on dependency different burst or loador subswath unbalancing, images, leading which to can these be algorithmsintrinsically being distributed executed to inmultiple parallel process on different thread bursts, as or shown subswath in Figure images, 3. which can be intrinsically distributedDuring to this multiple stage, processthe algorithms threads, process as shown the inimage Figure data3. with its original spatial resolution, and they Duringare the thismost stage, memory the- algorithmsand storage process-intensive the steps image in datathe whole with itsprocessing original chain spatial. In resolution, addition, andthe output they are intermediate the most memory- results andare almost storage-intensive three times steps the size inthe of the whole unpacked processing raw chain.Sentinel In-1 addition, images, theand output these massive intermediate I/O operations results are put almost excessive three times pressure the size on the of the hard unpacked disk at rawthe beginning Sentinel-1 and images, the andend these of each massive algorithm I/O operations step, especially put excessive for parallel pressure processing on the hard. Th diskus, the at the speed beginning of disk and access the end has ofbecome each algorithm a major step, limitation especially in parallel for parallel efficiency processing. improvement Thus, the for speed Sentinel of disk-1 access image has TOPS become-mode a majorprocessing limitation. in parallel efficiency improvement for Sentinel-1 image TOPS-mode processing.

FigureFigure 3.3. The parallelismparallelism on the burstburst andand subswathsubswath levellevel forfor Sentinel-1Sentinel-1 TerrainTerrain ObservationObservation withwith ProgressiveProgressive ScansScans (TOPS)-mode(TOPS)-mode imageimage processing.processing.

2.2.2.2. Sentinel-1Sentinel-1 Interferometry 2.2.1. Algorithm Steps for Sentinel-1 Interferometry 2.2.1. Algorithm Steps for Sentinel-1 Interferometry This part aims to generate small baseline multilook interferograms and an average coherence This part aims to generate small baseline multilook interferograms and an average coherence map for time series analysis. The processing steps mainly include interferometric pair selection, map for time series analysis. The processing steps mainly include interferometric pair selection, interferometry processing, removing flat-Earth and topography phases (differential interferometry interferometry processing, removing flat-Earth and topography phases (differential interferometry processing), and multilook filtering. processing), and multilook filtering. First, the interferometric pairs with short temporal and spatial baselines are selected according to First, the interferometric pairs with short temporal and spatial baselines are selected according the given threshold. As Sentinel-1 is characterized by its small baseline, the selected interferometric to the given threshold. As Sentinel-1 is characterized by its small baseline, the selected interferometric pairs by the threshold are still too numerous for later processing. To search the optimally connected pairs by the threshold are still too numerous for later processing. To search the optimally connected baseline network, we estimate the coherence for all interferometric pair candidates by randomly baseline network, we estimate the coherence for all interferometric pair candidates by randomly selecting several blocks from the image and calculating the average coherence of these blocks. To ensure selecting several blocks from the image and calculating the average coherence of these blocks. To the accuracy of coherence estimation, the block size should not be too small (no less than 16 16), ensure the accuracy of coherence estimation, the block size should not be too small (no less than× 16 × and the block size in our experiment is 64 64. Thus, the interferometric pairs with higher coherence 16), and the block size in our experiment× is 64 × 64. Thus, the interferometric pairs with higher are obtained. coherence are obtained. Then, we generate the interferograms by complex conjugate calculation in the interferometry processing for all the interferometric pairs. The external DEM is used to model and remove the flat-Earth

Remote Sens. 2020, 12, x FOR PEER REVIEW 7 of 19

RemoteThen Sens. ,2020 we, 12generate, 3749 the interferograms by complex conjugate calculation in the interferometry7 of 20 processing for all the interferometric pairs. The external DEM is used to model and remove the flat- Earth and topography phase contributions to generate the differential interferograms. Multilook andfiltering topography is the phase spatial contributions complex average to generate for thedifferential differential interferograms interferograms. used Multilook to reduce filtering the is thedecorrelation spatial complex noise average and generate for diff erentialthe spatial interferograms coherence used map to. reduceTo downsample the decorrelation the size noise of andthe generateprocessing the data, spatial 10 coherenceand 2 (or map. 20 and To 4) downsample looks in range the size and of azimuth the processing directions data,, respectively 10 and 2 (or, are 20 andtypically 4) looks used in in range InSAR and processing azimuth directions,. respectively, are typically used in InSAR processing. To save disk storage and reduce II/O/O operations,operations, complex image processing for for interferometry interferometry,, didifferentialfferential interferometry,interferometry, and multilook filteringfiltering can be combined without the need to output intermediate results. results. In In addition, addition, some some interferometry-related interferometry-related parameters parameters (spatial (spatial baseline, baseline, slant range, slant etc.)range, at etc. each) at rangeeach range sample sample and azimuthand azimuth line forline all for the all interferometricthe interferometric pairs pairs are are also also calculated calculated in thisin this part. part.

2.2.2. Parallelization Strategies for Sentinel Sentinel-1-1 Interferometry In the interferometric pair selection step,step, parallel computing can be naturally app appliedlied to the independent selected blocks to calculatecalculate the coherence. Thus Thus,, we mainly focus on parallelization for subsequent interferometry and multilook filteringfiltering processing.processing. TheseThese algorithms are mainly divided into two groups,groups, as shown in Figure4 4.. OneOne isis operatedoperated onon the pixelpixel basis, basis, such such as interferometry, as interferometry, flat-Earth flat and-Earth topography and topography estimation, diestimationfferential interferometry,, differential andinterferometry interferometric, and interferometric parameter calculation. parameter The calculation pixel calculations. The pixel arecalculation independents are independent of each other, of andeach accessother, and to data access from to data other from pixels other is notpixels required. is not required. The other The is other computed is computed on the on window the window basis, whichbasis, which requires requires operation operation on a local on regiona local of region the image, of thesuch image as, multilooksuch as multilook filtering. filtering. The window The processingwindow process also hasingno also dependencies has no dependencies on each other on ineach the calculation. other in the Both calculation. types of algorithmsBoth types areof localalgorithms operators are local with operators a good separation with a good of neighboring separation of units. neighboring units.

(a) (b)

Figure 4. (a()a Pixel) Pixel-based-based algorithm algorithm operation operation (interferometry (interferometry and and differential diﬀerential interferometry) interferometry).. (b) (Windowb) Window-based-based algorithm algorithm operation operation (multilook (multilook filtering). ﬁltering).

Therefore, algorithm parallelization for Sentinel Sentinel-1-1 interferometry processing require requiress little effort effort to partitionpartition thethe processing processing data, data, and and parallel parallel computing computing can can either either be performed be performed pixel p byixel pixel, by pixel or block, or byblock block by block after dividingafter dividing the image the image into diintofferent different blocks. blocks. To make To make use use of file of file reading, reading parallelism, parallelism on theon the image image line line level level is adopted. is adopted. The T SARhe SAR image image files files are readare read by line, by line and, theseand these lines lines of data of data are then are distributedthen distributed to the to di thefferent different process process threads, threads, as shown as shown in Figure in Figure5. 5.

Figure 5. The parallelization strategies for Sentinel Sentinel-1-1 interferometry processing processing..

Remote Sens. 2020, 12, 3749 8 of 20

The algorithm computations are not complex in Sentinel-1 interferometry processing, but they require intensive I/O operations due to the large number of interferograms and interferometry-related parameter ﬁles. Thus, the speed of disk access may inﬂuence the computing time of this processing step.

2.3. Time Series Analysis

2.3.1. Algorithm Steps for Time Series Analysis This part carries out the time series analysis of the interferometric phase for the coherent scatterers, and estimates the deformation results from wrapped interferograms. This processing stage mainly includes coherent scatterer selection, deformation parameter estimation, network integration, residual phase unwrapping, and atmospheric filtering. The high-coherent-scatterer candidates are identified by the average coherence map based on the given threshold, and the interferometric phase and interferometric parameters for these coherent scatterers are extracted from the corresponding files. Then, the following steps are performed on the sparse pixels with less memory and storage consumption. Coherent scatterers are first connected with the Delaunay triangulation network, and the atmospheric phase is mitigated through the double-phase difference of the adjacent pixel at the same arc. Deformation and residual height parameter estimation is performed on all the arcs of the triangulation network through time series analysis of the wrapped phase. For the double-differential phase ∆ϕij for Arc i–j, after removing the atmospheric phase, the residual topography phase ϕtopo, deformation phase ϕde f , and noise ε are remained. The residual topography phase and deformation phase are usually modeled as the known functions of the temporal baseline T and spatial perpendicular baseline B, respectively. Thus, the double-differential phase can be given as follows: 4π ∆ϕ = d T; v + βBh + ε (4) ij λ ij ij ij where vij and hij are the estimated deformation rate and height parameters, respectively. d T; vij is the deformation model, and usually the polynomial or seasonal model. β is the height-to-phase conversion factor. As ∆ϕij is the wrapped phase and cannot be directly calculated by matrix inversion, the parameters are estimated by the following optimization method:

X k 4π ( ( k )+ k ) 1 ∆ϕij λ d T ;vij βB hij γij = max e − (5) v ,h |K | ij ij k where γij is the temporal coherence and K is the interferogram number. The optimal parameters vij and hij can be obtained by searching in the solution space with various optimization techniques [34]. At the same time, some poorly estimated arc results will be rejected from the network if the temporal coherence γij is below the given threshold. Finally, the reliable coherent scatters are selected in this step. After estimating the deformation and height parameters for all the arcs in the network, the diﬀerential results are integrated by the network adjustment. To improve the inversion results of the adjustment matrix with the least squares solution, a weighted ridge-estimator is adopted to obtain more robust results by adding a regularization constraint, given as follows:

1 V = (ATWA + σI)− ATWD (6) where A is the adjustment matrix that defines the relation of the arcs. W is the weighted matrix, and the diagonal elements are the temporal coherence for the arcs; σ is a positive constant of the ridge-estimator parameter; D is the differential results of the estimated deformation rate or height for every arc; and V is the final estimated result with respect to the reference pixel. Thus, the linear deformation rate and residual height results can be obtained from Equation (6). Then, a series of residual phases are acquired by subtracting the estimated linear deformation and height phase from the differential phase, and the phase unwrapping is then used to retrieve the Remote Sens. 2020, 12, 3749 9 of 20 Remote Sens. 2020, 12, x FOR PEER REVIEW 9 of 19 integer phase ambiguity for residual interferograms before estimating the atmospheric and nonlinear deformation phase.phase. This process is performed by applying a two two-dimensional-dimensional spatial minimum cost flowflow (MCF)(MCF) phasephase unwrapping unwrapping algorithm algorithm [35 [],35 which], which is one is one of the of mostthe most demanding demanding steps steps in time in series time analysis.series analysis. The unwrapped The unwrapped residual interferograms residual interferogra∆ϕ forms the selected훥흋 for small the selected baseline interferometricsmall baseline pairsinterferometric are obtained pairs after are unwrapping obtained after processing, unwrapping and they processing can be expressed, and they by can the be following expressed function: by the following function: ∆ϕ = Bϕ (7) 훥흋 = 푩흋 (7) wherewhere ϕ흋 isis thethe residual-phaseresidual-phase timetime series with respect to to the the first first acquisition acquisition time time 푡t11,, and and 푩B isis thethe system matrix that defines defines the relation of the interferometric pairs. The The solution solution for for 흋ϕ can be seen as an L2-normL2-norm minimizingminimizing problem.problem.The The singular singular value value decomposition decomposition (SVD) (SVD) method method is implementedis implemented to T solveto solv thise this minimizing minimizing model model by by decomposing decomposing the the systemsystem matrix B푩 asasB 푩==USV푼푺푽푇,, while while thethe systemsystem matrix is common for all coherent scatterers. Thus Thus,, the residual residual-phase-phase time series relative to the firstfirst acquisition time t푡11are are acquired acquired in in this this step. step. To estimate and remove the atmospheric phase contribution from the residual phase time series, the atmospheric atmospheric filtering filtering is performed is performed based based on the spatial–temporal on the spatial–temporal characteristics characteristics of the atmospheric of the phase.atmospheric The atmospheric phase. The phase atmospheric time series phase can be time estimated series canby temporal be estimated high-pass by temporal filtering and high spatial-pass [[ ] ] low-passfiltering and filtering spatialϕ lowres HP-pass_time filteringLP_space. Thus,[[휑푟푒푠 removing]퐻푃_푡푖푚푒]퐿푃 the_푠푝푎푐푒 atmospheric. Thus, removing phase fromthe atmosphe the residual-phaseric phase timefrom seriesthe residual and the-phase final time displacement series and time the final series displacement can be obtained. time series Once can all thebe obtained. frames have Once been all processed,the frames mosaickinghave been processed, of the results mosaicking is carried of outthe toresults generate is carried the deformation out to generat resultse thefor deformation the entire studyresults regions for the entire by employing study region the overlappings by employing regions the overlapping among diff erentregions frames among or different by integrating frames the or GPSby integrating measurements. the GPS measurements.

2.3.2. Parallelization Strategies for Time Series Analysis From the above processing steps,steps, we can see that multiple types of algorithms are involved in the timetime series analysis,analysis, including the temporal dimension algorithm (deformation estimation step step),), spatial dimension algorithm (network integration and phase unwrapping step step),), and temporal temporal–spatial–spatial hybrid dimension dimension algorithm algorithm (atmospheric (atmospheric filtering filtering step). step Thus,). very Thus di, ffveryerent different parallelization parallelization strategies arestrategies adopted are to adopted address to these address different these algorithmic different algorithmic structures. structures. The conventionalconventional process proces ofs directlyof directly connecting connecting millions millions of coherent of scatterers coherent with scatterers a triangulation with a networktriangulation will lead network to solving will lead a large-scale to solving matrix a large in-scale network matrix integration; in network thus, integration the computation; thus, the in thiscomput stepation is of in low this e ffistepciency. is of low To reduceefficiency the. To computing reduce the scale computing and optimize scale and the optimize network the calculation, network parallelizationcalculation, parallelization is realized by is splittingrealized theby imagesplitting into the di imagefferent into blocks, different and block-level blocks, and parallelism block-level is adoptedparallelism in thisis adopted step. The in this image step is. T splithe image based is on split the based density on of the the density coherent of the scatterer coherent to balancescatterer the to computingbalance the load; computing thus, a series load; ofthus unequal-sized, a series of blocksunequal are-sized obtained. blocks As are a result, obtained the. originalAs a result, network the isoriginal replaced network by several is replaced small-scale by several networks, small- whichscale network means thats, which the large-scale means that matrix the large is decomposed-scale matrix intois decomposed a series of submatrices. into a series Inof addition,submatrices the. calculations In addition, for the these calculations small submatrices for these small can run submatrices in parallel oncan the run di ff inerent parallel process on threads,the different as shown process in Figure threads,6. In as this shown way, the in computational Figure 6. In this e ffi way,ciency the of networkcomputational integration efficiency can be of greatlynetwork improved. integration can be greatly improved.

Figure 6.6. The parallelization strategy for network integration on the block basis.

Remote Sens. 2020, 12, 3749 10 of 20 Remote Sens. 2020, 12, x FOR PEER REVIEW 10 of 19

AfterAfter establishingestablishing thethe network,network, thethe deformationdeformation estimationestimation cancan bebe carriedcarried outout forfor thethe wrappedwrapped phasephase ofof allall arcsarcs onon the the temporal temporaldomain. domain. AsAs thethe calculationscalculations amongamong didifferentfferent arcsarcs areare independentindependent ofof eacheach other,other, thethe arc-basedarc-based parallelizationparallelization strategystrategy isis adoptedadopted inin thisthis stage,stage, andand thethe deformationdeformation estimationestimation forfor thesethesearcs arcs runsruns in in parallel parallel on on the the di differentfferent process process threads. threads. TheThe nextnext stepstep isis phasephaseunwrapping unwrapping forforresidual residual interferograms. interferograms. TheThe MCFMCF phasephase unwrappingunwrapping algorithmalgorithm is is a globala global optimization optimization method, method, and the and diff erent the different parts of the parts image of have the strongimage dependencies have strong underdependencies the global under constraint. the global To avoidconstraint potential. To avoid unwrapping potential errors unwrapping by improper errors imageby improper partitioning, image thepartitioning task-parallel, the methodtask-parallel is adopted method by is adopted distributing by distribut the wrappeding the residual wrapped images residual to images the diff erentto the processdifferent threads, process and threads the MCF, and algorithm the MCF operatesalgorithm on operates these process on the threadsse process in parallel threads to in complete parallel all to thecomplete tasks, asall shown the tasks in, Figure as shown7. in Figure 7. Then,Then, the the unwrapped unwrapped residual residual interferograms interferogram follows follow up with up SVD with calculation, SVD calculation, which is conducted which is forconducted each coherent for each scatterer coherent in scatterer the temporal in the domain temporal independently; domain independently thus, its parallelization; thus, its parallelization strategy isstrategy very similar is very to similar the Sentinel-1 to the Sentinel interferometry-1 interferometry by parallel by computing parallel computing on a pixel on basis. a pixel The basis. final step The isfinal atmospheric step is atmospheric filtering, which filtering, consists which of consists two-step of spatial two-step and spatial temporal and operations. temporal operation The temporals. The operationtemporal isoperation the same is for the SVD same calculation for SVD as calculation a pixel operator as a pixel in the op temporalerator in domainthe temporal and the domain pixel-level and parallelismthe pixel-level is adopted, parallelism while is the adopted, spatial filteringwhile the can spatial use the filtering window-based can use parallelthe window computing-based methodparallel withcomputing parallelism method similar with to parallelism the multilook similar filtering. to the multilook filtering. TimeTime series series analysis analysis is is one one of of the the most most complicated complicated steps steps in Sentinelin Sentinel InSAR InSAR processing processing due due to the to temporal–spatialthe temporal–spatial three-dimensional three-dimensional structure structure data, data and, and the timethe time consumption consumption in this in this step step is close is close to thatto that of Sentinel-1 of Sentinel image-1 image coregistration. coregistration Moreover,. Moreover, the time the seriestime series analysis analysis step deals step withdeals sparse with spar pixelsse andpixels has and smaller has smaller I/O operation I/O operation and storage and storage requirements requirements than the than previous the previous steps. steps.

Figure 7. The parallelism on the image level for phase unwrapping processing. Figure 7. The parallelism on the image level for phase unwrapping processing. 3. Experiments and Discussion 3. Experiments and Discussion 3.1. Study Area and Dataset 3.1. Study Area and Dataset We apply the proposed P-CSInSAR method in the previous section to the North China Plain, with threeWe apply frames the of proposed Sentinel-1 P SLC-CSInSAR images method along the in samethe previous track (142-1, section 142-2, to the and North 142-3). China TheNorth Plain, Chinawith three Plain frames has one of ofSentinel the largest-1 SLC subsidence images along bowls the in same the world,track (142 and-1, most 142- of2, and the cities142-3) around. The North this regionChina havePlain suhasﬀered one seriousof the largest ground subsidence subsidence bowl problems,s in the whichworld, have and most become of the cities major around geological this hazardsregion have inhibiting suffered the serious sustainability ground of subsidence regional development. problems, which The have three become frames ofthe Sentinel-1 major geological images havehazards been inhibiting acquired the during sustainability the September of regional 2018–March development. 2020 time The span, three composed frames of of Sentinel 135 acquisitions,-1 images withhave descendingbeen acquired track during 142. the The September study region 2018– hasMarch covered 2020 time Beijing, span Tianjin,, composed Hebei, of 135 and acquisitions, Shandong provinceswith descending with an trackoverall 142. area The of more study than region 120,000 has covered km2, and Beijing, the location Tianjin, of this Hebei region, and in Shandong China is shownprovinces in Figure with 8ana withoverall red area boxes. of more The outlines than 120,000 of the km Sentinel-12, and the frames location are shownof this inregion Figure in8 Chinab with is blackshown boxes, in Figure and the8a with GNSS red stations boxes. ofThe the outlines China Earthquakeof the Sentinel Administration-1 frames are areshown shown in Figure as red 8b dots. with black boxes, and the GNSS stations of the China Earthquake Administration are shown as red dots.

Remote Sens. 2020, 12, 3749 11 of 20 Remote Sens. 2020, 12, x FOR PEER REVIEW 11 of 19 Remote Sens. 2020, 12, x FOR PEER REVIEW 11 of 19

(a) (b) (a) (b) Figure 8. (a) The locations of the study regions on the China map are shown in red boxes; (b) the FigureFigure 8. 8. ((aa)) T Thehe locationslocations ofof the the study study regions regions on on the the China China map map are shownare shown in red in boxes; red boxes (b) the; ( outlinesb) the outlines of Sentinel-1 frames for track 142 are shown in the black boxes, superimposed on the outlinesof Sentinel-1 of Sen framestinel- for1 frames track 142 for are track shown 142 in are the shown black boxes, in the superimposed black boxes, on superimposed the topography on image, the topography image, and the red dots are the GNSS stations of the China Earthquake Administration. topographyand the red image, dots are and the the GNSS red dots stations are ofthe the GNSS China stations Earthquake of the Administration.China Earthquake Administration. 3.2. The Comparsion Experiment 3.2. The Comparsion Experiment In this experiment, we compare our methodmethod with SNAPSNAP/GMTSAR/GMTSAR - GIANT software packages In this experiment, we compare our method with SNAP/GMTSAR - GIANT software packages over thethe SceneScene 142-2142-2 with with 45 45 images. images. The The P-CSInSAR P-CSInSAR method method is is ﬁrst first applied applie ind in this this region. region. The The image image at over the Scene 142-2 with 45 images. The P-CSInSAR method is first applied in this region. The image timeat time 20180912 20180912 is the is master the master image. After image. image After coregistration, image coregistration, 95 interferograms 95 interferograms with the perpendicular with the at time 20180912 is the master image. After image coregistration, 95 interferograms with the baselineperpendicular less than baseline 180 m, temporalless than baseline 180 m, lesstemporal than 50 baseline days, and less with than higher 50 coherence days, and are with generated, higher perpendicular baseline less than 180 m, temporal baseline less than 50 days, and with higher andcoherence the multilook are generated, numbers and are the 10 multilook and 2 in the numbers range and are azimuth 10 and 2 direction, in the range respectively. and azimuth Through direction time, coherence are generated, and the multilook numbers are 10 and 2 in the range and azimuth direction, seriesrespectively analysis,. Through more than time 1.8 series million analysis, scatterers more are than selected 1.8 million and the scatterers temporal are coherence selected threshold and the respectively. Through time series analysis, more than 1.8 million scatterers are selected and the istemporal set to 0.7. coherence As for thethreshold second is method, set to 0.7. SNAP As for or the GMTSAR second canmethod, only SNAP perform or theGMTSAR Sentinel-1 can SARonly temporal coherence threshold is set to 0.7. As for the second method, SNAP or GMTSAR can only imageperform coregistration the Sentinel- and1 SAR interferometry, image coregistration and a total and of interferometry, 101 interferograms and area total generated of 101 interferograms with the same perform the Sentinel-1 SAR image coregistration and interferometry, and a total of 101 interferograms baselineare generated threshold with and the multilook same baseline numbers. threshold Then, and GIANT multilook is used numbers. to perform Then the, time GIANT series is analysis,used to are generated with the same baseline threshold and multilook numbers. Then, GIANT is used to andperform the ERA5the time atmospheric series analysis, model and is selected the ERA5 in the atmospheric atmospheric model phase is correction.selected in Thethe deformationatmospheric perform the time series analysis, and the ERA5 atmospheric model is selected in the atmospheric resultsphase correction. by two methods The deformation are shown with results the by same two colorbar methods in are Figure shown9, and with the the processing same colorbar times for in phase correction. The deformation results by two methods are shown with the same colorbar in diFigureﬀerent 9, stepsand the are processing listed in Table times1. for different steps are listed in Table 1. Figure 9, and the processing times for different steps are listed in Table 1.

(a) (b) (a) (b) Figure 9. The deformation rate for Scene 142-2 estimated by P-CSInSAR (a) and GIANT (b). FigureFigure 9 9.. TheThe deformation deformation rate rate for for Scene Scene 142 142-2-2 estimated estimated by by P P-CSInSAR-CSInSAR (a (a) )and and GIANT GIANT (b (b).).

Remote Sens. 2020, 12, 3749 12 of 20

Table 1. Processing time for Scene 142-2 with diﬀerent software packages (h: Hour, d: Day).

Sentinel-1 TOPS-Mode Sentinel-1 Time Series Process Step Total Image Processing Interferometry Analysis SNAP 117.6 h (4.9 d) - - GMTSAR 79.2 h (3.3 d) - - GIANT - 72.1 h (3 d) 7.9 d/6.3 d P-CSInSAR 6.2 h 2.1 h 4.9 h 13.2 h

From Figure9, we can see that the deformation results estimated by our method are very similar to the results by GIANT, and the major subsidence regions can be identified from both deformation maps. The small differences between two deformation rate maps are mainly caused by the atmospheric effect and the different interferograms selected in two methods. As for the processing time, it can be seen from Table1 that the P-CSInSAR method has greatly improved the e fficiency of Sentinel-1 InSAR processing, reducing a week’s work of conventional processing to 14 hours, and the processing for all the three frames can be finished in several days. The highly automatic processing chain saves considerable human and time costs.

3.3. Results In this experiment, the three frames of Sentinel-1 images are processed by the P-CSInSAR method, and the total number of Sentinel-1 acquisitions is 135. The image at time 20180912 is taken as the master image, and all the other images are coregistrated to this reference image. TanDEM-X (90 m) is used in the geometric coregistration and differential interferometry processing. The interferometric pairs are selected with a small temporal baseline of 50 days and a small spatial baseline of 180 m, and they are further selected according to the estimated coherence. As a result, the total number of the interferograms for the three frames is 310. We multilook each interferogram with 10 and 2 looks in the range and azimuth direction, respectively, followed by smooth filtering with a window length of 8 pixels. The coherence threshold for the average coherence map to select coherent scatterers is set to 0.7, and more than 6.5 million scatterers are identified. Through the time series analysis of the stack of interferograms, the deformation products are finally acquired, including the mean deformation rate and displacement time series, with a total of 5.4 million measurements. The workflow is automatically executed based on a Bash script. After finishing all the InSAR processing, a fusion operation is carried out to merge the deformation results for three frames, and the mean LOS velocity map for the whole area is displayed in Figure 10. In Figure 11, we can see very serious subsidence in most of the cities around this region, which has been extensively investigated by many researchers, and our results have shown similar subsidence patterns with these previous works [36–39]. Several subsidence centers can be identified from the mean velocity map. Among them, Beijing-Tianjin regions have a deformation rate more than 60 mm/year, including the Tongzhou, Langfang, and Tianjin district, and subsidence in this region has shown a trend of gradually growing together. To illustrate the InSAR deformation results in the Beijing-Tianjin region with the geodetic measurements [40], both results are shown in Figure 11. Remote Sens. 2020, 12, x FOR PEER REVIEW 12 of 19

Table 1. Processing time for Scene 142-2 with different software packages (h: Hour, d: Day).

Process Sentinel-1 TOPS-Mode Sentinel-1 Time Series Total Step Image Processing Interferometry Analysis SNAP 117.6 h (4.9 d) - - GMTSAR 79.2 h (3.3 d) - - 7.9 GIANT - 72.1 h (3 d) d/6.3 d P-CSInSAR 6.2 h 2.1 h 4.9 h 13.2 h

From Figure 9, we can see that the deformation results estimated by our method are very similar to the results by GIANT, and the major subsidence regions can be identified from both deformation maps. The small differences between two deformation rate maps are mainly caused by the atmospheric effect and the different interferograms selected in two methods. As for the processing time, it can be seen from Table 1 that the P-CSInSAR method has greatly improved the efficiency of Sentinel-1 InSAR processing, reducing a week’s work of conventional processing to 14 hours, and the processing for all the three frames can be finished in several days. The highly automatic processing chain saves considerable human and time costs.

Remote Sens. 2020, 12, x FOR PEER REVIEW 13 of 19

Figure 10. The mean LOS velocity (mm/year) for the North China plain with descending track 142, from 12 September 2018 to 29 March 2020.

In Figure 11, we can see very serious subsidence in most of the cities around this region, which has been extensively investigated by many researchers, and our results have shown similar subsidence patterns with these previous works [36–39]. Several subsidence centers can be identified from the mean velocity map. Among them, Beijing-Tianjin regions have a deformation rate more than 60 mm/year, including the Tongzhou, Langfang, and Tianjin district, and subsidence in this region Figure 10. has shown a trendThe meanof gradually LOS velocity growing (mm /together.year) for theTo Northillustrate China the plain InSAR with deformation descending trackresults 142, in the from 12 September 2018 to 29 March 2020. Beijing-Tianjin region with the geodetic measurements [40], both results are shown in Figure 11.

Figure 11 11.. TheThe distribution distribution of deep of deep and andshallow shallow ground ground water, water, the major the majorsubsidence subsidence region in region Beijing in- Beijing-Tianjin,Tianjin, and the and deformation the deformation results resultsestimated estimated by our bymethod our method..

From Figure 1111,, the the subsidence subsidence in in Tongzhou Tongzhou and and Chaoyang Chaoyang district districtss of Beijing of Beijing and and Langfang Langfang are arelocated located in the in same the same areas areas with with the geological the geological investigations investigations,, and the and main the mainreason reason in this in region this region is the isexpansion the expansion of cities of cities and populations. and populations. The subsidence The subsidence in the in central the central urban urban areas areas of Tianjin of Tianjin is not is notsignificant signiﬁcant,, because because strict strict groundwater groundwater protection protection measures measures have have been been enforced enforced in recent in recent years years.. In Inaddition, addition, the the most most serious serious region region in in Tianjin Tianjin is is located located in in Wuqing Wuqing district district,, with with a a maximum maximum subsidence more than than 110 110 mm, mm while, while the the subsidence subsidence in North in North Xiongxian Xiongxian County County (38.9◦ N, (38.9 116.2°N,◦ E)116. is2 closely°E) is relatedclosely torelated the shallow to the groundwater shallow groundwater funnel. Therefore, funnel. theTherefore, deformation the deformation results estimated results by estimatedour method by agree our wellmethod with agree the geodetic well with measurements. the geodetic measurements. Due to to the the overexploitation overexploitation of underground of underground water for water agricultural for agricultural and domestic and usage, domestic subsidence usage, insubsidence Hebei province in Hebei is relatively province severe is relatively and complex. severe From and complex Figure 10. ,From we can Figure see many 10, strongwe can subsidence see many bowlsstrong insubsidence central Hebei, bowls and in central the subsidence Hebei, and rates the insubsidence some areas rate exceeds in some 120 areas mm/year. exceed To 120 validate mm/year. the InSARTo validate deformation the InSAR results deformation in this region, results the in groundwater-levelthis region, the groundwater measurements-level inmeasurement Cangzhou ands in HengshuiCangzhou urbanand Hengshui areas are urban compared areas withare compared InSAR displacements with InSAR displacements from January from 2019 Jan to July2019 2019 to July in Figure2019 in 12 Figurea,b respectively. 12a,b respectively.

(a) (b)

Remote Sens. 2020, 12, x FOR PEER REVIEW 13 of 19

Figure 10. The mean LOS velocity (mm/year) for the North China plain with descending track 142, from 12 September 2018 to 29 March 2020.

In Figure 11, we can see very serious subsidence in most of the cities around this region, which has been extensively investigated by many researchers, and our results have shown similar subsidence patterns with these previous works [36–39]. Several subsidence centers can be identified from the mean velocity map. Among them, Beijing-Tianjin regions have a deformation rate more than 60 mm/year, including the Tongzhou, Langfang, and Tianjin district, and subsidence in this region has shown a trend of gradually growing together. To illustrate the InSAR deformation results in the Beijing-Tianjin region with the geodetic measurements [40], both results are shown in Figure 11.

Figure 11. The distribution of deep and shallow ground water, the major subsidence region in Beijing- Tianjin, and the deformation results estimated by our method.

From Figure 11, the subsidence in Tongzhou and Chaoyang districts of Beijing and Langfang are located in the same areas with the geological investigations, and the main reason in this region is the expansion of cities and populations. The subsidence in the central urban areas of Tianjin is not significant, because strict groundwater protection measures have been enforced in recent years. In addition, the most serious region in Tianjin is located in Wuqing district, with a maximum subsidence more than 110 mm, while the subsidence in North Xiongxian County (38.9°N, 116.2°E) is closely related to the shallow groundwater funnel. Therefore, the deformation results estimated by our method agree well with the geodetic measurements. Due to the overexploitation of underground water for agricultural and domestic usage, subsidence in Hebei province is relatively severe and complex. From Figure 10, we can see many strong subsidence bowls in central Hebei, and the subsidence rates in some areas exceed 120 mm/year. To validate the InSAR deformation results in this region, the groundwater-level measurements in

CangzhoRemote Sens.u and2020, Hengshui12, 3749 urban areas are compared with InSAR displacements from Jan 2019 to14 July of 20 2019 in Figure 12a,b respectively.

(a) (b)

Figure 12. The comparisons between interferometric synthetic aperture radar (InSAR) displacement (pink circle) and the deep groundwater level changes (blue diamond) for Cangzhou (a) and Hengshui (b), from January 2019 to July 2019.

From Figure 12, we can see that the groundwater levels have remained in decline from January to June 2019, and then rise in July due to the increase in rainfall. The InSAR displacements have also shown the same trend with the groundwater level changes in the two places, which has proven that our deformation results are reasonable. The major subsidence areas of Hebei province are located in the small urban and rural areas, and we determine and analyze 12 strong individual subsidence bowls from Figure 10. To illustrate the spatial distribution of these subsidence bowls, a paraboloid interpolation is performed to generate a 3D model for the subsidence centers, as shown in Figure 13. The widely distributed subsidence areas in Hebei province have posed a great threat to the safety of transportation and infrastructure; thus, the proper measures should continue to be taken by the local government to protect underground water resources. Moreover, based on the subsidence bowl distribution in Figure 13, we can make targeted protection policies by restricting and prohibiting groundwater exploitation in the most serious subsidence locations. To better-validate the accuracy of the proposed method, we compare the estimated deformation results with GPS measurements. We exploit Continuous GPS coordinate time series from the China Earthquake Administration, and the GPS measurements are processed by GAMIT Software [41]. The locations of the three GNSS stations have been shown in Figure8b with red dots, and we prepare the GPS daily measurements with the same time span as Sentinel-1 acquisitions. While two of the GPS measurements are available from 20180901 to 20200430, the other one only provides the measurements from 20180901 to 20190630. The GPS displacement time series have been transformed into the LOS direction, and both the GPS and InSAR time series are relative to the reference image at 20180912. The comparison between GPS measurements and InSAR results for three stations are shown in Figure 14a–c in red stars and black triangles, respectively. In addition, some of the InSAR results of the displacement time series are removed based on the residual phase variation due to the strong atmospheric eﬀect, such as the images at 20190510 and 20190802. Remote Sens. 2020, 12, x FOR PEER REVIEW 14 of 19

From Figure 12, we can see that the groundwater levels have remained in decline from Jan to June 2019, and then rise in July due to the increase in rainfall. The InSAR displacements have also shown the same trend with the groundwater level changes in the two places, which has proven that our deformation results are reasonable. The major subsidence areas of Hebei province are located in the small urban and rural areas, and we determine and analyze 12 strong individual subsidence bowls from Figure 10. To illustrate Remotethe spatial Sens. 2020 distribution, 12, 3749 of these subsidence bowls, a paraboloid interpolation is performed to generate15 of 20 a 3D model for the subsidence centers, as shown in Figure 13.

Figure 13. The subsidence center and its 3D subsidence bowl distribution for 12 counties in Hebei region, and (a–l) Lixian, Anguo, Hejian, Shenzhou, Mengchun, Jingxian, Julu, Pingxiang, Weixian, Qinghe, Qucheng, and Guantao, respectively.

Remote Sens. 2020, 12, x FOR PEER REVIEW 15 of 19

The widely distributed subsidence areas in Hebei province have posed a great threat to the safety of transportation and infrastructure; thus, the proper measures should continue to be taken by the local government to protect underground water resources. Moreover, based on the subsidence bowl distribution in Figure 13, we can make targeted protection policies by restricting and prohibiting groundwater exploitation in the most serious subsidence locations. To better-validate the accuracy of the proposed method, we compare the estimated deformation results with GPS measurements. We exploit Continuous GPS coordinate time series from the China Earthquake Administration, and the GPS measurements are processed by GAMIT Software [41]. The locations of the three GNSS stations have been shown in Figure 8b with red dots, and we prepare the GPS daily measurements with the same time span as Sentinel-1 acquisitions. While two of the GPS measurements are available from 20180901 to 20200430, the other one only provides the measurements from 20180901 to 20190630. The GPS displacement time series have been transformed into the LOS direction, and both the GPS and InSAR time series are relative to the reference image at 20180912. The comparison between GPS measurements and InSAR results for three stations are shown in Figure 14a–c in red stars and black triangles, respectively. In addition, some of the InSAR Remoteresults Sens. of 2020the ,displacement12, 3749 time series are removed based on the residual phase variation due16 to of the 20 strong atmospheric effect, such as the images at 20190510 and 20190802.

(a)

(b)

(c)

Figure 14. GNSS displacement time series in the LOS direction (red stars) compared with InSAR time series (black triangles) for the three GNSS stations (a–c), respectively. The geographical locations of three GNSS stations are in the top middle of the ﬁgures.

In Figure 14, we can see that the deformation results estimated by the proposed method have shown good agreement with GPS measurements, and the standard deviation between GNSS and InSAR time series is less than 0.8 cm. Thus, the comparison results have proven that the P-CSInSAR method can retrieve the deformation results for Sentinel-1 IW SAR images with very high precision.

3.4. Performance Analysis The computation platform used in the experiment is an Intel Xeon Gold 6129 Processor with 64 cores, 256 GB memory, and 10 TB shared storage, accessible through a network file system with 1 GB/s bandwidth. The size of the input image data is approximately 0.9 TB, and the storage required for the total intermediate files and final results is more than 4 TB. The processing time for the entire process is approximately 41 h, with the three steps of the previous section accounting for 49%, 15%, and 36%, respectively. To further evaluate and quantify the performance of the parallel efficiency and the scalability of the proposed method, a speedup analysis is carried out for the different processing steps in the workflow with an increasing number of computing cores, which is displayed in Figure 15. The ideal achievable speedup (the ideal speedup value is equal to the number of cores) is shown in red Remote Sens. 2020, 12, x FOR PEER REVIEW 16 of 19

3.4. Performance Analysis The computation platform used in the experiment is an Intel Xeon Gold 6129 Processor with 64 cores, 256 GB memory, and 10 TB shared storage, accessible through a network ﬁle system with 1 GB/s bandwidth. The size of the input image data is approximately 0.9 TB, and the storage required for the total intermediate files and final results is more than 4 TB. The processing time for the entire process is approximately 41 h, with the three steps of the previous section accounting for 49%, 15%, and 36%, respectively. To further evaluate and quantify the performance of the parallel efficiency and Remotethe scalability Sens. 2020, 12of, 3749the proposed method, a speedup analysis is carried out for the different processing17 of 20 steps in the workflow with an increasing number of computing cores, which is displayed in Figure 15. The ideal achievable speedup (the ideal speedup value is equal to the number of cores) is shown squares, and the experimental speedup for the three steps is shown in blue squares in Figure 15a–c. in red squares, and the experimental speedup for the three steps is shown in blue squares in Figure Figure 15d shows the speedup for the overall processing chain. 15a–c. Figure 15d shows the speedup for the overall processing chain.

(a) (b)

FigureFigure 15.15. SpeedupSpeedup achievedachieved byby thethe proposedproposed methodmethod (in(in blueblue squares)squares) withwith increasingincreasing numbernumber ofof corescores forfor ((aa)) Sentinel-1Sentinel-1 TOPS-modeTOPS-mode imageimage processingprocessing step;step; ((bb)) Sentinel-1Sentinel-1 interferometryinterferometry step;step; ((cc)) timetime seriesseries analysisanalysis step;step; andand ((dd)) thethe overalloverall processingprocessing chain.chain. TheThe idealideal achievableachievable speedupspeedup isis shownshown inin redred squares.squares.

InIn FigureFigure 1515,, wewe cancan seesee thatthat thethe performanceperformance ofof thethe proposed methodmethod hashas enhancedenhanced greatlygreatly withwith increasingincreasing computingcomputing resources, resources, but but it it has has some some gaps gaps with with the the ideal ideal linear linear speedup. speedup. For For the the Sentinel-1 Sentinel- TOPS-mode image processing step in Figure 15a, the parallel computing efficiency is improved with an increasing number of cores based on burst-level parallelism. When the cores exceed the burst numbers, the algorithm efficiency shows very little improvement. Moreover, due to the large size of the Sentinel-1 image with the original resolution, the processing time has been slowed by intensive I/O operations. For the Sentinel-1 interferometry step in Figure 15b, the size of the multilook image is much smaller than that in the first step; thus, the algorithm efficiency can increase with a near-linear trend despite the massive I/O operations as the core numbers increased. In the time series analysis step, the block numbers in the data partition and the nonparallel computing parts (such as network integration) have greatly influenced the parallel efficiency, and we can see that the experimental speedup is increasing not as fast as the ideal speedup in Figure 15c. Finally, as seen in Figure 15d, the speedup for the overall processing chain demonstrates the efficiency and scalability of the proposed P-CSInSAR method.

4. Conclusions Due to the increasing interest in large-scale InSAR applications in recent years, we present a novel parallel computing-based P-CSInSAR method to tackle the challenge of massive Sentinel-1 InSAR data processing. Remote Sens. 2020, 12, 3749 18 of 20

In this work, we first introduce the conventional MT-InSAR algorithms, and then analyze their algorithm parallelizability for the whole processing chain. By adopting flexible parallelization strategies based on different data or algorithmic structures, P-CSInSAR has achieved high parallelization efficiency. To validate the effectiveness and accuracy of the proposed method, P-CSInSAR has been tested on the real Sentinel-1 data over the North China Plain with three frames of Sentinel-1 images in the same track. Compared with the results over the second frame dataset by open-source software packages, the P-CSInSAR method can obtain the same deformation pattern as the other software, while it outperforms the latter on the processing time with 14 h relative to one week, which has reduced much of the time and human cost. Moreover, the results processed by P-CSInSAR for the three frames of Sentinel-1 data are in good agreement with the geodetic measurements, and the mean standard deviation of GNSS and InSAR time series are less than 0.8 cm. These experimental results demonstrate that P-CSInSAR has not only obtained deformation results with high accuracy, but also has high efficiency and scalability, which can provide practical and effective technical means for local governments in urban planning and disaster monitoring. However, the proposed P-CSInSAR method cannot perform multi-task parallel processing, and has to finish different frames of Sentinel-1 data processing sequentially. Moreover, the parallelism of some algorithms is limited by data structures, and cannot achieve higher parallelization efficiency, such as the SAR image coregistration algorithm. Future work will migrate P-CSInSAR to cloud computing or supercomputing environments, and both multi-node and multi-thread programming will be employed to further improve the efficiency of large-scale Sentinel-1 InSAR processing. Some computationally intensive tasks in the P-CSInSAR method will be decomposed into subproblems and completed in different nodes by task scheduling. In addition, to better-exploit the computing resources of HPC, the processing steps of P-CSInSAR will be redesigned to reduce the I/O operation by eliminating the storage need for some intermediate results. This novel HPC-based InSAR big data processing platform will be applied in the routine monitoring widespread subsidence in the North China Plain and other national-scale geodynamic process investigations.

Author Contributions: Conceptualization, validation and investigation, W.D., Y.T. and H.Z.; resources, H.Z. and Y.T.; writing, W.D. and H.Z.; supervision, H.Z. and C.W.; project administration, H.Z. and C.W.; funding acquisition, C.W. All authors have read and agreed to the published version of the manuscript. Funding: This work was supported by the National Natural Science Foundation of China under Grants 41930110. Acknowledgments: The authors would like to thank the Copernicus programme for the free access to Sentinel-1 images processed in this analysis. The authors would also like to thank the GNSS data product service platform of China Earthquake Administration for providing GPS data. The 90 m TanDEM-X DEM was provided by DLR; and GACOS products were provided by Newcastle University at http://ceg-research.ncl.ac.uk/v2/gacos. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References

1. Hanssen, R.F. Radar Interferometry: Data Interpretation and Error Analysis; Springer Science & Business Media: Berlin, Germany, 2001. 2. Ferretti, A.; Prati, C.; Rocca, F. Nonlinear subsidence rate estimation using permanent scatterers in diﬀerential SAR interferometry. IEEE Trans. Geosci. Remote Sens. 2000, 38, 2202–2212. [CrossRef] 3. Ferretti, A.; Prati, C.; Rocca, F. Permanent Scatters in SAR Interferometry. IEEE Trans. Geosci. Remote Sens. 2001, 39, 8–20. [CrossRef] 4. Berardino, P.; Fornaro, G.; Lanari, R.; Sansosti, E. A new algorithm for surface deformation monitoring based on small baseline diﬀerential SAR interferograms. IEEE Trans. Geosci. Remote Sens. 2002, 40, 2375–2383. [CrossRef] 5. Osmanoglu, B.; Sunar, F.; Wdowinski, S. Time series analysis of InSAR data: Methods and trends. J. Photogramm. Remote Sens. 2016, 115, 90–102. [CrossRef] 6. Hooper, A.J.; Bekaert, D.; Spaans, K.; Arikan, M. Recent advances in SAR interferometry time series analysis for measuring crustal deformation. Tectonophysics 2012, 514–517, 1–13. [CrossRef] Remote Sens. 2020, 12, 3749 19 of 20

7. Intrieri, E.; Raspini, F.; Fumagalli, A.; Lu, P.; Del Conte, S.; Farina, P.; Allievi, J.; Ferretti, A.; Casagli, N. The Maoxian landslide as seen from space: Detecting precursors of failure with Sentinel-1 data. Landslides 2018, 15, 123–133. [CrossRef] 8. Hussain, E.; Wright, T.J.; Walters, R.J.; Bekaert, D.P.S.; Lloyd, R.; Hooper, A. Constant strain accumulation rate between major earthquakes on the North Anatolian Fault. Nat. Commun. 2018, 9, 1392. [CrossRef] 9. Osmanoglu, B.; Dixon, T.H.; Wdowinski, S. Mexico City subsidence observed with persistent scatterer InSAR. Int. J. Appl. Earth Obs. Geoinf. 2011, 13, 1–12. [CrossRef] 10. Torres, R.; Snoeij, P.; Geudtner, D. GMES Sentinel-1 mission. Remote Sens. Environ. 2012, 120, 9–24. [CrossRef] 11. Geudtner, D.; Torres, R. Sentinel-1 system overview and performance. In Proceedings of the 2012 IEEE International Geoscience and Remote Sensing Symposium, Munich, Germany, 22–27 July 2012; pp. 1719–1721. 12. Phiri, D.; Simwanda, M.; Salekin, S.; Nyirenda, V.R.; Murayama, Y.; Ranagalage, M. Sentinel-2 Data for Land Cover/Use Mapping: A Review. Remote Sens. 2020, 12, 2291. [CrossRef] 13. Copernicus Programme. Available online: http://www.copernicus.eu (accessed on 20 April 2020). 14. Costantini, M.; Minati, F.; Ciminelli, M. Nationwide Ground Deformation Monitoring by Persistent Scatterer Interferometry. In Proceedings of the IEEE Geoscience & Remote Sensing Symposium, Milan, Italy, 26–31 July 2015. 15. Haghighi, M.H.; Motagh, M. Sentinel-1 InSAR over Germany: Large-scale interferometry, atmospheric eﬀects, and ground deformation mapping. Z. Geodasie Geoinf. Landmanag. 2017, 142, 245–256. 16. Shamshiri, R.; Motagh, M.; Nahavandchi, H.; Haghighi, M.H.; Hoseini, M. Improving tropospheric corrections on large-scale Sentinel-1 interferograms using a machine learning approach for integration with GNSS-derived zenith total delay (ZTD). Remote Sens. Environ. 2020, 239, 111608. [CrossRef] 17. Raspini, F.; Bianchini, S.; Ciampalini, A. Continuous, semi-automatic monitoring of ground deformation using Sentinel-1 satellites. Sci. Rep. 2018, 8, 7253. [CrossRef] 18. STEP | Science Toolbox Exploitation Platform. Available online: http://step.esa.int/main/ (accessed on 10 March 2019). 19. Sandwell, D.; Mellors, R.; Tong, X.; Wei, M.; Wessel, P. Open radar interferometry software for mapping surface Deformation. Eos Trans. Am. Geophys. Union 2011, 92, 234. [CrossRef] 20. StaMPS: Stanford Method for Persistent Scatterers. Available online: https://github.com/dbekaert/StaMPS (accessed on 16 June 2020). 21. Agram, P.S.; Jolivet, R.; Riel, B.; Lin, Y.N.; Simons, M.; Hetland, E.; Doin, M.-P.; Lasserre, C. New Radar Interferometric Time Series Analysis Toolbox Released. Eos Trans. Am. Geophys. Union. 2013, 94, 69–70. [CrossRef] 22. Yunjun, Z.; Fattahi, H.; Amelung, F. Small baseline InSAR time series analysis: Unwrapping error correction and noise reduction. Comput. Geosci. 2019, 133, 104331. [CrossRef] 23. Blasco, J.M.D.; Foumelis, M.; Stewart, C.; Hooper, A. Measuring Urban Subsidence in the Rome Metropolitan Area (Italy) with Sentinel-1 SNAP-StaMPS Persistent Scatterer Interferometry. Remote Sens. 2019, 11, 129. [CrossRef] 24. Potin, P.; Rosich, B.; Miranda, N. Sentinel-1 Mission Status. In Proceedings of the EUSAR 2016: 11th European Conference on Synthetic Aperture Radar, Hamburg, Germany, 6–9 June 2016; Volume 100, pp. 1297–1304. 25. Rosen, P.A.; Shams, K.S.; Gurrola, E.M. InSAR Scientiﬁc Computing Environment on the Cloud. In Proceedings of the AGU Fall Meeting, San Francisco, CA, USA, 3–7 December 2012. 26. De Luca, C.; Zinno, I.; Manunta, M. Large areas surface deformation analysis through a cloud computing P-SBAS approach for massive processing of DInSAR time series. Remote Sens. Environ. 2017, 202, 3–17. [CrossRef] 27. Zinno, I.; Bonano, M.; Buonanno, S. National Scale Surface Deformation Time Series Generation through Advanced DInSAR Processing of Sentinel-1 Data within a Cloud Computing Environment. IEEE Trans. Big Data. 2020, 6, 558–571. [CrossRef] 28. Yu, Y.; Balz, T.; Luo, H. GPU accelerated interferometric SAR processing for Sentinel-1 TOPS data. Comput. Geosci. 2019, 129, 12–25. [CrossRef] 29. Casu, F.; Elefante, S.; Imperatore, P. SBAS-DInSAR Parallel Processing for Deformation Time-Series Computation. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2014, 7, 3285–3296. [CrossRef] Remote Sens. 2020, 12, 3749 20 of 20

30. Manunta, M.; De Luca, C.; Zinno, I. The Parallel SBAS Approach for Sentinel-1 Interferometric Wide Swath Deformation Time-Series Generation: Algorithm Description and Products Quality Assessment. IEEE Trans. Geosci. Remote Sens. 2019, 57, 6259–6281. [CrossRef] 31. Nestor, Y.-M. Interferometric processing of Sentinel1 TOPS data. IEEE Trans. Geosci. Remote Sens. 2016, 54, 2220–2234. 32. Prats-Iraola, P.; Scheiber, R.; Marotti, L.; Wollstadt, S.; Reigber, A. TOPS Interferometry with TerraSAR-X. IEEE Trans. Geosci. Remote Sens. 2012, 50, 3179–3188. [CrossRef] 33. Yague-Martinez, N.; De Zan, F.; Prats-Iraola, P. Coregistration of Interferometric Stacks of Sentinel-1 TOPS Data. IEEE Geosci. Remote Sens. Lett. 2017, 14, 1002–1006. [CrossRef] 34. Duan, W.; Zhang, H.; Wang, C. Deformation Estimation for Time Series InSAR Using Simulated Annealing Algorithm. Sensors 2019, 19, 115. [CrossRef] 35. Costantini, M. A Phase Unwrapping Method Based on Network Programming. In ERS SAR Interferometry; European Space Agency—Publications ESA: Zurich, Switzerland, 1997; Volume 406, pp. 261–272. 36. Zhang, Y.; Wu, H.; Kang, Y.; Zhu, C. Ground Subsidence in the Beijing-Tianjin-Hebei Region from 1992 to 2014 Revealed by Multiple SAR Stacks. Remote Sens. 2016, 8, 675. [CrossRef] 37. Ge, D.Q.; Wang, Y.; Zhang, L. Large scale land subsidence monitoring with a reduced set of SAR images. In Proceedings of the Geoscience & Remote Sensing Symposium IEEE, Cape Town, South Africa, 12–17 July 2009. 38. Duan, W.; Zhang, H.; Tang, Y. Land Subsidence Monitoring for Beijing-Tianjin-Hebei Region using Sentinel-1. In Proceedings of the 2019 SAR in Big Data Era (BIGSARDATA), Beijing, China, 5–6 August 2019. 39. Zhou, C.; Gong, H.; Chen, B.; Gao, M.; Cao, Q.; Cao, J.; Duan, L.; Zuo, J.; Shi, M. Land Subsidence Response to Diﬀerent Land Use Types and Water Resource Utilization in Beijing-Tianjin-Hebei, China. Remote Sens. 2020, 12, 457. [CrossRef] 40. Guo, H.; Bai, J.; Zhang, Y. The evolution characteristics and mechanism of the land subsidence in typical areas of the North China Plain. Geol. China 2017, 44, 1115–1127. 41. Herring, T.; King, R.; Floyd, M.; McClusky, S. Introduction to Gamit/Globk; Massachusetts Institute of Technology: Cambridge, MA, USA, 2018.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional aﬃliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).