Open Source Geospatial Analysis
Last Modified: September 2018
1 This reference and training manual was produced by the University of Maryland Libraries.
Permission to reproduce this manual or any of its parts for non-commercial, educational purposes may be granted upon written request. Appropriate citation is appreciated.
University of Maryland Libraries GIS and Spatial Data Center McKeldin Library, Room 4118 College Park, MD 20742-7011
http://www.lib.umd.edu/gis
2 Table of Contents
GIS Facilities at the University of Maryland McKeldin Library… ...... 04
GDAL Utilities Exercise… ...... 05
3D Thematic Web-Mapping Exercise… ...... 10
2D Thematic Web-Mapping Exercise...... 29
More Training and Information...... 42
Credits...... 44
3 GIS Facilities at the University of Maryland
McKeldin Library
ArcGIS 10.5 and QGIS (an open source GIS software package) are loaded on all public workstations in McKeldin Library, including those located in the McKeldin 6101 instruction laboratory. The laboratory is open to the public during library hours when not in use by a class or librarian. The laboratory schedule is posted on the window by the door and updated each week. Color printing and large format printing are also available in McKeldin Library.
The GIS laboratory is located on the fourth floor of McKeldin Library in room 4118. This laboratory is available for use by faculty, staff, and students using geospatial methods in their research at the discretion of GIS and Spatial Data Center Staff. ArcGIS 10.5 and QGIS are installed on all computers in this laboratory, as well as other geospatial, image processing, and statistical packages.
For a complete list of software packages available, please see our GIS Facilities link https://www.lib.umd.edu/gis/people-and-facilities
Other Facilities
There are other departmental (e.g., anthropology, architecture/urban planning, civil engineering, geographical sciences, and landscape architecture) GIS laboratories on campus that are restricted to faculty, staff, and students in each of those disciplines. Additionally, the Office of Information Technology's software licensing program (http://www.oit.umd.edu/slic) offers ArcGIS at special/reduced rates for faculty and staff at the University of Maryland.
4 I. GDAL Utilities Exercise
In this exercise, We will use GDAL/OGR Command to manipulate the Raster and Vector data.
1. Work with the Vector layer(OGR)—Dissolve Vector features
1.1 Open your Command Prompt through search CMD in Windows search box.
1.2 Move to your work directory by cd Command.If your data folder in your downloads folder, you use cd \Downloads\Open_Source_GIS\GDAL\states to move to that directory.
5 1.3 List vector layer general information by ogrinfo. Pass a name to the shapefile you want to check. For example, ogrinfo states.shp (Do not forget the file extension and case sensitive!)
1.4 Before the geoprocessing, we can open states.shp in QGIS to check.
To open a shapefile in QGIS, click on the Add Vector Layer toolbar button, or simply press ctrl+shift+v. This will bring up a new window.
1.5 Now we can dissolve all US states as one feature by the Command: ogr2ogr -f "ESRI Shapefile" Country_dry.shp states.shp -dialect sqlite - sql "select ST_union(geometry) from ‘states'"
1.6 After the geoprocessing, we can open Country_dry.shp in QGIS to check and compare it with the layer of states.shp.
6 2. Work with the Raster layer(GDAL)—GEOTIFF to KMZ
2.1 Now you can type the Command: cd / to go back your root path and move to your work directory by cd Downloads\Open_Source_GIS\GDAL\ Night_Light to move to that directory.
2.3 List raster layer general information by gdalinfo 2016.tif
7 2.4 Then we can convert the 2016 Night light image from GEOTIFF format to Google KMZ format by the Command: gdal_translate -of KMLSUPEROVERLAY 2016.tif 2016.kmz -co FORMAT=JPEG
2.5 Now we can check the folder.
Study Drills:
* 1.Use GDAL Command to mosaic adjacent two views of Landsat 8 images (any periods).The data can be download from earth explorer: https://earthexplorer.usgs.gov/.
** 2. Convert the Mosaicked data from tiff to KMZ format with GDAL Command.
***3. Why do we need to use GDAL to process spatial data?
8 II. 3D Thematic Web-Mapping Exercise
II.1 Background of QGIS
Quantum GIS (QGIS) is a powerful Open Source GIS platform, which integrated lots of other open source software and packages such as SIG GIS, Grass GIS. Many of developers or Cartographers would create their plugins or tools to contribute this society.
In this exercise, we will aggregate and grid 2016 Crime incidents Data of Washington D.C and also involved several QGIS plugins in creating a 3D web map. In the end, we will publish our 3D Map to Github Page.So everyone can see the Web Map through a Github page URL.
Qgis2threejs plugin is a QGIS plugin, which can export terrain data, map canvas image, and vector data to the web browser. We can view 3D objects in the web browser (You need a web browser which supports WebGL).
II.2 Tasks Including in this exercise:
1. Points Data Aggression. 3. QGIS Scripts tool 4. QGIS plugins Mangement 5. Hexagon Mapping. 6. HTML code modified. 7. CSS code modified. 8. How to work with Atom Text Editor 6. How to publish Github Page.
II.3 Exercise:
9 1. Using QGIS(Script Tool) to aggregate data
1.1 Open QGIS and add “Crime_Incidents_in_2016.shp” and “Washington_DC_Boundary.shp” by clicking on Add Layer > Add Vector Layer. (In the layers panel, make sure Crime_Incidents_In_2016 is on top of Washington_DC_Boundary layer)
1.2 Then Open Plugin > Manage and Install Plugins…
A plugin search box would pop up and select All in the left side of this window Then input “process” and install the plugin “Processing.” Then we close the current window.
10 1.3 Click Processing > Toolbox.
1.4 Please open Processing Toolbox > Scripts > Tools> Get script from on-line scripts collection. In pop up “Get Scripts and Model” window, select “Not installed”> Hex grid from layer bounds> OK. Now we have the Hexagon grid tool in our QGIS.
1.5 Now we can open Hex grid from layer bounds under Scripts > Polygons. Set the input to “Census_Tracts_in_2010” and set the cell size to “0.001818”, which means each diameter of the hexagon is 200 meters. Set
11 the grid to [Save to temporary files] and check the box of “Open output file after running algorithm”. Run it. The current grid layer is in the memory not on the hard disk.f you close the QGIS right now the grid layer would be missing, and you have to redo the whole process again.
(At equator, 1 degree roughly equal to 110 km. So 1km ~ = 1/110 degree and 0.2km ~ = 1/550 degree, 200 meter ~ = 0.001818 degree)
1.6 The hexagon layer will generate and show it in the QGIS canvas. Now we need to aggregate the crime incident points to the grid later. Open Vector > Research Tools> Select by Locations…
1.7 Choose “grid” in select features and “Census_Tracts_in_2010” in that intersect features and check the box of”Include input features that intersects and click Run then close the window.
12
1.8 Right click “grid” layer in Layers Panel >Save Vector Layer as. Choose a path and name it as “Hen_200m” for the output layer and check the box of “Save only selected features.” Click OK.
1.9 Now you can remove the temporary grid layer from the Layer Panel. Click Vector > Analysis Tools > Count Points in Polygon…
13 1.10 Now we will count the crime incidents in each Hexagon. In the popup window, set Polygons as “Hen_200m”, set the field of Points as “Crime_incidents_in_2016,” set the field of Count field name as “Count”. Finally, set the fields of Output and count field name to the same path and name it as “Hexbin_200m”. Check on the Open output file after running algorithm. Click Run and close the dialog window.
We finished the aggression of the crime incident points to the hexagon geometry.
1.11 Right click “Count” layer to open the attribute table. We can see there is a Count field in the attribute table. Single left clicks the field
14 Count and set the descending order of the crime incidents of each hexagon.Then we will get the biggest number in the hexagon, which is 337. 1.12 Right click “Count” layer to open properties > Style. Select Graduated symbols. Choose “Count” in Column and click Symbol > Fill > Simple fill Then choose “OrRd” in Color ramp and “Natural Breaks (Jenks)” in Mode. Enter “6” in Classes and click Classify. In the Symbol field, double-click and set the first field from 0 to 0. Then double-click the first symbol and make the Transparency to 100% and click OK.
1.13 Then click OK. Zoom out the view of Hexagon Map a little bit. Now we finish the Hexagon Map of crime Incidents of Washington DC in 2016!
15 2. Using QGIS plugin (Qgis2threejs) to create 3D Map
2.1 Open Plugin > Manage and Install Plugins…
A plugin search box would pop up and select All in left side of this window Then input “Qgis2threejs”and install the plugin “Qgis2threejs” . Then we install another plugin name as “quickmapServices” After we installed these two plugins. Close the current window.
2.2 Minimize the QGIS window and create a new folder on your PC Desktop and name it as Crime_3D.
16 2.3 Open Web > QuickMapServices > OSM > OSM Standard. Now the OSM Standard base-map would add to your QGIS canvas.
2.4 Maximize the QGIS window and Check off all layers except the OSM Standard layer in Layers Panel.
17 2.5 Now we open the Qgis2threejs tool (Web> Qgis2threejs> Qgis2threejs) and select Polygon check on box of “Count”, select “Count” in Height and input “0.0003” in multiplier. Choose the path to the folder, which was we created in our desktop.
Name the output HTML file as Crime_2016 and run it. The 3D map will pop up in our browser!
2.6 You can save your Project file to your PC Desktop, which in case that you can modified it later. You can save your project as “3D Map” By Project > Save.
18 3. Add Legend to the Map
Now we need to add a legend to make the 3D map more readable. The plugin of qgis2threejs can generate the HTML, CSS and JavaScript documents, which can compose interactive map.
If we want to add a legend to the map canvas, we need to modify the HTML and CSS files of generated map documents.
HTML(Hypertext Markup Language), a standardized system for tagging text files to achieve font, color, graphic, and hyperlink effects on World Wide Web pages.
CSS (Cascading Style Sheets) is a style sheet language used for describing the presentation of a document written in a markup language, which used to control the style of HTML elements such as font, color and so on.
JavaScript is OOD programming language, which used to create interactive effects within browsers.
Three.js is a cross browser javascript library used to create and display animated 3D computer
Atom Editor is an open source text editor which allows us to modify our code.
19 3.1 Open the Crime_3D folder and change name from “Crime_3D.html” to “index.html”
3.2 Now you can close your QGIS and Open your Atom Editor. Then you can open your 3D Crime folder by File >Add Project Folder… Then choose the path to 3D Crime.
In the Atom editor, all files of Crime_3D under the Project tab, now you can click “index.html,” which will display on the right side and different syntax text will be highlighted by different color.
3.3 Modify HTML File
3.3.1 Now you can use “Control+F” to call the search box and search “.”
3.3.2 Please insert following block of code between ……..
this two html tags.20
Washington DC Crime Distribution
# of Crime Incidents(Jan. to Dec. 2016)
0
0 - 20
20 - 41
41 - 89
89 - 195
195 - 419
Author Name: Put your name here
3.3.4 Now you can save your index.html file and close it.
3.4 Modify CSS File
3.4.1 Now you can click “Qgis2threejs.css”.
3.4.2 Scroll down to the bottom of the Qgis2threejs.css
3.4.3 Please insert following block of code between after the .print-label
21 .legend { line-height: 16px; width: 280px; position: absolute; z-index: 1000; left: 10px; bottom: 10px; color: #000000; padding: 6px 8px; box-shadow: 0 0 15px rgba(0,0,0,0.2); border-radius: 5px; } .legend i { width: 16px; height: 16px; float: left; margin-right: 8px; opacity: 0.7; } .legend p { font-size: 12px; line-height: 16px; margin: 0; }
22 3.4.4 Now you can save your Qgis2threejs.css file and close the Atom.
3.4.5 Open the “Crime_3D ” folder and open the index.html file. The 3D map will popup in your default browser. We finished the 3D web map!
4. Publish the map to GitHub page.
Currently, the map only can run on our local computer. It is limited that we share this map with other people.If you want to share this map to your friend with a link, you need to choose a server and push the map documents to the server such Amazon S3 or Google Cloud, but we need to pay the service after the free trail expired.
Github page can be considered as a free server service. So everyone can use GitHub page to hold the personal projects or web app (No server side programming such as node.js, PHP, and Database interaction).
23 4.1 Create a GitHub account
4.1.1 Open your browser and search GitHub or input https://github.com/ in your browser. Sign up your account.
4.1.2 After your account established, please verify your E-mail address before you sign in.
4.2 Create a Github Page
4.2.1 Now you can log into your account and select start a project
4.2.2 Now you can create a new repository and name it as Owner_name.github.io. Select Public and check on the box of initialize this repository with a readme. Then click Create repository.
24 4.3 Upload Map documents.
4.3.1 Click Upload files
4.3.2 Drag the entire Crime_3D folder to Drag files here to add them to your repository area and click Commit changes. It will take few minutes to complete.
25 Now you can use yourownername.github.io/Crime_3D/index.html this URL to access your 3D map in your browser!
26 III. 2D Thematic Web-Mapping Exercise
In this exercise, we will aggregate and grid 2016 Crime incidents Data of Washington D.C and also involved more HTML CSS programming in creating a 2D thematic web map. In the end, we will publish our 2D Map to Github Page. qgis2web plugin generates a web map from your current QGIS project, either as OpenLayers or Leaflet. It replicates as many aspects of the project as it can, including layers, styles (including categorized and graduated), and extent. No server-side software required.
CARTO (formerly CartoDB) is a Software as a Service (SaaS) cloud computing platform that provides GIS and web mapping tools for display in a web browser. The company is positioned as a Location Intelligence platform due to tools with an aptitude for data analysis and visualization that do not require previous GIS or development experience.
CARTO users can use the company's free platform or deploy their instance of the open source software. CARTO is offered as freemium service, where accounts are free up to a certain size. For larger accounts, a fee is applied.
CartoDB Plugin for QGIS. It allows to view, create, edit or delete data from your CartoDB account using your favorite opensource desktop GIS: QGIS.
27 1. Carto 1.1 Create a Carto account:
Navigate to https://carto.com/students-access/ and click where it says “Sign up with Github”. This will allow you to access the student version of Carto. Now Enter your GitHub credentials
1.1.2 The current version is free. You would have 250 MB space to store your data.
1.13 After logged into your account, click your head icon > Your API key(Some peoples’ account would not see their API Key. if you would not see your API Key here, just neglect the 2. Carto Plugin exercises and continue to 3. 2D web Thematic mapping, For more information about the API KEY, Please check this link).
28 1.1.4 Please keep your API key and we will use it later.
2. Carto DB Plugin
2.1 Open QGIS and Add “Crime_Incidents_in_2016.shp” and “Census_Tracts_in_2010.shp” from the Path: Open_Source_GIS/ 2D_Web_Mapping/Shapefile to “Layer Panel” of QGIS with Add Vector Layer icon.
29 2.2 Then Open Plugin > Manage and Install Plugins… A plugin search box would pop up and select All in left side of this window Then input and install the plugin “CartoDB.” Then close the plugin window.
2.3 Click Add connection in the tool bar and Select New. Now you can enter your Carto User Name and API Key and click Save.
30 2.4 Click Connect.
2.5 Now, your CartoDB account is connected to your QGIS and all buttons from the CartoDB toolbox are enabled. You will see your user name where the "Add connection" button once was.
You can upload and download layers, create a new map, define MetaSearch services, and open the Python Console from the CartoDB toolbox.
Study Drill:
*1.Try to upload the crime points layer to your Carto account.
**2. Create a Crime incidents map with the Carto and edit it in your QGIS.
***3. If you know how to write SQL or select attributes in QGIS, try to fillet crime types as Robbery and create a heat map in Carto. (Tips: You can check the which filed is Crime types in the attribute table at first. Then you can output the selected layer and upload it to the Carto. )
31 3. 2D Points Cluster Thematic Mapping(qgis2web plugin)
3.1 Aggregated points to census tract and Reckon Crime Rate.
3.1.1 Open Plugin > Manage and Install Plugins… A plugin search box would pop up and select All in left side of this window Then input “process” and install the plugin “qgis2web” . Then close the plugin window.
3.1.2 Now we would aggregate the crime incidents points to the census tracts by Click Vector > Analysis Tools > Count Points in Polygon…
32 3.1.3 Then we would count the crime incidents in each census tract.In the dialog window, set the file of Polygons as “Census_Tracts_in_2010”, set the field of Points as “Crime_incidents_in_2016”. Set the Field of Count field name as “Count”. Finally, set the field of Count to the same directory and name it as “Census_2010”. Check on the Open Output file after running algorithm. Click Run and close the dialog window.
3.1.4 Remove the layer of “Census_Tracts_in_2010” from layers Panel. Right click “Count” layer to open attribute table.The Count field shows in the attribute table. Meanwhile, there also have other demographic data in the different census tracts. The Count field shows the number of crime incidents in the different census tracts.
FAGI means Federal Adjusted Gross Income. Unit:$ FAGI_Media means Median of Federal Adjusted Gross Income.Unit:$ AIAN means Native American and Alaska Native NHPI means Native Hawaiian Pacific Islander
33 3.1.5 Now we will count Crime rate (Crime incident per 1000 people) by following equation with Field Calculator. Cr ime Records in Census Tract Cr ime Rate = × 1000 Total Population in Census Tract
3.1.6 Open the Field Calculator in the attribute table. Check the box of “Create a new field”. Input “Crime_Rate” in Output filed name and select “Decimal number (real)” in Output filed type. Then choose “3” in Precision. In the end, copy "Count" / "P0010001" * 1000 to the box under Expression. Click OK.
Now we have a new filed that named “Crime_Rate” in Census_2010 field.
3.2 Generate leaflet Map points style leaflet Map.
3.2.1 Minimize the QGIS window and create a new folder on your Desktop and name it as Crime_2D.
3.2.2 Click Web > qgis2web > Create web map or click the button with leaf.
34 3.2.3 In the Popup window, check on the layer of crime_incidents_in_2016 under Layers and Groups and Check on the box of Visible and Cluster under the layer. Select the leaflet option first, which is at the bottom of the window. Then select OSM B&W under the Preview Map window. Then set the export path to the Crime_2D folder that we created before. Then click the Export and close the Export to web Map window.
35 3.2.3 Minimize the QGIS window and go your Crime_2D folder and rename the sub folder to Cluster_Map.
3.2.4 Open the Cluster_Map folder and Open the index.html. The crime incidents of leaflet cluster map would pop up in your default window. You can zoom in and zoom out with your touch pad or mouse wheel.
StudyDrills:
*1.Please upload your Cluster Map documents to your Github page and share it with your friends!(hint: please reference the 3D Map Exercise 4.3 Upload Map documents. )
36 **2. What the difference between 2D Map and 3D Map for the crime incidents visualization? Please list at least 3 aspects of your insights.
*3. 3 Choropleth Thematic Web-Mapping(leaflet)
3.3.1 Move to the Choropleth Folder, which has an index.html file. In the Choropleth Folder, create a new folder name it as data(Case sensitive!). Back to Layers Panel of Qgis, right click Census_2010 layer >Save as, and select the format as Geojson and name it as Crime. Choose the data folder as your Output folder.
3.3.2 Use Atom text editor to open the Crime.geojson file and input “var Crime =” before the curly bracket “{”.
3.3.3 Now you can open the index.html in your browser to review your Choropleth Web Map!
37 StudyDrills:
*1.Please upload your Choropeth Map documents to your Github page and share it with your friends!(hint: please reference the 3D Map Exercise 4.3 Upload Map documents. )
**2. What the difference between points cluster Map and Choropleth Map? Please list at least three aspects of your insights.
***3. Based on current Code.(You can open the index.html with Atom) Try to use other tract types(PSA) to visualize your crime incidents! Other tracts data can be founded in Open DC data.
38 4. Mapbox
Mapbox is a mapping platform for developers. The building blocks make it easy to integrate location into any mobile or online application. You can customize your map tile though Mapbox, which data Mainly from the Open street map.
4.1 Create a Mapbox account
4.1.1 Open your browser and search Mapbox or input https:// www.mapbox.com/ in your browser. Sign up your account.
4.1.2 After you setup your account, you would receive a confirmation E-mail. Now you can log in your account. There are some basic styles, and you can create your base map style in the studio. Meanwhile, the Mapbox also allowed the clients upload their datasets(Raster and Vector).
39 4.1.3 After you create your style, you can share, develop & use your newly designed map. Meanwhile, you also can import your new styled maps to ArcGIS, Qgis, Carto or Leaflet.
4.2 Upload Map Tiles(GEOTiff)
4.2.1 Select Tilesets > New tilesets , and choose 2016.tif from the file path: Downloads\Open_Source_GIS\GDAL\ Night_Light.
40 4.2.2 Then refresh your browser and the tiles that we just uploaded would show up. Click the VIEW bottom on the tile. Then click the Add to style bottom. Then choose New style > Empty >Create.
4.2.3 Click Take me to the data > Create layer > Publish.
41 4.2.4 Back to the Main Menu and select style. The new base map should show up. You can change the name and or back to Map box editor to edit it if you want. Now the new base map be available for sharing and developing!
42 More Training and Information
University of Maryland at College Park students, faculty, and staff have access to free online courses provided by ESRI. These cover a wide range of GIS topics and skills. To gain access to online classes, please contact [email protected] with the name of the course you are interested in and your UMD e-mail address to obtain a registration code.
Course list: https://www.esri.com/training/unlimited-esri- training/
University of Maryland Libraries’ GIS and Spatial Data Services Center website: http://lib.umd.edu/gis/
University of Maryland Libraries’ GIS and Spatial Data Services Center workshop series: https://www.lib.umd.edu/gis/workshops
43
Credits
1. GDAL: http://www.gis.usu.edu/~chrisg/python/2009/lectures/ospy_slides1.pdf2
2. GDAL Cheat Sheet: https://github.com/dwtkns/gdal-cheat-sheet
3. QGIS Wikipedia: https://en.wikipedia.org/wiki/QGIS
4. Open DC Data: http://opendata.dc.gov/
5. QGIS hexagonal grid- Ryan Ruthart’s blog: http://ryanruthart.com/using-qgis-to-create-a- hex-bin-map-and-aggregate-data/
6. MDN HTML: https://developer.mozilla.org/en-US/docs/Web/HTML
7. Javascript Wikipedia: https://en.wikipedia.org/wiki/JavaScript
8. Three.js Wikipedia: https://en.wikipedia.org/wiki/Three.js
9. Qgis2threejs Documentation : http://qgis2threejs.readthedocs.io/en/docs-release/ Tutorial.html
10. Map Legend: https://github.com/jakobzhao/GeovisualAnalytics/blob/master/wk03/ wk03_1_lec05/readme.md
11. QGIS tutorial and tips: http://www.qgistutorials.com/en/
12. Atom Text Editor: https://atom.io/
13. CSS Wikipedia: https://en.wikipedia.org/wiki/Cascading_Style_Sheets
14. Github Pages: https://pages.github.com/
15. Carto Wikipedia: https://en.wikipedia.org/wiki/CartoDB
16. CartoDB Plugins: https://github.com/gkudos/qgis-cartodb/wiki/Connect-your-CartoDB- Account-to-QGIS
17. Mapbox About: https://www.mapbox.com/about/
18. Qgis2web : http://www.qgistutorials.com/en/docs/web_mapping_with_qgis2web.html
19. Leaflet: http://leafletjs.com/
20. Leaflet Choropleth Map: http://leafletjs.com/examples/choropleth/
44