Trends in development of Free and Open Source Software for Geomatics (FOSS4G) Markus Neteler FEM - Centre for Alpine Ecology, Italy [email protected] OSGeo Foundation [email protected]
CASCADOSS International Symposium 16 June 2008 in Warsaw, Poland www.cascadoss.eu
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw Welcome to FOSS4G!
MODIS: Land surface Temperatures (LST)
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw Welcome to FOSS4G! For comparison, see http://www.spatialserver.net/osgis/ Proprietary vs Open Source/Free Software Component Proprietary Open Source Metadata Catalog CubeWerx WRS GeoNetwork
Dedicated Viewer ArcReader gvSIG, QGIS, uDig
Cartography ArcMap/ArcGIS GMT, GRASS, JGRASS Analysis ArcINFO/ArcGIS, GRASS, OSSIM, SAGA, ERDAS, ENVI ILWIS Handheld ArcPad gvSIG Web Viewer ArcIMS Openlayers, Mapbuilder, Mapbender WMS, WFS ArcIMS Mapserver, Geoserver, Mapguide OS Spatial Database Oracle Spatial, PostGIS, (MySQL, ArcSDE SQLite)
Virtual Globe Google Earth, Virtual OSSIM Planet, NASA
Earth WorldWind Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw FOSS4G Interoperability Trends (1)
● GDAL/OGR started in 1998 with few formats, used by GRASS and few others
● today many raster (>80) and vector (>30) formats, used by many software packages (FOSS4G and proprietary)
● FDO goes open source in 2006 Software Using GDAL/OGR 3D DEM Viewer from MS MacroSystem ● Cadcorp SIS: A Windows GIS with a GDAL plugin. OGC standards are implemented in various CatchmentSIM: A Windows terrain analysis model for hydrologic applications. Demeter: A another OpenGL based terrain engine somewhat similar to VTP. ESRI ArcGIS 9.2+: A popular GIS platform. OSGeo software packages Feature Data Objects (FDO): Open source spatial data access libraries. flighttrack: GPS track viewing and downloading software for Mac. FME: A GIS translator package includes a GDAL plugin. GdalToTiles: C# Program (open source) for make image tiles for Google Earth with KML Superoverlay. ● GeoDjango: A framework for building geographic web applications. Example for community driven specification: GeoMatrix Toolkit, and GeoPlayerPro from GeoFusion: 3D visualization. Google Earth: A 3D world viewer. GRASS GIS: A raster/vector open source GIS uses GDAL for raster/vector import and export GeoRSS becomes OGC whitepaper in 2006 gstat: a geostatistical modelling package. gvSIG: Desktop GIS Client. ILWIS: Remote Sensing and GIS Desktop Package. IONIC Red Spider: an OGC Web Services platform includes a GDAL plugin. Leica TITAN: a geospatial data sharing and visualization environment. libral: A raster algebra implementation and an experimental GUI+CLI GIS with Perl and GTK+. MapGuide: Open source web mapping server. UMN MapServer: A popular web mapping application with GDAL support. MapWindow: open source ActiveX control with GIS functionality. MicroImages TNT products: advanced software for geospatial analysis Mirone: Matlab based package for geospatial, oceanographic and geophysical analysis of grids OpenEV: An OpenGL/GTK/Python based graphical viewer which exclusively uses GDAL for raster access. FOSS4G and Standardization? OSSIM: Another geospatial viewing and analysis environment which uses GDAL as one of several plugins. the PYXIS innovation: An application for viewing performing analysis and modeling on user's geo-located data. Quantum GIS (QGIS): A cross platform desktop GIS. ScanMagic: Win32 application for visualization, analysis and processing of remote sensing data ● Scenomics: Software for building terrain databases uses GDAL for projection and data import/export. OSGeo's policy is to not standardize SpaceEyes3D: 3D visualization software for cartographic data. SpatialAce: A GIS Rapid Application Development environment. StarSpan: raster/vector analysis. TerraGo Technologies: The GeoPDF file format is using GDAL for data import/export. ● Thuban: A multi-platform interactive geographic data viewer. Some OSGeo members are active in OGC vGeo: Interactive data fusion and visual analysis tool. Virtual Terrain Project: fostering tools for easy construction of the real world in interactive, 3D digital form. ● ... but see the OGC history:
1990 1992 1994 1997 2004
GRASS Interagency Open GRASS Open GIS Open Geospatial Steering Commitee Foundation (OGF) Consortium (OGC) Consortium (OGC) Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw FOSS4G Interoperability Trends (2) Open Standards are preferred in the geospatial industry. Why?
They facilitate
● Interoperability between agencies which essentially increases availability of data (in Europe, see INSPIRE initiative) – easier data sharing
● Cost reduction through easy data exchange and transparent procedures
● Value added in heterogeneous software environments: both proprietary and FOSS4G benefit from that
Without vendor lock-in traps, Open Source/Free Software successfully competes with proprietary software: Quality and costs decide!
Question: But Open Data in Europe??
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw Portability, Interoperability
Geostatistics Predictive View modeling Interact Teach
raster Spatial External GRASS Analysis vector data Modeling
Database OGC Web engine: Services (WMS, WFS, Tables, WPS, ...) attributes DBF Visualize
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw Trends in GIS Analysis with FOSS4G
● GRASS and others as free GIS backbone
● distributed and parallel computing (GIS number crunching)
● Parallelized numerical library in GRASS
● also (Desktop) GIS wants to be online: OGC Web Processing Service (WPS) is gaining interest (pyWPS, 52°N implementations)
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw FOSS4G WebGIS Trends Web 1.0 (1993): static pages ...
Web(GIS) 1.5 (1996): a bit dynamic pages
● grasslinks (first WPS style system)
Web(GIS) 2.0 (2006): the "I am GIS" Web
● collective intelligence, crowdsourcing, collective mapping (OSM, ...)
● you and me with GPS: mapping party (new fashion in Italy)
● GIS meets social Web
● Sensor web (SOS) and ubiquitous GIS
● NASA WorldWind, Mapserver (p.mapper, ka-maps, ...), OpenLayers, FeatureServer, Geoserver, MapGuide OS, pyWPS...
● it's all about protocols and mashups
Web(GIS) 3.0 (tomorrow?): the semantic Web or the Geoweb (?)
● our online data get (spatially) organized? Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw FOSS4G Community Trends Communication technology evolves...
● '80s: phone, letter, magnetic tape...
● 1989 (?): GRASS FTP server online - the first FOSS4G GIS server in civil Internet?
● 1991: GRASS Mailing lists started: grass-user and grass-dev
● 1991: WWW is invented - First GRASS Web site (US CERL) in 1993 (?)
● 1999: remotesensing.org established, hosting PROJ, GDAL, OSSIM, libgeotiff, ...
● 2000: FreeGIS.org mailing list and portal launched - fundamental for community building
● 200x: Blogs, Wikis, emails (OSGeo lists: > 14000, just in GRASS lists >4000 subscriptions), VoIP, ... Communication is easy!
Meritocracy...! National FOSS4G corporations: 2000: GRASS Anwendervereinigung e.V. - 2008: becoming OSGeo-DACH? 2007: Associazione italiana per l'informazione geografica libera (GFOSS.it/OSGeo.it) 2007: many OSGeo local chapters Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw User Communities
Note: not necessarily representative due to technical and GRASS User map cultural differences
QGIS User map
http://planet.osgeo.org/
Mapbender User map Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw FOSS4G Documentation Trends 80's started with tutorials 2002: Open Source GIS: A GRASS GIS Approach - first FOSS4G book ever 2004 (?): first FOSS4G Wikis come up: public participation 2006: OSGeo Education and Curriculum Committee is formed 2007: FOSS4G primer and other book-like initiatives 2008: Many FOSS4G books now available: bookshelf
wiki.osgeo.org/wiki/Library www.grassbook.org www.mappinghacks.com
www.spatialguru.com Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw FOSS4G Conferences International conferences:
● Many GRASS conferences since 80s in USA and Europe, drop in 1995, relaunch Trento 2002, then Bangkok 2004
● Mapserver conference 2005
● Lausanne 2006: Merge of Mapserver, GRASS, EOGEO conferences to FOSS4G conference series, with Canada 2007, now South Africa 2008, next Sydney 2009, ... Annual national conferences:
● in Italy annual conf. since 1999 (first GRASSoriented, now GFOSS.it)
● in Germany annual conf. since 2005 (first Mapserver oriented, now FOSSGIS.de)
● in Spain: gvSIG conf. Since 2005
● ....
FOSS4G formal training
● Companies and universities offering courses; online courseware
● ECDL (European Computer Driving License): efforts to prepare a FOSS4G version, especially in the Italian community: http://wiki.gfoss.it/index.php/ECDL_GIS_FOSS Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw GIS development and economical value: GRASS SLOC Analysis Basic COCOMO model, but slightly different parameters GRASS 6.3.cvs, 16 Apr 2007 http://en.wikipedia.org/wiki/COCOMO#Basic_COCOMO
Or do this analysis yourself - Download and run: http://www.dwheeler.com/sloccount/ SLOC Totals grouped by progr. language: (Status GRASS 16 April 2007) ansi-C: 473155 (84.30%) tcl: 44256 (7.88%) sh: 19821 (3.53%) python: 10517 (1.87%) cpp: 10142 (1.81%) perl: 1608 (0.29%) http://www.ohloh.net/projects/3666 ... GRASS 6.4.svn, 23 May 2008 Total Physical Source Lines of Code (SLOC) = 561,286 Person-Years = 154.05 ... Total Estimated Cost to Develop = $ 20,810,621 (average salary = $56,286/year, overhead = 2.40)
Generated using David A. Wheeler's 'SLOCCount'
+1 million USD
in one year Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw FOSS4G Release Management Trends Code Management
● 80's: GRASS source code is managed manually by the core developers (hey, no civil internet yet!)
● 12/1999: one day before Year 2000, GRASS CVS is started (GDAL, Mapserver as well some years later)
● 2007: integrated OSGeo development platform: SVN, trac, wiki, mailing lists, IRC bots, buildbots
Release Management
● till 2006 (?): separated, ad-hoc releases by the individual teams
● since 2006: attempts to synchronize releases for common feature support etc.
● since 200x: thorough test cycles with alpha, beta, release candidates (milestones in trac) cross platform tests before releasing (code portable: GNU/Linux, MacOSX, MS-Windows, ...) aim: deliver end user product (as requested by today large user communities and distro packagers)
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw FOSS4G Quality Assessment: the GRASS case (In)formal Request For Comments (RFCs)
Implementation of code submission standards for C, Shell scripts, Tcl/Tk, and Python
GRASS Code Quality Control System started in 2006 (currently under rewrite to open up for other OSGeo projects), various scientific papers on this
● supports clone detection, and
● software engineering measures on code style
GRASS QA Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw GRASS Quality Assessment Improvement of source code base
Ref.: S. Bouktif, G. Antoniol, E. Merlo, and M. Neteler, ICSM 2006. A feedback based quality assessment to support open source software evolution: the GRASS case study.
M. Di Penta, M. Neteler, G. Antoniol, E. Merlo, 2005: A Language-Independent Software Renovation Framework. Journal of Systems and Software (JSS), 77(3), pp. 225-240 S. Bouktif, G. Antoniol, M. Neteler and E. Merlo, 2006: A Novel approach to Optimize
Clone Refactoring Activity. Proc. GECCO 2006, Seattle, WA, USA, July 8-12 2006 Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw Changing GRASS source code: what happens? (1/2) tflag->description = _("Print topology information only");
if (G_parser(argc,argv)) exit(EXIT_FAILURE);
/* open input vector */ if ((mapset = G_find_vector2 (in_opt->answer, "")) == NULL) { G_fatal_error (_("Could not find input map <%s>"), in_opt->answer); }
Developer (e.g., Italy) changes and enters: svn ci -m”i18N macro added” main.c
Code differences email is auto-generated and sent to “grass-commit” mailing list
SVN USA source code Email notification triggers repository updated of GRASS Quality Assessment System
USA (OSGeo) Canada Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw Changing GRASS source code: what happens? (2/2)
Email notification triggers CIA open source monitor receives USA? update of GRASS Quality simplified QA message Assessment System
Clone detection is run as well as other quality measures, results sent out
Canada
Email about code quality is sent to “grass-qa” mailing list
USA – worldwide subscribers
Total time elapsed for QA: SIGQ & CIA-IRC robot feed < 60sec #grass IRC channel
on freenode.net Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw OSGeo: Incubator (following Apache Foundation model) http://www.osgeo.org/incubator
The OSGeo Incubator is run by the Incubation Committee. It is responsible for: (1) Preparing and recommending incubation process and criteria to the board. (2) Reviewing applications for incubation and making recommendations to the board on projects that should be admitted into incubation. (3) Appointing a mentors for incubation projects. (4) Reviewing progress of projects against incubation requirements, and recommending projects for full project status to the board, when deemed appropriate. (5) Reporting on the status of projects in incubation and the incubation process to the board periodically (via Chair). (6) Suggest to graduate or reject
Incubation Process Procedures ● Incubator Application Questionnaire ● Incubation Process (code vetting, intellectual property etc.) ● Project Evaluation Criteria ● Incubation Mentor Guidelines ● Code Provenance Review Process
● Project Status monitoring (Wiki) ● Project Graduation Checklist Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw Sponsored development
http://wiki.osgeo.org/wiki/Google_Summer_of_Code_2008
http://code.google.com/p/google-summer-of-code-2007-osgeo/ Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw FOSS4G Business Trends
● OSGeo Service Providers directory Europe: 57 1 Israel 1 Hungary ● Voluntary registration (~100 registered) 1 Norway 1 India ● 30 companies with >= 10 employees 2 Belgium 2 Thailand 2 Finland 2 Japan 2 Ireland 2 Poland 1 South Africa 2 Sweden ● FOSS4G business model: 3 Netherlands 4 Australia service oriented and customization 3 Switzerland 4 New Zealand 4 Denmark ● Value is the solution, not the technology 4 Spain 3 Brazil 5 Italy ● Solutions drive the technology - not the other way around 6 England 9 Canada 6 France 30 USA ● Commercial (don't mix this up!) 14 Germany Proprietary GI solutions versus FOSS4G OSGeo Service ● FOSS4G can be commercial used! Providers (6/2008)
● Ongoing discussion... Provide commercial training directly as OSGeo brand?
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw Software operating costs (for customer)
Proprietary Software Free Software
Total Cost of Ownership (TCO) Cost of Ownership Total Overhead due to training and rewrite of missing software Time
B. Reiter 2004 after D. Wheeler 2004
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw Conclusions
FOSS4G delivers solutions:
● Software at end user level, even with multi-language user interfaces
● Documentation and training
What's inside:
● Communities large and yet growing
● Quality control procedures established
● Organizations at national and international level (OSGeo)
The impact of FOSS4G:
● Geospatial industry starts to adopt concepts and even directly FOSS4G software (in case of permissive licenses)
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw License of this document
This work is licensed under a Creative Commons License. http://creativecommons.org/licenses/by-sa/2.5/deed.en
“Trends in development of Free and Open Source Software for Geomatics (FOSS4G)”, © 2008 Markus Neteler, Italy http://www.grassbook.org/neteler/conferences.html [ OpenDocument file available upon request: neteler at cealp it ]
License details: Attribution-ShareAlike 2.5 You are free: - to copy, distribute, display, and perform the work, - to make derivative works, - to make commercial use of the work, under the following conditions: Attribution. You must give the original author credit. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.
For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above.
Markus Neteler Markus Meeting CASCADOSS.eu 16 June 2008, Warsaw