During the last decade, open source technologies for geospatial data have dramatically expanded in range, content and functionality. A whole range of mashups have been encouraged using Google Maps and Earth, Microsoft’s Bing, OpenStreetMap, and Yahoo Maps, and these APIs can often be customised with relatively basic programming skills. These mashups harness a growing volume of freely available geographic information, including maps, satellite imagery, and geo-referenced features, and they also benefit from a growing range of free and open-source tools and software libraries (Bocher & Neteler, 2012; Sherman, 2012). To different extents, these tools can be customised and employed by map libraries, particularly in developing more dynamic and innovative applications, search interfaces, and in delivering images of historic maps. Whilst there is considerable potential, these tools require somewhat different methodologies, skill sets, attitudes and institutional contexts from conventional proprietary technologies. This paper offers case studies of the National Library of Scotland’s recent experiences in using open source technologies, all with the help of Klokan Technologies GmbH, in the hope that this information may be of some value for others in making a more informed judgement on using this software in other map libraries.
These projects between the National Library of Scotland (NLS) and Klokan Technologies are the result of several years of successful collaboration using historical maps in web applications. The NLS has large sheet mapping collections, numbering some two million maps, and has also a large and growing online collection of historical maps at http://maps.nls.uk (numbering 46,000 online maps in March 2012). In recent years NLS has focused on improving access to these maps through georeferencing, and through georeferenced search methods (Fleet, 2005; Fleet & Kowal, 2007; Henrie, 2009). Both funding and technical expertise have been limited, and there have been limitations too due to the NLS IT infrastructure and support, with a lack of institutional support for specialist applications such as maps. Klokan Technologies have provided expertise and a number of innovative tools including MapTiler, Georeferencer, the NLS Historical Maps API, OldMapsOnline, and a customised GeoServer/OpenLayers application. The Klokan Technologies team members have considerable relevant skills including an advanced knowledge of GDAL (the core of MapServer and other open-source GIS tools), and have also contributed to several of these open-source projects of the OSGeo foundation.
Below we review the main development of these applications, and the ways in which they have improved access to historical maps, particularly through improved functionality and user experience. We conclude by discussing some of the practical experiences of developing these tools, and how the necessary technical knowledge is extendible to other new open-source technologies relevant to map libraries.
2. Using open-source desktop geospatial software: GDAL and QGIS
At a basic level, open-source geospatial technologies can be employed to very practical ends on the desktop. Historically, NLS map library has used the ArcGIS software for desktop purposes, and we still pay for three desktop licences for this; this is particularly because we have found the ArcGIS georeferencing functionality superior to using other open-source software so far. However, with our move away from ArcGIS/ArcIMS server in 2011, and for better integration with other open-source software, we have increasingly made use of the GDAL open-source command line programs and QuantumGIS/QGIS.1 GDAL commands can allow simple and sometimes quicker ways of viewing and converting between different image formats, or changing map projections. Whilst ArcGIS and QGIS can often perform the same functions, GDAL commands can be more effective, particularly for batch programs, for integrating with other programming scripts, and for creating more open/non-proprietary results, so better for long-term preservation. For example, we have found ArcGIS can sometimes create georeferenced files that are only readable by ArcGIS, whereas QGIS and GDAL usually create more open files. Converting vector data between shapefiles and KML files, for example, has not always been easy with ArcGIS, but very easy with ogr2ogr, whilst the gdaltindex command can create a clickable index to a set of georeferenced series maps.
Some of the most common GDAL/OGR commands we have used include:2
- gdalinfo — useful for listing information about a map image, for example, its format, size, coordinate system, extent, and any blank/nodata pixels.
- gdaltindex — useful for creating a shapefile of raster image extents (e.g., for creating a clickable graphic index) from a set of georeferenced images
- gdal_translate — useful for converting map images between different formats, for example, from TIFF to JPEG2000, or from JPEG to PNG.
- gdaltransform — useful for converting coordinates between different projections and spatial reference systems, for example, from the British National Grid to the Global Spherical Mercator projection.
- GDAL2Tiles — the earliest open-source program for creating a Tile Map Service from georeferenced images, later developed into MapTiler (see below).
- ogr2ogr — useful for converting vector map files to different formats (for example, an ESRI shapefile into a KML file).
- ogrinfo — useful for listing the properties of a vector file — its different layers, features, coordinate system, etc.
The GdalTools plug-in in QGIS presents these GDAL commands, and others, through a graphic interface with natural language for options and settings.
3. MrSID OpenLayers Viewer (ungeoreferenced images)
This project created in 2009 a new and enhanced web viewer for all the NLS zoomable images. In the late 1990s when NLS first developed a website for making available zoomable images of maps in the Pont manuscript maps website, the best available image formats for delivering large, multi-resolution zoomable images of maps were felt to be the proprietary MrSID and ECW formats. NLS acquired and used LizardTech’s MrSID compression software, for creating MrSID images, and their Express Server for delivering them online. Express Server has in fact been robust and reliable, but over the succeeding years, significantly cheaper, open-source, and/or better alternatives have been developed for viewing large images. Some of these newer tools are reviewed on the Old Maps Online Publishing page3, and include Zoomify, the IIPImage JPEG2000/TIFF Image Server, supported by client-side browsers such as OpenZoom, OpenLayers, IIPMooViewer, Seadragon Silverlight and PanoJS. Zoomify uses conventional JPEG images, tiling them into a set of directories for rapid delivery, and has viewers that can be customised in a wide range of ways. IIPImage and Djatoka can use JPEG2000, which like MrSID and ECW is a multi-resolution raster format based on Waveleyt compression, but an open standard. Unfortunately, NLS IT staff had difficulties setting up IIPImage Server on the NLS Windows-based IIS web servers, as IIPImage Server is arguably easier to install on LINUX servers.
NLS has been keen to move to a non-proprietary image format such as JPEG2000 for delivering image surrogates online, and keen too to move to a better image server software. The problems of the LizardTech Express Server viewer can be summarised as follows:
- fixed image extents on screen
- non-AJAX based viewing — a whole — image refresh on each click
- limited ability to present map image within wrapper
- printing and downloading images only possible with downloadable browser plug-in
- no Permalink feature.
The presentation of the OpenLayers image within a customisable header and footer (using easily editable CSS) allowed a better presentation of groups of maps, the delivery of specific map website database metadata into the header, relevant links, and more recently, e-commerce buttons for ordering copies. These individual map pages are now directly accessible by search engines and for external links directly into the website, and so they provide a new and improved means of access to maps in the website. The use of OpenLayers also allows the Viewer to be upgraded as new versions of OpenLayers are released, which can allow better display of images, integration with new mobile applications, and new design features. In conclusion, this new viewer considerably enhanced the quality, look-and-feel, and functionality of viewing the NLS historic maps, and has been very robust and easy to maintain.
The Georeferencer application is an international collaborative online project, developed by Klokan Technologies GmbH and Moravian Library in Brno, allowing free, crowdsourced collaborative online georeferencing of map images from a number of libraries. Whilst the Georeferencer software is not open-source, it shares some of the same values, and the results in terms of metadata and georeferenced images, as well as their visualisation, support other open-source technologies. Georeferencer allows a cheaper, public, collaborative way of georeferencing maps, sharing several of the advantages of MapWarper and the New York Public Library’s Map Rectifier. However, unlike MapWarper, it does not rely upon images being uploaded to a website, making use of zoomable images on existing library web servers. Following a pilot project with the Nationaal Archief in The Hague, NLS used Georeferencer in a similar pilot project in late 2010 on 1,000 early maps of Scotland (Figure 2). Since that time, the Georeferencer software has been enhanced, as can be seen in the British Library’s Map Georeferencer, launched in February 2012.
Georeferencer is funded through a collaborative model, whereby individual libraries pay for development work, but the results of this can then be shared in improved functionality for all partners. The shared administration of Georeferencer across libraries also allows for efficiencies in sharing maintenance and hosting arrangements. Libraries maintain complete control over their images and metadata. It is also easy to export the georeferenced metadata that Georeferencer gathers, which can be used for populating the MARC21 034 field, a Dublin Core Bounding Box, and related geo-enabled search methods such as in the MapRank Search Engine or OldMapsOnline search portal (http://www.oldmapsonline.org/).
The NLS Georeferencer project can report modest success after 15 months of active usage. The main NLS Georeferencer home page has recorded over 8,000 page views in this time. Some of the categories of maps, such as Great Reform Act town plans (75 maps), have all been completely georeferenced. Town plans and larger scale maps have been the most popular, and in general, over half of all the original maps have been georeferenced. Greater success could have been achieved with better promotion by NLS, easier login, better incentivising tools, better tools for multi-sheet maps, better presentation of metadata, perhaps the incorporation of the MapAnalyst accuracy tool, and an improved interface for visualising the results. Most of this functionality has been developed already (and applied in the extremely successful British Library pilot) and we are looking forward to bringing the NLS maps into this new improved version of Georeferencer in the near future. We hope that related technology, such as the ability to crop map sheet neat lines and link the resulting cropped map image to pre-existing shapefile bounding box extents, can be collaboratively funded. In this way, series maps could be more easily mosaicked together where shapefiles of sheet boundaries exist.
5. MapTiler — Tile Map Service Mosaics
Through a number of collaborative projects, NLS has actively prepared Tile Map Services (TMS) of historic georeferenced maps for web delivery in the last few years. We now have over 100 TMS mosaics taking up about 1.2 Tb of disk space, focusing on Scotland, but also covering Ireland, Great Britain, and Belgium. These include:
- Ordnance Survey ten-mile to the inch Planning Maps of Great Britain (1940s–1950s)
- Ordnance Survey one-inch and 1:25,000 mosaics of Great Britain (1940s–1950s)
- Three Ordnance Survey one-inch to the mile mosaics of Scotland (1880s–1950s)
- 62 Large-scale Ordnance Survey town plans of Scottish towns (1847–1895)
- 33 Bathymetrical Survey charts of Scottish lochs (1897–1909)
- 221 Air photo mosaics of Scotland (1940s)
- 30 Maps of Edinburgh (1765–1945)
- Bartholomew (1:253,440) and War Office (1:63,360) series maps of Ireland (1940s)
- British War Office maps (1:250,000–1:50,000) of Belgium (1940s).
These mosaics were prepared as TMS tilesets, as described by Pridal and Zabicka (2008), providing a set of pre-rendered image tiles at fixed scales. We have found the TMS tilesets to offer a very fast and reliable way of delivering large mosaics fast over the web, with tiles pre-created at a set of zoom levels for a specific projection and datum. As Google Maps, Microsoft Bing, and other geo-data APIs use the same projection and tiling profile, the extents of all tiles as well as the zoom levels are predefined for the whole Earth on the Spherical Mercator projection (see also Tiles à la Google Maps: coordinates, tile bounds and projection for more information).
The open-sourceGDAL2Tilescommand-line program, and its later development as the MapTiler GUI were used to create the tilesets.4 The MapTiler software is very simple and powerful, with excellent results.5 Some of these georeferenced maps were also used to create an Historical Maps API by creating a tileset of historical maps of Great Britain in the 1920s to the 1940s, and then distributing it across multiple servers for fast and reliable access (Fleet & Pridal, forthcoming 2012). Upon creation, the tilesets can be loaded onto any web server and made available through easily customisable front-end HTML files as dynamic georeferenced overlays with adjustable transparency (Figure 3). Our main map overlays page has recorded over 10,000 page views in the last year, but this ignores all the direct tile usage in other applications. The tilesets can be readily consumed by OpenStreetMap6, and they also were used extensively in the Visualising Urban Geographies project (Rodger et al., 2009), as tilesets using the Google Map API, and within ExtMap. A related Python script, also developed by Petr Pridal, creates the necessary shapefiles to deliver the same tileset as a OGC Web Map Service using MapServer. We also have made the tilesets available as overlays within our GeoServer/OpenLayers application as a Mosaics Viewer (see below).
6. GeoServer/OpenLayers Sheet Viewer and Mosaic Viewer applications
6.1 Limitations of ArcIMS
This has been the largest and most significant project, to migrate our existing web-mapping applications to open-source software in 2011, and is therefore described in most detail. From 2005, NLS developed six georeferenced historical mapping applications using ESRI’s ArcIMS/ArcGIS Server (Fleet, 2005; Henrie, 2009). Four of these were search interfaces for historic maps using clickable maps, whilst three of them presented georeferenced mosaics of historical maps:
The applications were intended for a broad general public — genealogists, local historians, students, planners, consultants, archaeologists, and library staff — to access historical maps of Scotland. The applications used ArcGIS, ArcSDE with SQLServer for delivering large geo-referenced mosaics, and with queries to shapefiles and gazetteers handled using Coldfusion.
However, there were a number of problems of the ArcIMS application:
- Expense of ArcGIS, with annual licensing costs of £5,000–£6,000
- Limited web-mapping functionality — out-of-step with contemporary OpenLayers/Google/Bing web-mapping interfaces, and with non-slippy mapping, with a click and whole-map refresh.
- Application instability, due particularly to difficulties of handling of high usage spikes.
- Fixed screen width for map interface (500×500 pixels).
- Basis on British National Grid (EPSG:27700) rather than Spherical Mercator (EPSG:900913), so unable to easily interact with standard Google/Bing web mapping APIs.
6.2 GeoServer development
Following a public tender in May 2011, Klokan Technologies were selected to develop a replacement application with open-source technologies, particularly using GeoServer and OpenLayers. GeoServer is an open source server software, written in Java that allows users to share and edit geospatial data, developed over the last decade by The Open Planning Project (TOPP). Those involved with GeoServer also founded the GeoTools project, an open source GIS Java toolkit, which links closed to GeoServer. GeoServer has been very much designed for interoperability, and has very close links to OSGeo and the Open Geospatial Consortium, publishing data from any major spatial data source using open standards including WMS, WFS, and WMTS. It also works very well with OpenLayers, automatically preparing an OpenLayers-based presentation of geodata such as shapefiles with GeoServer, and GeoWebCache, allowing much faster delivery of geodata through pre-prepared caches (in our case, of shapefiles).
6.3 Sheet Viewer and Mosaic Viewer
The move to GeoServer was also used as an opportunity to rationalise and simplify the NLS georeferenced map applications. The previous categorisation with a map application devoted to a particular map series was replaced by two main applications based upon functionality: the Sheet Viewer allows the selection of particular map sheets that cover a location of interest, whilst the Mosaic Viewer allows the display of georeferenced mosaics as overlays on top of modern map layers.
The Mosaic Viewer makes available sets of georeferenced mosaics, and in fact, simply utilises the TMS layers that we have created using MapTiler. We did experiment with loading large raster images and mosaics into GeoServer, but not without some difficulties, partly due to the nature of NLS’s existing georeferenced images, and a lack of suitable large storage space. This is an area that we could test more in future, as well as the use of a spatially-enabled database such as PostGIS linked to GeoServer. However, the TMS images load very fast into the Mosaic Viewer application, which provides an easier way of locating places on the mosaic using the gazetteer, etc., and then comparing them to one another and to the present day.
6.4 Search methods
Originally NLS used the Ordnance Survey Landranger gazetteer in the ArcIMS applications, a large database of some 250,000 placenames from all the 1:50,000 maps in the United Kingdom, which is still available at no charge through OS OpenData. Partly due to performance limitations of queries to this gazetteer using Coldfusion, and the fact that the Google Maps API gazetteer also includes street names and postcodes, with autosuggestion of names during typing, we took a decision to use the Google Maps API gazetteer. This has on the whole been popular, but not universally (the Landranger gazetteer offers better rural coverage), and we are considering adding the Landranger gazetteer back as a special option.
GeoServer makes it particularly easy to add new layers as shapefiles, or alter and revised existing layers. Our shapefiles contain bounding boxes of map sheet extents, and GeoServer has provided a good impetus to bring these together into one application, now with consistent metadata within them for sheet references, dates, and the image ID for viewing/zooming. Export of consistent metadata with sheet extents, for example for the OldMapsOnline application, is now relatively easy. The shapefile data can be presented in a range of styles for colour, thickness and fill, and GeoWebCache is able to then seed each shapefile layer into pre-cached content at a range of zoom levels for very rapid display. The NLS application allows different layers of mapping to be chosen by the user from a radio-box list in the left-hand panel (Figure 4), and then particular sheets are selected for any chosen location. GeoServer is queried through the open Web Feature Service standard, with map sheet metadata returned in JSON.
Although NLS has not undertaken any formal usability or feedback testing, we know that our user community generally preferred and appreciated the new GeoServer/OpenLayers application. The full-screen slippy mapping, the use of the Google maps and gazetteer, and the integration of separate series applications, have all been appreciated and commented upon. However, we did also receive some negative feedback, particularly from those familiar with the old ArcIMS application and unhappy at it disappearing. The new application works better with a high-bandwidth Internet connection, and with more modern web browsers, and this also disenfranchised a small proportion of our users.
6.5 Future development
There are clearly many advantages for map libraries of using open-source software. The tools are often cheaper, and more standards-compliant, allowing collaboration across a broader community of institutions. Development and customisation is delegated more to users, rather than owners of the software, and map libraries have more scope to be directly involved in developing and maintaining applications. Very often, these applications are being actively developed by an international community, a geospatial community much broader than libraries, and new features and technologies developed by others can be rapidly implemented. The technologies may also support related improvements, such as increasing the quantity of mapping online, expanding georeferenced mapping and georeferenced access methods, and of significant improvements in the visual and aesthetic appeal of applications.
In most institutional environments, there is a need to use both open-source and proprietary software, and different balances and combinations of them both often work well. There is also plenty of scope to alter the mix over time, especially as the availability and performance of open-source applications continues to grow. Map libraries often operate within institutional contexts that may only allow limited usage of open-source tools, and have longer-term strategies relating to catalogues, digitisation, and preservation, for example, that any technologies need to support. Through shared practical experiences, map libraries will be better able to make decisions on which of these open-source technologies are appropriate.