“Infrastructure” can mean a wide range of things for different people; but in this project, we’re talking about making sure that the software and libraries that are ‘under the hood’ of GRID are updated to the latest versions and to take a look at other opportunities for improving the product.  Let’s start with two of the biggest components of the GRID engine; GeoServer and PostgreSQL.

GeoServer (http://geoserver.org/) is what powers all the spatial components of GRID.  We have extensively used GeoServer in the past, but due to some issues we kept hitting with it, we were actually considering a potential replacement.  However, as is the case in the open source community, since that time there have been a wide range of upgrades delivered for GeoServer (just look at that consistent commitment graph below), and as a result, we have chosen to stick with GeoServer and to undertake the required upgrades to update it.

The GeoServer Commit graph from Github – a nice solid baseline of ongoing commitments – see more at Github

PostgreSQL (https://www.postgresql.org/) is our back-end database that stores all the GRID data, and this is something that we had also considered a potential replacement for.  However, after reviewing the sheer amount of work of uncoupling from PostgreSQL, and in also reviewing the upgrades that have occurred in the background, meant that we’ve also chosen to stick with it, and to undertake upgrades instead.

Meanwhile, programming libraries like Django and Dynatree, have been upgraded; which has meant a lot of effort consolidating and testing these new versions.  In addition, we’ve also undertaken a bunch of additional enhancements for GRID, including additional web services, absolute co-ordinates, metadata downloads and release notes.  So, when we start rolling out the upgrades in the coming weeks for all of our GRID instances (see the map below), they will all benefit – and we’ve been able to ensure that we are providing a highly valuable upgrade as a result.

Meanwhile, we’ve also been refactoring, enhancing and restructuring the codebase behind GRID itself, to make sure that the deployment and ongoing management of the system is much easier.  This is a key productivity boost for our own team, but is also setting the scene for some further upgrades and broader uptake we are seeing for GRID into the near future.

We are heads down at the moment heading towards the next set of deliverables, and we continue to work with the NRM GRID Champions around the State to determine the next set of functions that we will be delivering.

We’ll be posting more on the project as it progresses – stay tuned via the blog or via Facebook, Twitter or LinkedIn.

Piers

The post Under GRID’s hood appeared first on Gaia Resources.

I heard a phrase recently which was really upsetting:

“What are the disadvantages of your software apart from it being open source?”

The implication was that the open source nature of that particular software was a disadvantage, which actually came as a shock to me.  This is something we had heard years ago but to be frank, we hadn’t heard this in a long time, and I thought this sort of thinking had been well and truly debunked.

If you want a good primer on the business case for open source you can always head to the Open Source Initiative – this page in particular! However, if you want a more specific example, back in 2012, Andrew gave a presentation at a Surveying and Spatial Sciences Institute “Going Places” event that busted five myths (warts and all) about open source:

• Open source software is unsupported and unreliable
• Big companies don’t use open source software
• Open source = free
• Open source is unsustainable
• Open source improves faster

I’ve included his presentation from way back then here – it’s still relevant today.

Open-source software is software whose source code is published and made available to the public, enabling anyone to copy, modify and redistribute the source code without paying royalties or fees” (Wikipedia)

In running Gaia Resources for over 12 years, we’ve worked with a wide range of open source software, and built the business on it.  But it appears that sometimes we need to explain open source again from scratch.

Let’s look at three examples; QGIS, CollectiveAccess and Geoserver.

QGIS in particular has become one of my favourite pieces of software, spatial or otherwise.  We first implemented projects within it a number of years ago to implement a new spatial system for a client, which still is in production to this day.  Since then, we have developed a specific environmental GIS training course around it; as this freely available software fits really well with the not-for-profit Natural Resource Management (NRM) community that we work with well.  On a daily basis, Andrew, James, Jake and Tracey all use it for a variety of projects (and I do occasionally, although I get told off for “meddling”).

QGIS is not only open source, it is also freely downloadable as well.  It effectively – and comprehensively – replaces commercial GIS software packages, leading to significant cost savings for groups that use it (including us).  And as open source software, we are able to contribute to it – and back with our first project we did just that – to make QGIS work for the client, we made our first contribution to the QGIS project source code and have worked with it ever since.

We do a lot of work now in the GLAM sector, and the open source CollectiveAccess platform is one package we’ve done a lot of work with – and continue to do so.  Our work on this gave me an excuse to visit New York last year while on holiday to visit the team at Whirl-i-gig, who are the project owners of CollectiveAccess.  What’s not to like about open source software if it gives you that opportunity?

We’ve been working with the Western Australian Museum, CSIRO and the Royal WA Historical Society (as well as other organisations) in implementing CollectiveAccess.  Again, as open source software, it is freely available, and we can contribute to the source code directly, which has meant that Kehan and Ben are actually two of the most frequent contributors to the software.  We are also discussing a range of things internally about how we can help with the future directions for the software as well, and make it even better for other collections here in Australia (and beyond).

For us, CollectiveAccess comes free as a platform to start a project; and then we charge for our time in implementing it.  As an example, with the Royal WA Historical Society, while they didn’t pay any software licence fees, they paid for our time to implement the software, migrate their data from their existing platform, and then build the bells and whistles that they required – including a new web site, online shop and a new collections search portal.  And the benefit of the software being freely available means that there are no ongoing software licencing costs, so the Society only covers hosting and maintenance costs – a lot less than commercial licence costs for other collection systems.  For a small not-for-profit organisation this means they get a system that is the envy of larger organisations, at an ongoing cost that is actually affordable.

We’re also doing similar things with the AccesstoMemory (AtoM) and Archivematica open source products in the Archives space as well.

Let’s turn to one of our favourite “hidden treasures”.  Geoserver is the back end system for several of our projects, and most notably, it powers our GRID product, that delivers an online, enterprise level GIS for the NRM community (again, a not-for-profit community).

Our experience with Geoserver has been a little checkered.  Under the hood of GRID, Geoserver has historically created a few issues for us as we push it through some pretty big processes in GRID here and there.  But that said, I am really happy to say that we are about to make a whole raft of changes and upgrades to GRID to take advantage of the upgrades that have been implemented by the many people that contribute to the Geoserver code base (and this will be another blog in the near future, as well).  This will be a great improvement for the GRID product, and it didn’t cost the community that uses it anything.

We also use Mapnik, another open source mapping engine, on a number of our projects, such as Corals of the World, IGEM and others.  These projects – using either Mapnik or Geoserver – would be unlikely to be in existence without these tools being an open source product – both due to it the freely available nature of the software, and the collaborative efforts that are put into the software development.

So, QGIS, CollectiveAccess, Mapnik and Geoserver are all open source software packages that we have implemented, contributed to and collaborated on.  To me, open source means that we deliver more for less; and the funding for a project is spent delivering on the requirements that the client wants, not on providing the base level software.  I’m no Richard Stallman, and I’m not going to proselytise about open source, but for us, and our clients, it makes complete business sense.

Groups like the Open Source Initiative, Open Geospatial Consortium , OSGEO and a whole range of others actively promote open source and try to explain all the points that I’ve outlined in this post.  There’s obviously a whole range of work still required.

Meanwhile, we’ll keep on delivering high quality projects using open source products.

Piers

P.S.  You can keep in touch with us as always via Facebook, Twitter or LinkedIn, or leave a comment below.

The post The Disadvantages of Open Source appeared first on Gaia Resources.

I have been using this library recently to publish layers from PostGIS and had to do a bit of digging to uncover some parts of the process, so I wanted to document the process I went through in case anyone else is looking to do the same thing.

1. Install gsconfig

This is well covered in the documentation here

2. Connect to your GeoServer catalogue

The first step once you have gsconfig installed is to make a connection to your GeoServer instance

cat = Catalog('http://localhost:8080/geoserver/rest')

3. Create a new workspace to publish your layers

Now create a new workspace if you don’t have one already setup

ws = cat.create_workspace('newWorkspaceName','http://example.com/testWorkspace')

4. Create a new store from your PostGIS database

The next step is to create a new feature store pointing to an existing PostGIS database. A tip here is if the layer you want to publish is in a schema other than ‘public’ make sure you supply the schema in the connection string. I found that when I didn’t GeoServer wouldn’t pick up the projection of my layer when I was publishing it.

ds = cat.create_datastore('newDatastoreName','newWorkspaceName')
ds.connection_parameters.update(host='localhost', port='5432', database='postgis', user='postgres', passwd='password', dbtype='postgis', schema='postgis')
cat.save(ds)

5. Publish a layer using the new workspace and store

Now use the workspace and feature store you just created to publish a layer from PostGIS

ft = cat.publish_featuretype('newLayerName', ds, 'EPSG:4326', srs='EPSG:4326')

The first epsg code is the native projection of the layer and the second is the published projection. They don’t have to be the same, but in this case the layer is already on the projection I wanted to publish.

6. Create a new style from an SLD

If you want to apply a different style to your published layer you can create a new style from SLD, otherwise skip this step and GeoServer will apply the default style.

with open(path_to_sld_file) as f:
   cat.create_style('newStyle', f.read(), overwrite=True)

Setting overwrite=true will overwrite any existing SLDs with the same name.

7. Apply the new style to your layer

Apply your newly created style to your new layer.

layer = cat.get_layer('newLayerName')
layer.default_style = newStyle
cat.save(layer)

gsconfig is a great library and with a bit of effort to figure it out it abstracts away a lot of the work involved in making REST requests to GeoServer.

Contact us through Facebook, Twitter or LinkedIn.

Andrew

The post Using GeoServer REST API with Python appeared first on Gaia Resources.

To avoid any nasty cross domain issues, the first step is to turn on JSONP support in GeoServer, to do this open the web.xml file in the ../geoserver/WEB-INF/ folder of your GeoServer deployment and add this statement at the bottom of the file

<context-param>
    <param-name>ENABLE_JSONP</param-name>
    <param-value>true</param-value>
</context-param>

Once you have done this, restart GeoServer and do a get capabilities request and you should see JSONP appear in your list of result formats

GeoServer is now ready to receive JSONP requests, so the next step is to start requesting data with Leaflet. There is a existing plugin for JSONP ajax requests, but it is pretty straight forward to roll your own using JQuery.

Create the GeoServer JSONP request, GeoServer uses a slightly weird syntax for this so you need to include the callback in the URL:

var geoJsonUrl = “http://000.000.0.000:8080/geoserver/cite/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=cite:BoreHoles&outputFormat=text/javascript&format_options=callback:processJSON”;

Send the request with jQuery

$.ajax(geoJsonUrl,
{ dataType: “jsonp” }
).done(function ( data ) {
});

Create the callback function to turn the json into a vector Leaflet layer

function processJSON(data) {
boreHoles = data;
    boreHolePoints = L.geoJson(boreHoles,{
onEachFeature: onEachFeature,
pointToLayer: function (feature, latlng) {
return L.marker(latlng, {
icon: getIcon(feature.properties.type,’unhighlight’)
});
}
}).addTo(map);
}

All done You should now have a new json layer added to your map, which will look something like this

You can see this GeoServer, Leaflet, JSON combo in action on a site we built that has just gone live for CSIRO

http://www.groundwatercooling.csiro.au/feed.html

Email me directly here or find me on Twitter

Andrew

The post JSON with GeoServer and Leaflet appeared first on Gaia Resources.