These projects cover a wide range of GIS tasks our team have been involved in, including data management, basic and complex processing, database creation, GIS strategy, spatial plugin development and yes, making maps!

We’ve been helping the Department of Water and Environmental Regulation (DWER) in support of a few important initiatives, including:

• The Region Estuaries Initiative (REI) is a “program to improve the health of six estuaries in the South West.” Our objective here was to create a map of the regional catchment group boundaries for estuaries alongside other data collated from the catchment groups.
• We provided quality control resources for the Department’s Healthy Rivers program, which was setup to “collect and interpret data on rivers and their catchments, and use the knowledge gained to support development of strategies to best protect the environment.” Through this comes the creation of the River Reaches spatial layer shown below. With well over 50 areas mapped quality control using a documented process becomes a major part of ensuring consistency and a high level of quality. This essentially requires the use of numerous GIS processes ranging from automated geometry checks to manual spot checks on attributes and linework.

Gaia Resources have had a great partner in Robin Connolly from Hydrologia, who has been sending through a number of small projects involving data processing tasks and map production for their surface water hydrology reports. This work often covers aspects of stormwater management, flood risk and environmental impact assessment around mine sites and other infrastructure. Our role as his extended GIS team is to source data and work through how best to represent options and hydrological modelling results with numerous other themes for each area. For more information have a look at our projects summary.

We had the opportunity recently to do a deep dive into the GIS system at Garnet Mining Australia (GMA) and help them to set up a best practice spatial environment for their core business workflows. This started with our standard GIS health check followed by the migration of data to a cloud-hosted database. We helped GMA to design file and folder naming conventions, and built up a guideline and set of procedures to help them easily manage their spatial data into the future.

This will create a more seamless experience for their workflows by using QGIS desktop GIS software to discover and load the most up-to-date information. Next up we will provide QGIS refresher training to GMA staff and a comprehensive handover of their new setup. In these difficult times with COVID-19 restrictions affecting businesses everywhere, we are looking at maintaining social distancing with a remote training session and combining classroom style training with our free online video tutorials.

Our software developers even got in on the act with GIS software this year when we worked with the Yamatji Marlpa Aboriginal Corporation (YMAC) to rebuild their QGIS plug-in for layer discovery and automated map production. Have a look at our previous blog about that project.

A special mention should also be given to our GIS analysts’ work on the recently released Retromaps website, where we carefully stitched together over 2000 georeferenced sewerage scheme plans from past decades. That website has been a big hit for the public to check out building outlines and historical mapping from their own neighbourhoods, and our blog from last week provides more detail on the work involved.

If you’d like to know more about GIS and our work in this space, please leave a comment below, connect with us on Twitter, LinkedIn or Facebook, or email me directly via jake.geddes@gaiaresources.com.au.

Jake

The post All things Spatial! 🌏 appeared first on Gaia Resources.

As part of the project, the SROWA digitised and georeferenced 2,149 historical sewerage plans for the Perth area, thanks to a whole range of volunteers.  These resulting georeferenced images are pretty large – around the 400MB mark each – and provide an unparalleled, highly accurate and high resolution snapshot of the Perth metropolitan area, from dates ranging from 1910 through to 1964.

There were a number of challenges associated with making these maps available in a way that is performant, but also clean and consistent to provide an exceptional experience for anyone using the site.

The Challenge

We needed to make sure that our solution had the following characteristics:

• Simple and easy to use for SROWA, their clients, and the general public,
• It has the basic map tools to navigate easily and efficiently, including search capabilities,
• It will show the plans displayed “in the real world” (i.e. on a mapping interface), and
• The mapping interface should also show current satellite imagery.

In addition to these basic requirements, we also wanted to make sure that the solution:

• Delivered the plans in way that is most useful to as wide an audience as possible,
• Will be expandable to be able to add in other map series if they become available,
• Represents exceptional value for money in both the creation of the system and the ongoing maintenance of the system for SROWA, and
• Works with the existing investments that the SROWA have put into their internal systems.

No small challenge, and there were a couple of problems to deal with along the way…

The Problems

The first problem was reducing 2,149 large images (400MB) without losing the high resolution quality.  So, we used some pretty handy spatial software tools and developed a bunch of scripts to help deal with the volume of images and reduce the size (on average) to around 3 – 4% of the original image – with minimal loss of resolution.  We also put the site online using infrastructure from Amazon Web Services – which means we can scale it up really easily if the demand requires it (and in the meantime, it’s great value for money!).

The second problem, and the most time sapping task, was to deal with the “overlap” between the maps.  Each of the raw original plans overlaps the ones near to it (see figure below), so we needed to get around this to create a seamless mapping layer.

The maps when overlaid on each other block out parts of the map, so with our solution we had to remove these to make them more useful as a seamless layer.

We did this by digitising the “real world” boundaries of the maps (where there were actual features to see) and after some final checks and quality control we could now start stitching everything together.  Creating a seamless layer involved multiple spatial scripts and more free and open source spatial tools.  After a run through of the scripts, we had a nicely tiled map layer, all ready for use on the web.

But of course, you still need to be able to download the individual map tiles so that you can see all the important marginalia on the sides of them (like the date of the map) so we also were able to allow the maps to be individually downloaded from within the site itself.

Putting it together

The resultant Retromaps website was designed and developed to cope with the problems we had identified and delivered on the challenges we had set ourselves (and SROWA set us).  The resultant web site has seen a lot of people already signing up – there were over 7,000 people signing up on the first Saturday it was out, thanks to the launch being shared across a whole range of sites on social media.

An example of the sorts of detail and maps that you can see within the site – but go visit it for yourself!

When SROWA first approached us about this project, it was a bit of a dream job and a labour of love for us – we’re all about archives and spatial so to get the chance to pull these together has been a great opportunity.  It was something that we’ve been thinking about for some time (since our blog in 2015, “Hacking the Archives“) and it was great to see it come to fruition.

If you’d like to know more about RetroMaps and our work in archives or mapping (or both!), please leave a comment below, connect with us on Twitter, LinkedIn or Facebook, or email me directly via jake.geddes@gaiaresources.com.au.

Jake

The post RetroMaps appeared first on Gaia Resources.

In spatial technology, the word “topology” is all about the building blocks of geometry. Like other ‘ology’ words (e.g. geology and biology) it has its root in the study of something – and in this case we are talking about the study of geometrical properties and spatial relationships between constituent parts of a shape or a set of related shapes. Topology considers anything from the individual feature to the relationship between features. Individual features can be in the form of points, lines or polygons. Furthermore a polygon can be broken down into its constituent nodes/vertices, segments/edges and faces (see schematic below). Topology usually deals with the way in which features interact with adjacent or connecting features.

A 2-dimensional or 3-dimensional shape consists of different geometry types (Source: Cc-by-sa-3.0)

The job of maintaining topologically consistent data is critical if you are in the business of using spatial data to do anything beyond make basic maps; for example, accurately calculating area of different vegetation types in a study area.  Encompassing this process is a huge range of software, tools, and features that enable users to maintain a healthy dataset or database – and we’ve been crazy dedicated enough to go through these for you. Scroll down to find out how you can get a handle on your funky topology.

Topology Errors

In your GIS travels you may come across a number of topology errors, each with their own whirlwind of disaster! Sometimes the topology issue is not easy to spot until you try and run a geoprocessing tool and it just falls over with a generic error message. The problem is often so small and at such a minute scale, that you wouldn’t know it is a problem until you get these errors trying to work with the geometry.

Example of how ET Geowizards software deals with overlapping polygons (Source: ET SpatialTechniques).

Some of the common topology errors include:

• Overlaps – where polygons overlap each other (and they should not),
• Gaps – where there is dead space between polygons (but there should not be),
• Slivers – our favourites, caused by tiny little pieces not matching up,
• Dangles – where lines hang over the edge of something, and,
• Bowties – where there’s been issues from the digitising of the data and polygons are self-intersecting.

Depending on the purpose of your data, there are several other error types you can look for with the topology software and tools available.

If you encounter any of these it is probably safe to say any further analysis on the data will create more problems, and you need to get that stuff fixed up!

Quantum GIS (QGIS)

QGIS (a free and open source GIS package) has a topology checker tool built into its core product. Users are able to set specific rules which will point to that error’s location and allow you to focus on it and fix it. This can be more of a manual task of fixing one error at a time from the list generated, but very helpful nevertheless considering that some of these errors could be tiny.  I had the pleasure of working on one native wildlife habitat project with over 25,000 errors in one dataset, and I should point out there are ways of streamlining these tools to efficiently deal errors of that scale.

Example of the topology checker tool in QGIS

Check out our QGIS video tutorials where we go through this tool and others. There are also new tools in version 3.x to ensure topology is digitised correctly and errors are kept to a minimum.

The added bonus with QGIS is the multiple GIS software packages and plug-ins that extend the core product. This opens up other avenues in order to investigate topology tools for your particular purpose.  In particular, check out GRASS “v.clean” in the QGIS processing toolbox!

ArcGIS

ArcGIS is a leading proprietary GIS desktop software and has multiple tools for establishing topology rules, as well as checking and fixing errors. Using topology features in ArcGIS requires an  upgrade to the middle tier license – called Standard – that comes at an additional price to the Basic product.

There are excellent topology tools and workflows that can be used to setup new data or import existing data – embedding specific topological rules (i.e. no gaps, no overlaps, etc.) so by the end of your project you will be the proud owner of topologically correct data.

Other commercial software out there also deals with topology issues in a range of other ways.

ET GeoWizards

ET GeoWizards is an extension tool for ArcGIS as well as having its own standalone software product. It has a once-off cost – a relatively small investment for something that we consider to be good value for money. This link will give you an overview of what ET GeoWizards offers in terms of topology and how it goes about identifying, maintaining, and fixing topology errors.

For the functionality and quality of output that ET GeoWizards offers to ArcGIS Basic users out there the relatively low price point is justified in our opinion.

PostGIS

PostGIS is a free and open source spatial database software that includes a small number of topology tools. It’s a bit more complex than the ‘usual’ desktop tools, but it’s worth reviewing for larger corporate systems.  Check them out here!

So what?

So, you’ve come to the realisation that you have a (topology) problem. Great! You’ve accepted the challenge and your data is on the road to recovery; so, which toolset is best for you? Well, there’s a bunch of different ways in which these software packages can be used on their own or in combination to deal with different topology issues; as shown below:

Matching topology needs and software in a “simple” diagram

We have come across multiple instances where datasets needed some topology love (click here for project examples). Depending on the use case the software chosen usually stood up to the task and in some workflows the process could be automated.  In our disturbance mapping work for various mining companies for instance, their tenements, disturbance and various other datasets were overlapping and convoluted in ways that made it impossible for them to get an accurate a reliable snapshot and statistics for environmental reporting.  With a mixture of ET GeoWizards, QGIS, and automated processes we were able to cleanse their data and output a result for the client to report efficiently and effectively.  We also documented the on-going processing that was needed for their team to continue to maintain a topologically clean data store into the future. This is something that we’re working up right now into a separate blog article, focusing on the entire chain from capturing disturbance through to processing results, from start to finish.

If you would like to know more about this subject or how we can help your business understand, cope, or survive topology in GIS, then please drop me a line, or start a conversation on Facebook, Twitter or LinkedIn.

Jake

The post Topology in GIS appeared first on Gaia Resources.

Tracey started collating shark sighting data (as much as we could find from various media sources, and various shark warning sites) and compiling it into the Coastal Walkabout citizen science dataset. This was a lot of detective work, and had Tracey scouring all sorts of different places for this data (mainly from December 2015 onwards), such as the Surf Life Saving WA Twitter feed and the Facebook page Dorsal Perth Shark Reports – a bit like some historical blogs that Piers wrote about a while ago.

Once the data was all loaded, we created an interactive web map, which you can see below. The web map will allow you to drag the time slider across to view a range of data from different dates, and zoom in and out (it’s another implementation of CartoDB, similar to the one I wrote about previously).  The data is linear in this view, flashing on and off within time ranges.

Following on from just displaying dolphin and shark sightings, we decided to compare the two in order to find any significant correlations.

These were then brought through QGIS as a Comma Separated Values (CSV) spreadsheet, ready for some analytical processes. The first step was to calculate a 1km buffer around the locations of the dolphin watch sightings. Next was to intersect the shark sightings with the 1km dolphin watch sighting buffers. In order to find date and distance intervals between sightings I wrote a simple script to do the actual legwork.  While we didn’t have heaps of data to work with, the calculations worked and could lead to further studies, and here’s some highlights:

• 46 shark sightings intersected with a 1km buffer around dolphin sightings.
• Of these 46, they were logged as the following species:

• Some of the closely correlated results can be seen in the following table:

This sort of analysis is just the start of the type of analysis that you could do with citizen science data (assuming you can get people interested in your subject) and by doing spatial investigations of this data it really can show some interesting correlations.

So were people confusing dolphins and sharks?  Well, we think the jury is still out, but it’s certainly possible – more observers, and more data would definitely help solve that question!  If you want to know more about how we did this, drop me an email, or start a conversation via Twitter, LinkedIn or Facebook.

Jake

The post Dolphins or Sharks? appeared first on Gaia Resources.

Our initial discussions with Ross were focused around breakaways and how these could be mapped and analysed. Starting off with a few trial stations we derived breakaways from Landgate’s morphology dataset and terrain discontinuities from Geoscience Australia’s Geodata 1:250k series. Specifically, we extracted the slopes from Geoscience Australia’s 1 second Shuttle Radar Topography Mission (SRTM) Derived Digital Elevation Model (which has around a 30m resolution) and identified medium priority (10 – 15 degrees) and high priority (+15 degrees) areas, with low priority areas populated at the edge of rivers and waterbodies.

Some example preliminary maps from the process

In the second phase of the project we merged the above to form general areas to prioritise. As we had breakaways/discontinuities (lines) and slope above 10 degrees (polygons) we needed to do some data manipulation in order to create a propbability style heatmap, which was hoped to be able to show the areas where the baiting program should focus on. So we just ran a simple script to convert both the lines and polygons into points (centroids), and from this heatmaps were processed for the stations to enhance highly dense areas.

The final phase was to produce these results for a further 80 stations. With a combination of a mapbook layout, dynamic raster loading on each station, and complimentary layers each station now had its own prioritised areas for future dog baiting, like the one shown below.

An example product, showing areas with higher probabilities for the presence of feral dogs

From these initial studies and analyses we are hoping to extend the project with GNRBA to produce additional products that are ready to be provided to the pilots who do the baiting, so that they can be more targeted in their baiting of areas, and to have these updated regularly based on additional exclusion areas.  All this will help the GNRBA and the stations protect their stock, and their local environment, in a more targeted way.

If you found this interesting and would like to know more, then get in touch with me through the comment section below, email or through one of our social media streams such as Twitter, LinkedIn or Facebook.

Jake

The post Helping protect the Goldfields Nullarbor region appeared first on Gaia Resources.

We recently ran an internal evaluation of a range of web mapping technologies and one of the ones we wanted to get ourselves an update in was CartoDB.  I know Piers and Andrew have been followers of Javier and his team as they have developed CartoDB from their early days, and have seen many developments and iterations of the software come along.  It seemed like a good time to have a go at this first challenge using CartoDB software, and it is an area I’m particularly interested in myself, so I got the go-ahead to get it sorted!

The end result is embedded below:

Firstly, though, we had to capture the places where we have done work.  To do this, we used our own GRID instance to capture the centroids of areas where we thought we’ve done work.  It’s been an interesting exercise for me as a relatively new starter at Gaia Resources to go back over nearly 500 projects (the next job we win will be our 500th job!) and 12 years of work to see the wide range of work that we’ve done.  We’re still going through and checking the locations – often we will have done projects that cover wide geographic extents, or a wide range of things, and so finding that elusive centroid – that is appropriate – requires a fair bit of detective work!  That will be an ongoing process for us to keep tuning this source data.  Once we had this data captured via GRID, we could then extract it in a CSV format, and then use that to power our CartoDB map separately (we have yet to investigate the WMS capabilities but that’s on the list).

Once the data was collated, though, it was pretty easy to get an embeddable web map set up for this blog post. CartoDB does most of the heavy lifting back-end processes for its users, so it is quite a short introduction to become acquainted with the website. For this instance it was as simple as a quick registration process, and basically a “drag-and-drop” of the CSV format data into my CartoDB datasets dashboard. We were now able to create a map and customise it with a range of handy tools ranging from symbology wizards, sql statements, css control, and filters. Once the map was ready for publication CartoDB gives you a few options; a URL link, a JavaScript file, or how we intended to use it – an embeddable iframe link.

Once we did embed it, we found a few little things we’d need to do to make this jump through the hoops we want to.  Since we have points “on top” of each other, we found out that you can’t use the info tool to dig deeply into those points, so we thought about clustering the points instead.  Clustering works reasonably well out of the box, but you’d have to put in a workaround to get it to explode the points out and display the attributes using the info tool (like we’ve done in the Leaflet maps we’ve done in the past, like this one or the picture on the right), so we left it off for now.  We’re just doing some reviews of this “simple” installation to pick up on things like that, and we’ll be collating this sort of information.

Our evaluation on all of the web map tools we’ll be trying out will be collated in a future blog post, but certainly this one with CartoDB was an easy one to implement.  If you are interested in finding out how we can help with your web mapping needs, then leave me a comment below, email me or start a conversation with us on Facebook, Twitter or LinkedIn.

Jake

The post Web Mapping Testing – CartoDB and Job Locations appeared first on Gaia Resources.

These online collections range from historic hand drawn maps to diary entries written across the trips of many explorers. So, given that some of us are pretty keen on maps, we have been looking for a project to use some of these maps and to see what we can do with them.  I thought I’d start by taking some of the Perth metropolitan town site maps, dating back to 1966 (*).

Credit: State Records Office of Western Australia

There are numerous ways in which you could use these maps, but the first step was to catalogue and georeference the 33 town site maps I started with. For those non-savvy GIS people this involves assigning “real-world” coordinates to a correlated spread of points on an image (in this case, the town site maps).  To find these these coordinates, I used Open Street Map roads, pipelines, and reserves, and to get the data onto my desktop, I used MapZen, where the Perth Metro area is readily available to download in multiple formats.

The 33 town sites cover most of the central and west metropolitan area of Perth (with a single one down near Fremantle).  As I was zooming into the maps my interest grew as I was finding significant differences and I started to speculate about what’s changed in that area. From a first glance up close the first thing that comes to mind is the substantial increase in major infrastructure. The majority of minor roads have not changed, but certainly a new freeway and railway line shows up very nicely – have a look around our office in Leederville to see what that area looked like before there was a Freeway!

This was all sitting in my desktop tools, and we wanted to go a step further than just having a bunch of images that only we could see.  So, with some support from Andrew and Serge, I loaded the images into our own GRID instance (we run one where we can play ourselves) and then used the features in GRID to turn this into a public Web Mapping Service.  Then we used the WordPress Leaflet plugin we’ve developed for GRID (and which we use in places like the Pilbara Corridors site) to make these maps available through the map below.

This is just a starting point.  The ways in which you could use these maps are extensive, and with many more regional town site maps catalogued in the SRO database this project could extend outwards:

• Overlaying road, rail, and transport layers (there is a transport layer included in web map, give it a try!).
• Overlaying 3D buildings
• Graphic presentations and visuals
• Feature extraction and encoding
  • Vectorise historical data into numerous features such as; anthropogenic features (roads, buildings, places, place names), natural features (rivers, lakes, parks, reserves), suburb information, parcel/property boundaries, built up areas and all sorts of other datasets, and
  • You could even take this vector data and do some analytical processing to evaluate things such as land use or land cover changes.

It has been great to have some time to take the resources that the State Records Office are making freely available and to do some professional development with.  While I already knew how to take images and georeference them, to do this and then expand my knowledge of our other products, like GRID and our WordPress Leaflet plugin, has been a great way to learn about a few new tools.

If you want to know more, drop me an email, or start a conversation with us on Facebook, Twitter or LinkedIn.

Jake

* Piers’ note: 1966 isn’t that old you young whipper snapper!

The post Hacking the Archives appeared first on Gaia Resources.