The before picture – a simple traditional top-down 2D map

The first inclination was to somehow feed the data into an existing 3D globe like Google Earth or NASA World Wind but that would require installing a client program or a Java Applet. Not my first choice, so I decided to have a look at the latest offerings from Open Layers 3. A quick bit of research later and I found ol3-cesium, an integration library between Openlayers 3 and Cesium, which is an open-source javascript library for globes and maps. A quick play around with the examples and the rest is history.

Once you click “Enable 3D” your map can be tilted, panned and zoomed around as much as you like

I’d like to say that I made a really neat 3D fly through but all the hard work was done by the fantastic OpenLayers and Cesium libraries.  Once I had the libraries installed, I set up a specific report in GRID that had some pre-canned locations in it so that when you got lost driving around the map you could automatically zoom into these areas to re-orient yourself.

As always, you can leave a comment below or start a conversation with us on Facebook, Twitter or LinkedIn.

Ben

Piers Edit: I couldn’t resist playing around with this again, so here’s a screenshot of one of my favourite places in the Pilbara, Hearson’s Cove:

The post 3D in GRID appeared first on Gaia Resources.

First, let me start by introducing the GRID cast of characters,

GRID Leads:

• Ben – That’s me. I write the code that makes GRID work, so I am the Product Lead for GRID.
• James – Most of you will know James. James does the GRID training and is the person who answers many of the GRID queries that come to us. James spends the most amount of time with clients and knows what features will make GRID most useful, so James is our Support Lead.

GRID Team Members:

• Tracey – Tracey is the latest addition to the GRID team and has been taking on more of the GRID testing to let James spend more time on things like training. Tracey will be picking up more of the level 1 support for GRID so next time you give us a call about a GRID issue and Tracey picks up the phone she will probably be able to answer your question (and wait until you see what she’s done with the GRID Help manual in the near future, too).
• Serge – Serge is one of our software engineers and will be picking up more of GRID tasks into the future.
• Piers – Like James, most of you will know Piers. His day job is to be the Director of Gaia Resources but every so often he gets pulled into the GRID team as a subject matter expert (or he jumps in unannounced to keep us on our toes).

Now, onto the tale of a new GRID feature…

Our story begins when I type the final keystroke and declare the new feature perfect! Bah typical engineer hubris. That is why the first step^* after finishing a feature is to write automated tests. Automated tests are small code snippets that can be run independently to tests parts of GRID. The great thing about automated tests is that you write them once, but you run them often – and since it is the computer running them, it never gets tired, it never complains and it never makes a mistake. They are great at picking up problems where I add a feature or fix a bug but introduce a new problem somewhere else in GRID. So I write the tests for the new feature and then run all the automated tests to make sure no new bugs have been introduced. Now it is time for a code review.

The idea behind a code review is that two pairs of eyes looking over a piece of code are better than one. Serge and I are now working on the GRID code reviews together, and his job is to go over my code meticulously to try and find any problems that I have overlooked. He is going to try things like type in nonsense input or upload corrupt files – essentially anything he can think of to make my code break. It is a bit of a game that we play. I try to make my code as robust as possible, and he will try his best to break it. It keeps life interesting and it helps to make GRID a more stable product for you. If he is happy that the code is stable, he will pass the review and the code is merged into the main GRID source repository.

When the code is merged into the GRID source repository, our continuous integration server detects the change and runs the automated test suite. This happens automatically every time. If a problem is discovered the system sends out an email and it becomes the offenders responsibility (usually myself) to fix the issue. This should never happen of course, so it is a bit of a humbling/shameful experience to break the build – what was I saying about engineer hubris? Still its better to break the build than to deliver poor quality software. When the build passes it is time to update the staging server.

Each GRID instance has two servers – a staging server and a production server. The staging server is where GRID gets tested aggressively. This is usually done by the support team, James and Tracey. They know how GRID gets used in the real world. They know the mistakes that users tend to make and how users expect the system to behave. They are the main point of contact when it comes to GRID support so this is their chance to test the system and make sure that they are happy with the quality of the latest version because when it goes out to you the client, they will be the ones on the receiving end of your phone calls. Often they will also involve the client point of contact to test the latest version of GRID to make sure that the new feature is working as expected.

The ground rules when it comes to the staging server are,

• Do not put any valuable data on the staging server. Testing happens here and as such, you may lose your data since it does not get backed up.
• We will internally update the staging server as new work is completed. This means that the server may go ‘down’ from time to time without any notification.

If the support team and the client are both satisfied, the support team lead James will organise with the client for a time when the production server can be updated. When that happens the new feature becomes available to you.

The whole thing is outlined in the diagram below…

So, what’s new in GRID?

The following is a list of new features in the June GRID update, which we rolled out a month or so ago and has bedded down really well.  We’re about to release another update thanks to some additional work commissioned by the South West Catchment Council (SWCC) across all of the GRID instances in the near future.

Baselayer

• When you search for Baselayers/User Layers or Activity Layers the list of results will show a legend icon. Clicking on this icon will navigate directly to the symbology screen.

Miscellaneous

• The “Data” menu has been renamed to “Baselayer” (Administrators Only)
• “Managed My Layers” has been renamed to “Manage User Layers”.

Bugfixes

This update of GRID includes fixes for several bugs.

Baselayer

• Upload of non GeoTIFF files will be rejected with a message.
• Upload of shapefiles with a space will be rejected with a message.
• Label symbology can now have a halo of zero.

Baselayer

• Non-required image fields can now be cleared successfully.

Bulkdata

• Shapefiles that are not point, line or polygon geometries will be rejected with a message.

Backward Incompatible Changes

Bulkdata

• Shapefile templates will no longer contain spaces. Spaces have been replaced with underscores.

We’ll have some more to include in another GRID release in the near future, as we’ve been working on additional enhancements to GRID over the past month, thanks to the additional funding from SWCC!

If you have any questions about our GRID release procedures please get in touch with us directly, via our Twitter, Facebook or Linkedin pages, or leave us a comment below.

Ben

* Some people like to write tests first and then the feature second – its mostly a personal choice.

The post GRID Release Management appeared first on Gaia Resources.

This week, Ben Richardson from the Western Australian Herbarium visited the Gaia Resources office to spend a day working with me to put the finishing touches on a new image management application.

We spent the day in the conference room, just the two of us with no interruptions. Just think of it – when was the last time you had the opportunity to spend your entire day with no meetings, no email, no one tapping you on your shoulder to ask you just one-quick-question – and all of that time dedicated to fixing the niggling software issues for your customer.

When I am programming, I don’t like to be interrupted. It is an annoyance faced by programmers the world over because unlike many other tasks, it can take up to 15 minutes just to get into “the zone” before anything productive can occur. Each time I am interrupted, I have to restart those 15 minutes just to get back to my previous level of productivity. It doesn’t take very many interruptions before I discover that the entire day has passed and I leave the office wondering what I have accomplished all day. There is, I have found, one exception to this general rule.

This exception is when the feature is so minor that the solution is obvious. The 15 minutes that I’m spending getting into the zone isn’t just wasted time. I am actively trying to construct an mental model of the software in my head so that I can work out what I need to do in order to write the next chunk of code. When the feature is minor however, you don’t have to spend as much time constructing a mental model of the software. The solution is typically obvious and all that is necessary is to physically type in the changes.

This was the case last week with Ben’s visit. He had a slew of minor issues that he wanted to have fixed and I had an entire day set aside just for these issues (some of which fell under our corporate policy of undertaking bug fixes for free). I didn’t have to worry about requirements because my customer was right there to explain them to me. I didn’t have to worry about prioritisation because my customer simply told me what his next biggest annoyance was. I didn’t have to worry about timelines, issue tracking, budgets or user acceptance testing. I simply had to fix one issue at a time, give a quick demo and move on to the next issue.

From Ben’s perspective, he didn’t have to worry about any long winded procedures or communication delays either. He knew what he wanted and all he had to do was tell me how he wanted the software to work. Being in the same room, he could see how long each task took to implement. When completed, he could give immediate feedback and watch the adjustments being made. Being a programmer himself, this was also the perfect opportunity for a code review.

In all it was a very productive day for the both of us. Ben (R) managed to get many of his issues resolved quickly and to his satisfaction and I get a happy customer with software that actually suits his needs. A win-win situation for everyone.

Leave me a message in the comments below, or drop me a line on email.

Ben (K)

P.S. The limes in the photo warded off scurvy nicely.

The post Pair Programming with the Customer for Fun and Profit appeared first on Gaia Resources.

1. Used the Python programming language – leveraging Akeal’s professional development project,
2. Used the Gaia Resources Bird Database – based on the Biological Data Recording System (BDRS), and
3. Involved spatial analysis.

No matter what we chose to do, we had to complete the sprint in 1 working day. Challenge accepted!

With some consultation with Piers and AJ, I decided that we would create a reporting framework for the BDRS. It was going to be a technically challenging task. We had to find a way to communicate between Python and Java, perform spatial analysis and generate reports. Three tasks for three people. I tackled the task of passing data between Python and Java.

At first it seemed obvious, why not use Jython? Jython is an implementation of the Python programming language designed to run on the Java platform. If Jython is supposed to work with Java and the BDRS is implemented in Java then integrating the two components together should be easy. As it turned out, that was indeed the easy part. Less obvious however, was how I was going to provide spatial analysis capabilities in Jython. Most spatial libraries such as GDAL and Shapely have a Python component, however, they also have a C component. Typically these components handle the computationally intensive tasks which happen to constitute almost everything about spatial analysis. Simply put it was like fitting a square peg into a round hole – they were never going to work together. Then I discovered Jepp.

Jepp embeds CPython (the default and most widely used implementation) in Java. I’ll admit that I was dubious. Jepp simply didn’t seem to have the level of documentation or user community to give me that reassuring feeling. Still, I had nothing to lose. I tried it out and much to my surprise it simply worked. It didn’t have much documentation because you simply don’t need it. I tried installing Shapely and it worked right away without any errors. We had a winner.

Akeal tackled the task of spatially analysing the Gaia Resources bird sightings, and this is what he went through:

Akeal: I had just returned from a week-off when I discovered that I was going to be participating in a code sprint – what the heck is a code sprint? I had never used Shapely before, and now I was being asked to work at the same pace as the Developers – all within a day!

So after getting myself stressed out, I decided to get Ben’s help in giving me a push-start in. Once that was done, I was a lot more comfortable sitting with the Developers. At 3:40pm I realised I was running out of time, fast, and made an exclamation to that effect! Luckily, the others had overheard me and told me it was still 2 hours earlier than I thought. Reason? The laptop the guys set me up on had Eastern states time. That’s all I’m going to say about that…

After I was once again comforted (my nerves had taken quite a battering!), I began to enjoy my coding session. My part involved reading in a series of latitude / longitude coordinates representing occurences of a species, then finding the convex hull coordinates for the distribution, and returning the coordinates that make up the convex hull. I used the open source Python module ‘Shapely’ which has many more great features.

Aaron worked on the reporting framework, and his comments are below:

Aaron: I was responsible for putting together the reporting framework on the Python side. This involved using the bridge between Java / Python that Ben developed, providing a user interface and integrating Akeal’s spatial analysis work into the report.

The first step was getting the marshalling between Python / Java working properly. This had a few hiccups and used more time than we had planned for. Next we had to make a web page to show the user. For our templating engine we used Cheetah. Cheetah is based on Velocity which is already used in the BDRS hence it was a logical choice. Finally I had to use the API provided by Akeal and Ben. Get some data from the BDRS, process it using Shapeley and then using the Google static map API, draw a map!

The three of us started the day with the goal of creating 4 reports. However, as 1 pm came and went, we hadn’t yet produce a single report. This was serious! It was time to start cutting features. We decided to drop 3 reports and to redeploy Aaron and Akeal. Akeal would continue with the spatial analysis while Aaron would focus on integrating the spatial analysis part with the browser and BDRS.

It took us right up to the last minute, and perhaps a little more, but we did it. Below is a Google Static Map showing the Occurrence Map of the Cacatua rosiecapilla, based on the records we have in our version of the BDRS – so obviously not an exhaustive series of points to use, but it proves that it can be done.

It was a long day and there were some stressful moments but I think we all learned a lot from this experience. We learned that:

• Ben: When you cannot meet a deadline consider dropping features. In our final presentation to Piers and AJ, they didn’t realise we had originally planned for many more reports.
• Aaron: I am a novice with Python and through this code sprint I learnt a some important points. Code sprints are a great way to develop mental agility and small team coordination.
• Akeal: Working across teams is great! The Spatial and Development teams need to work together more, and it would be great if we can keep finding more opportunities to do this.

The saddest part of this code sprint was that we were only demonstrating that integrating Python and Java to create a reporting framework was possible. We didn’t plan on creating robust and maintainable software and our code showed ample evidence of this. So it was now time to retire our little experiment, store the code in a branch of source control and continue with our usual engineering process.

Ben

The post Spatial Reports with Java and Python appeared first on Gaia Resources.