As we wrote in our previous blog, the “Counting Fish” challenge was put forward by the Australian Institute of Marine Science (AIMS) as part of a call-out to look at innovative and streamlining technologies for a widely used method of marine research data collection. The Commonwealth Government’s Business Research and Innovation Initiative (BRII) has provided the grant funding and program to bring the best minds and solutions to tackle the challenge. Together with our partners at the Global Wetlands team from Griffith University, we’ve recently finished up the first stage which was an intensive 4 month Feasibility Study. 

The study focused on BRUVS (Baited Remote Underwater Video System) footage, and leveraging artificial intelligence (AI) and machine learning (ML) technologies to collect data and accelerate our understanding of fish in our oceans. AIMS and other researchers spend a lot of time manually capturing data from the videos, so finding efficiency measures and improvements to data consistency and quality would be of tremendous value. Out of the study we built a prototype application for processing and visualising BRUVS data, including automatically identifying and counting tropical fish species.  

Taking the OzFish open dataset and many hours of AIMS BRUVS footage, the team focused on training the AI model to accurately identify a range of fish species representing rare and common fish, fast moving and very small species, schools of overlapping fish and also differentiating morphologically similar species. Demonstrating  the effectiveness of our method for these specific challenges, allowing us to produce quantified, highly accurate results. We are now able to look confidently ahead towards tackling hundreds of species that live in Australia’s tropical waters.

The Fishscale online prototype – video metadata and playpack showing annotations and count statistics.

When we look back at it, we’ve achieved an incredible amount in a short space of time. Our nationally distributed team (Perth, Brisbane, Darwin) worked really hard to make sure we were on the same page and productive with online meetings, collaborations and workshops. This was no small feat when you think we had two COVID-19 lockdowns affecting our Queensland team members.

With a new Fishscale prototype web interface, a new BRUVS video can be uploaded and processed within minutes. While the researcher grabs a coffee, it generates the statistics they need to help model and understand population ecology and fish behaviour. There’s an important human quality control element as well, meaning that fish experts have the ability to make corrections, improve the model and increase the value of their data. 

We really enjoyed the regular interaction with the AIMS team as well, which helped us to design our Fishscale prototype with exciting features that will eventually deliver lots of value and efficiency gains for research workflows and other industry applications. 

So what happens next? Well, there is still plenty to do if we progress to the next phase. We know there are still challenges around much larger numbers of species, variations in water quality and environmental factors. In phase 2, our plan includes customising the user interface to adapt to different user types depending on their requirements for data capture and output. Different products based on the AI framework will have different audiences in mind depending on whether they come from research, monitoring, education, or not for profit groups.

We are confident this is just the beginning of an exciting journey to develop a highly valuable product for streamlining research workflows and generation of important statistics. In fact, the Proof of Concept phase starts up around September, and we are hopeful we can progress and continue working with AIMS on this key initiative. 

If you are interested in this space or are someone who works with underwater videos and fish identification, we would love to get your perspective for future development. Feel free to give me a call or an email though if this type of work interests you – strike up a conversation on Twitter, LinkedIn or Facebook. 

Chris