We have had a lot of learning to do as a bunch of technologists working in a specialised area like archives.  We’ve invested a lot of time in training our teams in how to “do” archives – from behind the scenes tours of archives that just give you so much context for everything you see on a screen, to attending professional development events.  And of course – our team themselves consist of people who not only deliver technology solutions – but also that have worked as archivists and collections managers themselves.  This means that we have developed a lot of the subject matter expertise that is absolutely necessary to delivering a good solution (I’m particularly grateful for having some wonderful people who have helped me – clients and colleagues – and my ongoing membership of the Australian Society of Archivists as well).

The Queensland State Archive holds over 67 linear kilometres of records

Open source technologies are a wonderful opportunity for archives -both in terms of costs, and in terms of transparency and accountability –  and it’s something that over 20 years we have built Gaia Resources upon.  This has included originally using open source spatial software like QGIS (which we still deliver training courses in to this day) right through to delivering archival management systems based on software like AtoM, Archivematica and ArchivesSpace.  Unfortunately we do still find ourselves busting myths around open source, and explaining how we support it with enterprise level agreements and the like, but thankfully that’s becoming more and more rare these days.

Open technologies, like open source software, can deliver really excellent outcomes for archives and other collecting institutions.  These technologies can provide transparency in many ways, and we also strive to do that in our project delivery; down to giving our clients a lot of visibility into the “under the hood” ways in which these things work.  It is challenging, but across our work with so many organisations, this has helped us to develop a strong relationship based on trust and openness that has really helped us to deliver across some challenging situations.  So you can see that this open-ness is right through our projects, from the tech stack up right to how we manage our relationships with our clients, and that’s an important thing for me personally.  While we say that we support open software and open data, it’s the “open” bit that’s the most important.

Our work with the Queensland State Archives, using open source software continues to this day

It’s fitting that this year’s Australian Society of Archivists conference is themed “opening the archive” and we’ll hopefully be presenting some of our experiences and ideas in Christchurch in October.  We are also just working on some other archive related initiatives around openness and technologies, and we’ll talk more about these in the coming months leading up to the conference.

In the meantime, if you want to know more, then feel free to drop me a line or start a conversation with us on LinkedIn, Facebook or Instagram!

Piers

The post Opening Archives appeared first on Gaia Resources.

This year, you’re going to see a fair bit from us recapping what we’ve done to get to where we are over the last 20 years, and we thought we’d start this off in January with a bit of a look back to the start of things.

Back in 2004, I had left working in a biodiversity survey company to finish my Masters in Business Administration, and then started the current iteration of Gaia Resources (technically, I used the name back in 1997 and 1998 when I was doing some short term contracts for various organisations).

Gaia Resources started from my home – technically, from a small table in a side area off our lounge room – and look how we’ve come full circle – now, post COVID, we are mainly working from home these days once more.  

The story of Gaia Resources – right back from that table – is one that I look back and think about regularly – seeing some of the challenges we’ve faced and overcome really does help you to have the strength to take on new challenges and to solve other problems that arise.  So I thought I’d just write a little about the areas we’ve worked in over the twenty years as a starting point, and how far we’ve come.

When I started the company we were providing a range of spatial data services to the biological survey part of the environmental industry.  We were busy digitising the maps and tables from a range of different biological survey reports, creating digital datasets from paper sources.  Oh, how far we’ve come since then – like the projects we’ve done like using Artificial Intelligence to capture biodiversity data (such as our work with the Northern Territory).  One constant though – it has also been important to interacting with biodiversity data standards when doing this data collection (like the TDWG standards and the new Australian Biodiversity Information Standard), as we have done that throughout our history.

This pic was from The Stagg cafe in Hobart during the TDWG 2023 conference

In 2005 we saw the arrival of Google Maps, and that started to change the landscape of spatial data quite quickly – all of a sudden it was much more desirable to present spatial data through the browser.  Google Maps started to get traction over the next few years and we realised that digitising and producing paper maps was on the way out, so we started to hire software engineers into the company to build systems to manage spatial data.  To this day we are continuing to develop these biological data systems, such as our work on the Western Australian Biodiversity Information Office (both design and build) and the federal government’s Biodiversity Data Repository.

During those early days, we also found ourselves working with the Western Australian Museum, where we were supporting their collections databases – registers of all the vouchered specimens that they have in their collections.  This led to a chance meeting and discussion around Archives, and then we were providing services to a whole new sector, which has become one of the areas I’m very proud of.  

We have delivered a range of open source collections databases to the Archives sector in Australia, across Western Australia, Victoria and most notably Queensland, where – on the back of our work with the Queensland State Archives – we set up our second office in Brisbane.  All of a sudden, we were an Australian company – not just a Western Australian one.  So, from our origins around that little table, now we have offices and staff right around Australia – that’s been a big change!

As an aside – the Archives bug had bitten me in particular, and how!  Being at the most recent national Archives conference in Melbourne just reaffirmed how important this sector is – and how much we enjoy being part of it.  Going from our simple first steps of implementing collections databases to now implementing complete archival systems, including comprehensive digital preservation systems – all of this has been a big shift from our origins, but in the right direction!

Luke, Sarah and I went to the ASA conference in Melbourne in 2023 – our sixth one. Dennis Lillee was there already

The Environment and Collections areas have become pillars of what we do at Gaia Resources; there are other areas we also work in, but these two have come to be our mainstays.  We’ve even now designed the company to have these as our “units” – so that we are focused on our clients in these areas, and delivering high quality services to them.

There are a lot of people that have helped Gaia Resources get to the 20 year mark; clients, colleagues, friends and family.  But throughout the whole thing we could not have done what we have, as well as we have, without our team – our staff, our family away from home.  These people – past and present – have all contributed in some way to the organisation and without them we would not be here.  So to talk about the history of Gaia Resources without the people that came on the journey – so thank you to everyone who has, still does, (or will) work here at Gaia Resources – the place wouldn’t be what it is without your input along the way.  Thank you.

The team at team week in 2022 in Perth – one of the fondest memories from the last couple of years (which will be repeated this year – hopefully without COVID!)

I’m forever grateful to lead this team on our mission of making the world a better place, through the delivery of sustainable technology solutions in a responsible manner.  This year is going to be one where we get to celebrate that just a little bit, and that’s going to make for some fun times ahead.

Stay tuned for more about our history over the year, and for some bright new initiatives that we’re heading into.  Meanwhile, if you want to know more about us, why not drop me a line on email, or through our social media channels – Facebook, LinkedIn, X/Twitter and now Instagram!

Piers

The post Twenty Years appeared first on Gaia Resources.

Sadly, there’s a lot of misinformation out there about topics that impact DPS.  I’ve heard all sorts of things that are either mistaken or misguided.  For us, the key is to work transparently and honestly with your customers (or rather, as we call them, partners) and to help the customer to develop the knowledge and skills they will need to move forward on this journey – and putting more information like this out there hopefully helps.

First, let’s talk bout what Digital Preservation is, and how important it is for a range of areas – not just collecting organisations.  

There are various definitions of Digital Preservation:

• Digital Preservation refers to the series of managed activities necessary to ensure continued access to digital materials for as long as necessary  …(digital preservation) refers to all of the actions required to maintain access to digital materials beyond the limits of media failure or technological and organisational change. (Digital Preservation Coalition)
• In library and archival science, digital preservation is a formal endeavour to ensure that digital information of continuing value remains accessible and usable. It involves planning, resource allocation, and application of preservation methods and technologies, and it combines policies, strategies and actions to ensure access to reformatted and “born-digital” content, regardless of the challenges of media failure and technological change. The goal of digital preservation is the accurate rendering of authenticated content over time (Wikipedia – which we don’t usually quote too often but this one is nicely holistic)
• Digital preservation consists of the processes aimed at ensuring the continued accessibility of digital materials. To do this involves finding ways to re-present what was originally presented to users by a combination of software and hardware tools acting on data. (UNESCO)

They’re all well and good, but the simple way that we think of Digital Preservation is this: it’s the way in which you make sure that the digital assets that you have under your care last forever.  

But what does that actually mean?  Well, it can be an addition to a digitisation program, but it’s definitely not just a hard disc sitting on a desk somewhere.  It’s a combination of a range of things – careful planning, careful ingestion of digital files, careful storage and monitoring of those digital files, a lot of skill and knowledge development in your people, and above all, careful and regular review of how you treat these digital files.

So you want a Digital Preservation System? Let’s talk about what this looks like for you moving forward.

First of all, we firmly believe that you need to understand exactly what this journey to a full live digital preservation system will look like before you start doing anything.  The journey itself will probably include stages of roadmapping, digitising or collecting born-digital materials, implementing the Digital Preservation System (DPS), training your people and then going live, as shown below.

But what do each of these mean and entail?

Roadmapping

Here we really are talking about a discovery and planning phase for your journey into the realm of digital preservation.  You need to be aware of what this journey is going to look like so that you can prepare for it and get all the right gear ready!

We recommend that organisations work on this through a basic series of capacity analyses – looking at some of the key pillars of people, systems, infrastructure, data and processes.  By doing this sort of a review you can then work out the gaps of what you need to have running for your desired future state – of preserving your digital information.  At Gaia Resources we do this as collaboratively as we possibly can, through interviews, workshops and all sorts of other methods.

Once you know where you are, and the gaps you need to fill to get to where you want to go – then it’s a matter of eliciting options, determining the pros and cons of each and then putting that into a roadmap.

Digitisation

You may have already started collecting born-digital materials, or you may be creating them from your existing collection by digitising them. Whilst Digitisation isn’t Digital Preservation, it can be a step towards. It. 

Digitisation is the process of taking something that isn’t digital and making a digital representation of it.  This is usually things like scanning pages, or books, but can be doing complex 3D scans of objects, or transferring from historical tape media to digital media.  Digitisation is a big and complex beast, and it requires a lot of expertise to be done right.

Some organisations may have already started digitisation, but you will need to be looking ahead to how you’ll preserve these digital materials for the long term, store it securely so it’s not lost, and prepare for migration of the data about these digital objects (something we talk about later).

We do recommend that the digitisation component of a DPS implementation can start once you’ve done the roadmap – i.e. once you’ve made some decisions about how you will be operating the DPS and the sorts of decisions you will make around storage quantities, file formats and so on, you can then start digitising – and storing it in a secure way so that it won’t get lost.

But you can start digitising in a “do no harm” approach like this pretty quickly after the roadmap is underway.

Implementation

In this stage, you will be well on your way to your digital electric dreams. This is going to be a complex information technology, business process and risk management project – and like most projects, you will also need a good change management process around it.

The way in which this stage will play out will depend on what you’ve decided in the roadmap, and where you’ve decided to go in terms of the technical aspects of the project.  If you’re doing this in house, on your own servers, then that’s a vastly different beast than going down an “as a service” approach (like we do with the Queensland State Archives, and others).  We strongly advocate for open source solutions (especially for archives, museums, and other collecting organisations), for various reasons we’ve outlined in places like here and here.

An example digitised item from Queensland State Archives: Item ID: ITM1640266

A key aspect I’d like to highlight of this stage – don’t rush it.  You need the time to implement this into your organisation.  You’re talking about procedural changes, technology introduction, and there is a lot to consider about how it will change your organisation and practices – so be careful and considered about this part. Most of the time it’s better to do digital preservation “right” than “fast”.

And before you know it, the big day is here…

Go Live

Congratulations – you made it to your system going live! Going live is always a mixture of anxiety and excitement; sometimes it’s a big ceremonial bang and sometimes it is just a quiet button pressed on a screen.  

For Digital Preservation, one of the things that’s going to be a bit of an anticlimax is that as soon as you do that first ingest, the files go into quarantine to make sure that virus definitions catch up and we don’t end up digitally preserving viruses (which of course, you might want to do if you run something like The Malware Museum!).

But what about those other two areas on the diagram – skill development and running the system?  A quick couple of final points:

Skill development

Skill development takes time.  

While you should be ergonomically developing the system (making the system meet the needs of the people using it), you also should be training the same people and developing their skills along the same time.  Skill development is something that we feel should start as early as possible, and ideally you should be doing it as you develop the system.

Some of this training is going to be targeted and specific, but a lot of it is going to just be about exposure.  Remember, this is quite a different beast than an Archival Management System, and requires some different skills – so you will need to be aware of this and cognisant of the skills of your team working on the DPS.

We would strongly recommend that you go “live” with an internal version of the system first.  Then you can take the material you have been digitising and start ingesting it through the DPS as the key way to do some skill development. 

Running the system

Once it’s up and running – then the fun starts.  You’ve realised your (electric) dream!

It’s time to tune your machine for maximum performance and flexibility.  To do that, you need to be looking at how the DPS runs – you will be tuning infrastructure, and making sure you have storage available, and a range of other technical tasks.  But you will also be turning into a “format whisperer” – you’ll be looking at what existing and new formats come through the system and determining how to best deal with those for the future as well.  There’s a lot to do here – but there are lots of resources available to help, and people that can support you, like your implementation partners.  You might get to play with a few interesting technologies as well – including Artificial Intelligence, like we’ve been doing lately.

So there’s a run through of what it takes to get cracking on a journey involving Digital Preservation.  If you would like to know more about any aspect of this, then feel free to start a conversation with us on social media –  LinkedIn, X, or Facebook or to drop me a line at piers@gaiaresources.com.au. 

Happy World Preservation Day – may all your bits be in the right order and your formats readily readable into the future!

Piers

The post Digital Preservation: building the Electric Dream together appeared first on Gaia Resources.

Earlier this month Piers Higgs, Sarah Aldrich and I took a trip to Melbourne to attend the Australian Society of Archivists (ASA) 2023 conference at the Melbourne Cricket Ground (MCG). With a sense of enthusiasm on the flight over I was keen to get down into the details about archives and solve some challenges with my fellow conference participants.

My trip started Monday at Perth airport grabbing a coffee with Piers before our flight to the great city of Melbourne. Upon arrival in Melbourne we met up with Sarah, one of Gaia Resources’ Collection Analysts based in Canberra, and headed to our hotel located near the MCG. After a social dinner we cracked down on what our plan for ASA 2023 was, who is attending each session and what we wanted to learn. A screenshot of the event agenda can be seen below.

DAY 1:

An early morning run was the start of the day for Sarah, Piers and I decided to keep it a bit more relaxed… We met Sarah for coffees and breakfast at a local cafe then got ready to attend ASA 2023 – Rising to our Challenges. Upon arrival I was amazed to see the variety of archivists from all over Australia and even from Singapore, archivists representing the state archives around Australia but also archivists from local historical societies and schools. Day one had a focus on first nations archives and living history, as well as discussions on the use of AI in archives and the challenge for institutions in archiving big data. 

Day 1 Pre-conference selfie L-R Piers Higgs, Luke Connelly and Sarah Aldrich

Day one went by in a flash, there was a flurry of activities, plenty of notes and a new perspective on the challenges that archives are facing. After wrapping up the day the Gaia Resources fly-in team caught up with some of our Melbourne based team members. An interesting dinner in Federation Square was followed up by some ‘team-building’ at a few bars in the Melbourne CBD. Although Gaia Resources is an online based company it is great to meet co-workers in person and learn more about their lived experiences.

L-R, Sally O’Callaghan (Past Gaia Resources Employee, now working for the NAA), Drew Robinson, Sarah Aldrich, Luke Connelly, Piers Higgs

Day 2:

After a relaxed morning and breakfast, day two of the conference began with a panel discussion from the Queer Archives of Victoria. Then we began our sessions for the day covering digital preservation and digital transformation, with sessions hosted by the National Archives of Australia and the Public Records Office Victoria. It was great to see how archives benefit from technology like the ones developed by Gaia Resources. At the conclusion of the conference we heard from two industry experts, Pia Andrews and Barbara Reed, discussing governance in record keeping and how to personally stand out in archives. The launch of the 2024 ASA Conference wrapped up the final day and Gaia Resources may be heading off to Auckland next year…

Each of us also tried to put together a short summary of our conference highlights, including:

“It’s been 8 years since my first archives conference, and as an IT professional I find that I’m still learning more about how archives undertake their practice – but I’m glad to keep learning from them and trying to make sure we can deliver even better solutions for their problems.”

• Piers Higgs

“Having worked as an archivist prior to pivoting into tech, attending the ASA conference gives me the chance to act like a kid in a candy shop. It’s always a pleasure to connect with archivists, hear about their wins over the past year, talk shop, and discuss how tech can work to their advantage. I enjoy learning and keeping up with the current practices.”

• Sarah Aldrich

“The ASA Conference gave me insight into the challenges that archives are facing and how Gaia Resources can help these organisations grow through custom technological solutions.”

• Luke Connelly

In wrapping up my brief conference breakdown here are some key learnings highlighted by the Gaia Resources team from the 2023 ASA Conference:

1. Archives are essential to preserving our history both personally and on a government level: each record has at least one person attached to it, we must remember the importance of each record and the personal value attributed to it.
2. Conversations around digital preservation and technology such as AI are progressing. 
3. There’s still work to do around bridging the gap between Collecting organisations and technology solution providers. 
4. Gaia’s work and with our clients has demonstrated innovative solutions in the forefront of the field, notably our implementation of the archival management and digital preservation system for QSA and our work on Clio.

If you would like to know more about our digital preservation services and archival storage solutions please don’t hesitate to contact me or start a conversation with us on one of our social media platforms Twitter, LinkedIn or Facebook

Luke

The post Rising to the challenge: Recap of the 2023 Australian Society of Archivists Conference appeared first on Gaia Resources.

The ‘Workflow Webinar’ series was hosted by the DPC-led, Community Owned Workflow (COW) group. It is a forum developed out of the need for shared Digital preservation knowledge and workflows amongst practitioners. The COW resources are hosted through the Community-Owned digital Preservation Tool Registry (COPTR) wiki. It is a great online resource and is worth looking through when you have the time. https://coptr.digipres.org/index.php/Workflow:Community_Owned_Workflows 

QSA and Gaia responded together to the request for presenters and we were booked into Episode 3 of the webinar series in March. The aim was to highlight the primary decisions and the tools that were implemented to enable the Archivist workflow. Of key importance was demonstrating how QSA planned to do Digital Preservation at scale – including large ingests, as well as collecting and preserving the large influx of digital material in the pipeline. 

On the day, QSA and Gaia Resources split the twenty-minute time slot, with QSA leading the discussion. They laid out their plan to address legacy records transferred to them on physical media types, which are quickly becoming obsolete (think floppy disks). Gaia Resources followed with an explanation of how and why with our consortium partners, Hudson Molonglo, Record Keeping Innovations, and Artefactual, we built a pre-ingest workflow into ArchivesSpace. The integration of the Digital Preservation System into the existing Archival Management System provides the Archivists with a single point of truth, ArchivesSpace. The pre-ingest workflow in ArchivesSpace authenticates the digital material sent by state agencies, ensuring files have not encountered alteration or bitrot before being imported into the Digital Preservation System. Once the checks are complete, the user clicks the ‘Begin Import’ button, initiating the Digital Preservation workflow. 

Our presentation was well received and nicely expanded on the previous talks in the session. The first session was a great talk by Julia Miller from the PARADISEC Archive. She focused on small archive workflows and talked through the tools that are used at PARADISEC. However, another excellent component of her talk was about the education tools PARADISEC provides for other small archives. Building on Julia’s talk, Andrew Martin from DAMSmart discussed the work that DAMSmart has done with PARADIESEC to digitise and preserve their collection, as well as, provide education around tools and uses. Thus, Episode 3 of the series covered ways that small archives can get started in digital preservation and moved into decisions and considerations when conducting digital preservation activities at scale.

Our team has enjoyed working in digital preservation, collaborating with clients and staying in tune with the wider field. We have participated in continual education opportunities in the digital preservation field and work to gain insight into the processes that others are implementing across the world. Some of our members of the collection team recently attended a DatacomIT webinar ‘Where to Start with Digital Preservation’ in February. It was a high-level session that gave an overview of the fundamentals of implementing digital preservation in various organisations. This session included an introduction from Robin Wright, the Head of the local DPC Australasia and Asia Pacific branch. It is always a good feeling at these events when you fact-check conversations that are had internally and with clients and you know we are headed in the right direction. 

We look forward to exploring future opportunities and expanding our engagements.

Are you interested in digital preservation? Get in touch with us via email or start a conversation with us on one of our social media platforms Twitter, LinkedIn or Facebook

Sarah

The post Working on and Speaking to Pre-ingest Digital Preservation Workflows appeared first on Gaia Resources.

It’s wonderful when you can actually find your tribe in this rather interesting world, and I think both Sarah and I felt like we were well accepted into the archival community through these events.  It’s great to be talking about all sorts of things and attending talks that expand on our knowledge of this area, and to be able to have those collegiate discussions in the breaks with contacts new and old.  It was also great to have an environmentally themed keynote (and a few other sessions), since we are actively trying to bring the knowledge of archiving and digital preservation to the environmental technology side of our business.  It was extra special to see our colleagues and clients from the Queensland State Archives present on the Digital Archiving program that we have been working on for a long time (2017 feels like a long time ago? A lot has happened since then).

Josephine Marsh presenting (virtually) on the Digital Archiving Program for Queensland State Archives

The content of this year’s conference had us thinking about a few key takeaway points:, including the fact that smaller archives are still struggling with their remit in a very resource constrained environment and that there are a lot of organisations starting on some sort of journey.  There were also a few of “the usual” presentations from the community – in these difficult times it seems that innovation is lagging a bit, but we did have a few conversations around Artificial Intelligence in the archives, following on from a previous blog and our work with transcription.  However, in watching these talks, the most pressing thing that we saw was a need for digital literacy and digital education amongst the community.

Digital literacy and digital education across a sector is something that I remember working through in my time with a different industry, the spatial industry.  A key segment of that community were cartographers, focusing on making maps, and the rise of digital technologies means that those positions were considered to be “under threat” from people with backgrounds in technology.  They were facing a very real change in that some of the profession was ‘stuck’ with old skillsets.

In archives, we face a similar challenge.  As archives embrace the digital, we need to work on building digital literacy and digital educational programs that will help our archivists build the skills that they will need to be archivists into the future.  This is readily transferable – the mindset of someone assessing a box of paper versus assessing a hard drive of documents is the same, but the skills that are needed to do it are the things that we can develop as a community.  Sarah and I came up with a bunch of ideas for digital archival training and we will be working more on that as part of our strategy for the next year, and approaching some of our archival training organisations as well.

It was great to be back with our tribe, and for me it was great to be back in Canberra to meet a range of colleagues face to face – the next one is all about “Rising to our Challenges” – so we’ll put a few challenges out there for that conference as well, and look forward to meeting everyone again in Melbourne!

In the meantime, if you’d like to know more, start a conversation on our social media platforms – Twitter, LinkedIn or Facebook or send us an email.

Piers

The post Here we are (again?) appeared first on Gaia Resources.

Our latest proof-of-concept activity with our valued clients at Queensland State Archives (QSA)  has seen us helping to tap into the information stored in the archives. To do this, we used Clio to transcribe handwritten letters and historical documents that tell the story of the turbulent early contact between Australia’s First Nations people and European settlers in Queensland. 

This work with QSA’s First Nations Access and Engagement Team is part of their Frontier Wars project that includes a significant collection of digitised 150 year old handwritten documents and letters, a community transcription initiative, an exhibition and a three-part SBS documentary series called Australian Wars. The documentary series premiered on Wednesday, 21 September, and it tells the story of Australia’s frontier conflicts that swept the country over a 100-year period.

When David and his team approached us with the challenge of transcribing 250 scanned handwritten letters and documents, we quickly jumped on the opportunity to be involved in such an interesting project. David was considering his options, including:

• Open community transcription, where volunteers manually transcribe documents through FromThePage software,
• Confidential online transcription, where the transcriptions are done manually by a small number of trained and certified professionals, and
• Automated transcription using Gaia Resources Clio prototype.

Using Clio for this project was an attractive option for transcribing sensitive handwritten documents quickly and efficiently; however, the varying cursive styles, condition of documents, formats and orientations of text presented several challenges. The benefit of Clio is that it is very fast to turn handwriting into a text document, so it can be easily scaled to hundreds and thousands of input documents.

The results from Clio can vary between getting just about every word correct with a few typos here and there, to only getting about half the content correct; and there are several factors that control that. As shown in the samples below, there are several variables to contend with, including blurred and crossed-out words, damaged pages, angled writing, newspaper formats and scans of books. Some of these require image pre-processing to get the best possible result.

Examples of the various types and conditions of the Frontier Wars documents

David opted for a combination of automated and open community transcription, and worked with us to take the results from Clio as a starting point for more efficient community transcription in FromThePage software. We all knew that automation alone would not solve the problem, but Clio could help speed up the process. The idea of going forward with that combined approach was that if someone was to start from nothing, it might take them an average of 20 minutes to do a transcription; but with the Clio result as a starting point, they might only take 5-6 minutes to correct errors and move onto the next document.  

What the QSA team found was that the Clio results – although variable – were better than they had expected, and were a great help in saving many hours of manually intensive work. Not only that, but enthusiastic members of the public enjoyed having a look at how the ‘machine’ had interpreted certain bits of text. In some cases, they could raise an eyebrow over something that is quite obvious to the human eye; but the efficiency gain becomes undeniable when scaled to a whole collection of documents. 

Delving into our process and results a little, we were able to use a Clio-derived (e.g. computer) confidence in prediction value to categorise each document’s predicted transcription accuracy. Each term, word or text component in a document is assigned a confidence in prediction value based on how confident Clio is of its transcription for that component. The Average Confidence Level can then be calculated across the entire document.t. 

Below are the Confidence Level categories used and their corresponding average confidence in prediction ranges:

• Very Good (between 90% and 100%)
• Good (between 80% and 89%)
• Above Average (between 70% and 79%)
• Average (between 60% and 69%)
• Poor (between 40% and 59%)
• Very Poor (between 0% and 39%)

Before we started, the QSA team suspected they might only get about 40% of the content accurately transcribed using Clio. The graph shows the percentage of documents that fell into each Confidence Level category.

Predicted Transcription Quality

We had no documents in the Very Poor category. So 100% of documents had an average confidence level exceeding 40%. Although it is only a computer derived measure, it hinted at how well Clio was coping with these historical documents. 

This combination of automated and manual process saves time and enables people to get on with the important stuff that AI can’t do, and it follows that there are consistency and security benefits to go with a more efficient workflow. 

Some other improvements we were able to implement in pre-processing the scanned documents included splitting ‘double page spread’ formats into separate pages. Below is an example of the inputs and outputs before and after splitting the pages:

An example of a double-page spread transcription that was rated as Good

While the statistical results of these may not have changed, the correctness and readability of the output is certainly better and important for the downstream review process. 

As mentioned above, the condition of the original documents varied for a number of reasons. When it comes to documents with multiple orientations of text, Clio will only read one orientation per document n a single page, or it may merge text from adjacent blocks that are meant to be read separately – such as margin text or dates. 

Future improvements could be gained with additional pre-processing by segmenting multiple text orientations, as illustrated below:

An example of multiple text orientations (left) and future approach of isolating blocks of text (right). (for illustration purposes only)

We also could do further post-processing as well as the pre-processing outlined above to improve results.  For example, we could implement a method using the Levenshtein distance algorithm which would assist in finding similar strings in other documents and replace it with a more likely string. In this way we could correct ‘Social socuriy gumber’ to ‘Social Security Number’.

Overall, we were really impressed with the accuracy of the Clio prototype in reading 19th Century handwritten letters and folios. Clio’s models have been well-trained and are very good – not perfect, but very impressive and they are continually improving as more information is passed through it and the corrections are fed back in. Often the model is better at reading old handwriting than I am!

From our point of view, this has been an exciting Proof of Concept as it shows that it is possible and viable to unlock the vast and valuable information stored in archives, making the content more accessible and searchable to the general public. 

If you’ve got data in a soft or hard copy format that you need transcribed, then reach out to us and let’s see how we can help you solve your problems.  In the meantime, if you’d like to know more, start a conversation on our social media platforms – Twitter, LinkedIn or Facebook or send us an email.

Gail

The post Harnessing the power of AI to unlock the archives appeared first on Gaia Resources.

Many of our blogs on open source software focus on spatial tools and the hard sciences. What we haven’t delved into recently is the importance of open source software in the GLAM sector (Galleries, Libraries, Archives and Museums). Ten years ago, Piers wrote a blog about open source collection management and it’s time for an update. Today we will focus on the growth of these solutions, our continuing support of them (and of that sector), and the open source projects we’ve been putting in place recently in GLAM organisations. 

First, to recap, open source software means that the source code is published freely – anyone can download a copy of the code, use it, and customise it. There is also the benefit of reduced ongoing costs – rather than continuing to pay annual licensing fees, open source software installations require only hosting, and upgrades to the code. In addition to financial sustainability, open source software provides technical sustainability. Because the code is freely available, it means that the community of users can contribute bug fixes and improvements in an ongoing fashion. Realising that many GLAM organisations don’t have specialised IT departments, implementation of the software and upgrades is where Gaia Resources can assist, and we have various levels of support that can be tailored to an institution’s needs. 

Open source software also supports the evidential value and provenance of your records – the source code is freely available and can be audited to ensure that your data is not being changed or manipulated by the system. Open source software provides a level of transparency for institutions that need to be able to attest that their collections are untampered with. 

When working with a collecting organisation, we usually recommend one of three open source tools; Access to Memory (AtoM), CollectiveAccess, or ArchivesSpace. The solutions we provide can be out of the box – that is, it is installed as “vanilla versions” without customisation – or in some cases, they can be heavily customised such as with our Queensland State Archives implementation of ArchivesSpace. We have particular experience in implementing the Australian Series System for archives in all of these software packages. 

Access to Memory is an open source tool that is developed and maintained by Artefactual (one of our partners in the current Digital Preservation project for Queensland State Archives). Artefactual also develops and maintains Archivematica – a tool for Digital Preservation. AtoM, as it is known colloquially, is a great tool for small to medium size archives. It provides not only an easy to use interface for staff, but offers an immediate web presence that allows public access to search the collections. For many of our customers, we have implemented add-ons or plugins to provide customised functionality, ranging from subscriber-only access to digital materials to online ordering. 

CollectiveAccess is another popular open source collection management tool. Its flexibility is a key benefit of the software, and it can be implemented to manage collections of museum objects (including all SPECTRUM functions), archival records, and digital materials. We have implemented CollectiveAccess for several clients from across the GLAM sector, from archives to high-end art collections, and even not for profits managing their own historical collections. Whirl-i-gig, out of the United States, maintains the source code for CollectiveAccess, and our own Gaia Resources developers have contributed bug fixes and added new features to the source code as we make improvements for our clients. 

ArchivesSpace is the tool that we have implemented at the greatest scale. Queensland State Archives uses a highly customised version of ArchivesSpace to manage their 64 kilometres of government records. While ArchivesSpace does come with a public interface, our work with QSA included a custom-built public interface, ArchivesSearch, and we have also implemented these systems for clients in Tasmania. Lyrasis, again in the United States, maintains ArchivesSpace core code.      

With our experience over the last fifteen years, we understand the needs of collecting organisations and can recommend solutions that fit the needs and scale of the client. We stand by our passion for open source software solutions and advocate for them as the most sustainable solution for collecting institutions. Whether your organisation is small and volunteer-run or you have kilometres of records, one of the above systems can be implemented in a way to suit your needs. 

Think we can help assess or even customise the best tool for your organisation? Get in touch with us via email or start a conversation with us on one of our social media platforms – Twitter, LinkedIn or Facebook.

Sarah

The post Open Source Solutions in the Museum and Archive Worlds appeared first on Gaia Resources.

When we saw that the DPC were putting on an event that included both archives and spatial data, I was super keen to be involved.  My first degree in Geography was all about spatial data – the love of which persisted through my ecology and environmental stage of my career to see the creation of Gaia Resources.  Along the Gaia Resources journey, as I talked about briefly at the recent Archival Society of Australia WA branch meeting, we have ended up working with Archives all around Australia… and then suddenly there was the DPC event, combining those two topics and giving me an excuse to stay up late and enjoy myself.

The event itself was run online, and included a series of talks followed by a short question and answer piece at the end, with some good breaks in between to present attendees from going mad looking at a screen for four straight hours.  The event itself was expertly facilitated and run by the DPC staff, which is still no mean feat these days.

The talks were varied, and included groups from a variety of organisations and industries, including:

• Organisations that were decommissioning first generation nuclear power plants in the UK, and trying to deal with the challenges of legacy spatial data,
• The British Geological Survey, who provided an overview of how they’re going to deliver their spatial data into the future (more on that later)
• Data standards bodies like Geonovum in the Netherlands, who pointed out how the W3C and OGC standards have been separated for way too long, but have been brought together with their joint working groups, 
• The UK Geospatial Commission, who were discussing how to provide data of high quality, that was also Findable, Accessible, Interoperable and Readable (FAIR), 
• The US Library of Congress, talking about what sorts of formats they accept into their archive through their Recommended Format Statement, 
• A case study in the archaeology profession in the UK (and the lingering love of the shapefile), and
• A final case study on the National Library of Scotland, and how they are preserving historical maps and making those available (and I then spent ages browsing their website – linked below).

The National Library of Scotland “Map Images” website – find it at https://maps.nls.uk/geo/explore/

There was a lot of content in here to digest, and I pulled out a few points that I’ll summarise as the highlights for me, which were:

• The rise of the Geopackage – as highlighted early in the British Geological Survey talk, they are moving towards delivering data in API style feeds (using the OGCAPI feeds) and delivering files in GeoTIFF, GeoJSON and Geopackage only.  This was echoed in the Library of Congress preferring the GeoPackage format as well – and for many good reasons.  If you’re working with digital spatial data, then you need to start investigating Geopackages and how they’re emerging as a great open standard,
• International differences – listening to talks from the nuclear decommissioning crews and the archaeologists reminded me of how spatial data in the mining industry was at way back in the early days of Gaia Resources when we were working with Western Australian mining companies to digitise all their legacy spatial data, and how far we’ve come since then – which will be the subject of a future blog from me as well,
• Active community – there are a whole raft of people out there in the world who are looking at the same challenges we are in terms of digital preservation of spatial data, and the DPC seems to be a great place to connect to them – which we’ll be doing, and we’ve reached out to the DPC to see how we can help and be involved, and
• Practical, pragmatic implementation – while I love detailed standards like any other spatial/archiving nerdy type, the practical applications of these were what really stood out in the talks.  Standards can often be developed in ivory towers, away from the practical implementation of them, but I saw some very close touchpoints here across the talks and as a result, there is plenty of good guidance for operators to follow.

I was also pleased – and somewhat relieved – to see a whole bunch of things in the talks that reminded me of work that we’ve done.  Apart from the example of our digitisation work in the mining sector here in Western Australia, the National Library of Scotland “Map Images” site is our Retromaps project on steroids, and there was a fair bit of thought around the use of GeoPackages, which we’re building into several of our larger data collation and aggregation projects at the moment.

It’s definitely useful to see historical data, and making it available is a challenge. We had a go at @gaiaresources with the Retromaps project with @StateRecordsWA https://t.co/PmFxm92qPi #dpc_geo #archives #spatial

— Piers Higgs (@piers_higgs) March 30, 2022

This was my first introduction to the DPC – I’ve heard about them before, but never been actively able to participate in an event.  It certainly was worth staying up late to get these different perspectives and how these different players deal with those challenges of archiving spatial data.

If you’d like to talk more about the event, or find out more about how we’re working on capturing, managing and archiving spatial data in a range of industries here at Gaia Resources, feel free to drop us an email or start a conversation on our social media platforms – Twitter, LinkedIn or Facebook.

I must also say thanks once again to the team behind DPC, and all of the speakers, for an interesting Wednesday night!

Piers

The post Archiving Spatial Data appeared first on Gaia Resources.

Over the past several years, Gaia Resources has grown its team considerably due to an increase in demand for our services. However, there are still plenty of occasions where we acknowledge that a specific project or undertaking requires additional resources or skills, beyond what our own team delivers. Sometimes this is in an opportunity to move into an entirely new area (when we partnered with the Global Wetlands project for a fish identification solution using AI), or in delivery of a specialised solution that we want to ensure we bring all of the best knowledge and skills to the area (such as the consortium of organisations we have brought together for the delivery of the Queensland State Archives Digital Preservation project.)

Sometimes it is because our values align so well with a partner it is almost like considering them an extension of the team.

This is the case with our partners in the digital experience agency, Liquid Interactive. We have worked so well with them, and so often that we will not only consider them for projects that we know will utilise their skills and specialisations, our own team is excited for each opportunity to work alongside them.

We have worked with them as sub-contractors to deliver development support for Drupal websites in GovCMS; they have assisted us on projects where providing data delivery with a great user experience was key; when one of our team members relocated to a city without any other Gaia Resources staff, we set up her office to be colocated with Liquid Interactive team members. I have also been fairly regularly featuring their Future Led Series of events in my own blogs, and while the events themselves have been a fantastic opportunity to hear informed and varied opinions on topics that matter to our future – I also have to confess to enjoying an opportunity to get to know their team a little better.

Liquid Interactive themselves was first introduced to us through one of our previous employees, Morgan Strong, who has engaged them successfully in previous roles. Morgan has moved on to a role as the Digital Transformation Manager at Queensland Art Gallery & Gallery of Modern Art (QAGOMA), but is also still heavily involved in our organisation as a Drupal and Open Source advocate, and as a client in delivering Collection-based projects.

The world is ever-changing, but I believe there are some things that remain the same. One of these is that as much as what we deliver is important, how we do it is also impactful. It is through our connections with other people that we come to understand the world, so building positive connections can only help us to find our place to deliver the most to our community.

If you are interested in working with an organisation that is collaborative, people-focused, and provides a supportive culture for teamwork, reach out to us via email at info@gaiaresouces.com.au or connect with us on Facebook, Twitter or LinkedIn.

Sophie

The post Partnerships, people and change appeared first on Gaia Resources.

We were asked to present on the work that we do in the archival collections around Australia, so we gave a quick run through of a range of projects that we’re working on around Australia, and talked about some common and different functionality. We then talked through some of the technology and future things that might be of interest to archives we’ve been looking at. 

The talk was given in person (with a COVID safe setup) and also on Zoom – and the Zoom call was recorded as has been made available below.

https://vimeo.com/ausarchivists/review/686909468/047a43727d

It was nice to see a few of our clients and some new faces as well.  One thing that I really like about the way the ASA operates is that it welcomes in new organisations that are seeking to learn about archiving, and some of the people we met that night asked a whole bunch of interesting questions in the session and afterwards around the technology side of archiving.

Over the years, the ASA has facilitated ways for us to learn a lot as a team about archives, and it was good to be giving something back to them and their members!  If you have questions about archives, then feel free to get in touch and ask either Meg or myself questions via our enquiries@gaiaresources.com.au email address – or start a conversation with us on our social media via Facebook, Twitter or LinkedIn.

Piers

The post Archives now and forever! appeared first on Gaia Resources.

Item ID: ITM1640266

Gaia Resources is a proponent of open-source solutions for cultural institutions, particularly for long-term sustainable information management. Without restrictive licensing conditions, you know that the underlying code of a system can be visible to you and within your control in the future. If a specific vendor is no longer available, the code itself can still be taken up by others without risking a loss of access to your digital assets.

In 2020, Gaia Resources along with our consortium partners Hudson Molonglo and Recordkeeping Innovation, built and implemented a new Archival Management System (AMS) using a customised version of the open-source software platform ArchivesSpace, as well as a portal for Agencies (ArchivesGateway) and a new public interface for searching the archives (ArchivesSearch). This project was delivered to the Queensland State Archives (QSA) with the support and assistance of the Digital Archiving Program (DAP).

Following the successful delivery of this trio of systems, the next important task in the digital transformation of Queensland’s archives was digital preservation. To ensure the successful delivery of a comprehensive digital preservation solution, our next step at Gaia Resources was the validation of key questions and assumptions via delivery of a Proof of Concept (PoC) to QSA.

One of the original implementation team project managers, Megan Edward, began working on the PoC for a new process to integrate the transfer of digital records from agencies with a digital preservation system. The groundwork of the PoC demonstrated that our combined Waterfall/Agile methodology could both deliver to the requirements set while allowing for a feedback loop during implementation stages which gives QSA the flexibility to make smaller decisions once the initial system is up and running in a test environment.

With our original consortium partners, Hudson Molonglo and Recordkeeping Innovation, in the POC, we were able to design, develop and implement an end-to-end digital transfer process. The workflow was built upon functionality delivered in the AMS and was subsequently integrated with Archivematica for long-term digital preservation. This allowed us to validate assumptions made before implementation, and identify areas requiring further investigation for a complete solution.

At Gaia Resources, we prefer to work closely and collaboratively with our clients; as well as call on the expertise of subject matter experts when required to ensure we can provide the best service and solution. In the current DPS project, we have also partnered with Artefactual. Artefactual is not only responsible for maintaining Archivematica but they also manage Access to Memory (AtoM), another open-source archival solution that Gaia Resources has delivered for clients. With this team of specialists, the project will implement digital preservation capabilities for QSA, enabling them to preserve and provide access to born-digital and digitised records of cultural and historical significance for Queensland. We are privileged to be part of this important work preserving Queensland’s heritage, delivered for QSA via DAP as part of the technology transformation program underway at QSA. It is wonderful to continue our partnership with Record Keeping Innovation and Hudson Molonglo and to welcome Artefactual to the team. 

The Queensland State Archive holds over 67 linear kilometres of records

More recently, I wrote about how lucky the Brisbane-based QSA team were in getting a tour around some of the amazing facilities there as we came closer to the next phase of our digital preservation work with QSA and DAP. Our continuing relationship with QSA themselves; our partners from the original consortium; and now Artefactual, brings together specialist knowledge in record-keeping best practice as well as the two open-source solutions underlying the future and current systems.

As our relationship with QSA continues, Gaia Resources is excited to see the outcomes of this stage of work and bring it into use for the benefit of future generations. 

If you are interested in discussing how Gaia Resources might assist your organisation in managing archives or collections solutions, please get in touch via email or our socials: Facebook, Twitter or LinkedIn.

Sophie

The post Bringing Digital Preservation to Queensland State Archives appeared first on Gaia Resources.