We have been working for quite some time on the Western Australian New Music Archive (WANMA), and recently we finished up our development efforts and have moved into the support phase of the project – so we thought it was time for a blog about it (and we have a project recap added to our portfolio, as well).

A screenshot from the WANMA Portal

The WANMA Portal is based around the digital media records of Western Australia’s New Music history, from the 1980s (and earlier) to the present day.  This includes video and audio recordings of performances and other events, as well as photographs and documents such as brochures and posters, giving a rich presentation of the state’s history of producing music within this unique genre – which is best explained in the words of the WANMA team:

New music is often thought of as an extension of the western art music tradition that developed throughout the 20th century, through an expansion of the compositional and performative possibilities of music. New music practice includes contributions from those working in other genres such as jazz, rock, electronic music and various global musical traditions. In drawing from this cross-disciplinary aesthetic, new music attempts to push the boundaries of what music is and can be.

When we first met the team behind WANMA (including Cat Hope from Edith Cowan University and Tos Mahoney from TURA) to discuss this project, we were thinking that it might be a similar project to our work with the State Records Office of Western Australia.  However, the nature of the project, especially the open contribution model, meant that the open source packages we worked with didn’t really fit the bill.  So instead, we looked at a bespoke solution that would enable them to develop the project in the direction that they wanted to (and within the Agile framework, providing additional flexibility).  This initial review of the requirements and available solutions was a very important part of the project, and it’s one area we don’t take lightly when looking at solutions for our clients.

As well as the digital media, there are a range of articles about the people, ensembles, organisations, locations and events that are involved in the local New Music scene.  This metadata includes biographies and histories, years active, recorded works, and so on, and there is a rich network of correlations between all of these different entities. This means that you can easily see more information about, say, members of an ensemble, or the location where a particular video was recorded.

The collation, curation and archival storage of the media is being done through collaboration with the State Library of Western Australia.  This process involves the Library team digitising the original recordings, and then cataloguing the items appropriately according to their data standards.  Additional data is taken from ABC Classic FM in the form of podcast recordings, and the WANMA Portal then harvests these into the site.

As I mentioned, the site is open to public contributions.  It is simple to create an account and then make changes via the editor built right into the page, which then gets edited and approved by site administrators before it is publicly visible.  This helps maintain the consistency of the site, as well as the usual prevention of spam or vandalism.

I’ll hand over to Ben to provide more of the technical overview!

<technical stuff>

Technically, the WANMA Portal consists of four primary components: a storage engine, a web Application Programming Interface (API), data importers, and a Single Page Application (SPA), which is the part you will see in the web front end.  Here’s a technical architecture diagram:

The technical architecture behind WANMA (click to enlarge)

The storage engine is MongoDB, which is a popular, flexible-schema, document database.  MongoDB is suitable for very large data sets, many orders of magnitude greater than that required for WANMA, however it also allows for flexibility during development and for future enhancements, without the additional overhead of managing the fixed schema inherent in a traditional, typically SQL-based, database platform.  We’ve done a few projects using MongoDB now.

The API is built in NodeJS using Loopback, which is a powerful, opinionated framework based on the nearly-ubiquitous express middleware engine and the swagger specification.  The API provides data in JSON or XML format from the database about entities and the correlations between them, using common and consistent URL constructs.  The API also handles authentication and access control.  Importantly, the API is publicly accessible, which means that other services can retrieve the data programmatically, making the data stored in WANMA much more openly accessible.

The bespoke data importers retrieve data in various formats and map them into records in the WANMA database.  This component is critical for the operation of the Portal, as without it, there would be no media to watch and no articles to read.  Specifically, the data importers convert from the MARC format used by the State Library, and from several spreadsheets, into the WANMA data model.  This has been designed to allow import of data from any number of sources with additional work to provide the mappings between the source data and WANMA’s data model.

The user interface is an SPA, which means there is a single web page that hosts a complex, JavaScript application.  When the user clicks on an internal link, rather than a whole new page being requested from the server, the application changes the URL in the browser’s address bar and then modifies the UI dynamically; all that is loaded from the API is the raw data.  For the user, this results in a smoother user experience, more like an application than a website.  In this case, the application is written using Durandal, which is an SPA framework based on knockout.js.

The WANMA Portal is hosted on Amazon Web Services and utilises their commonly known products EC2 (which are virtual servers) and S3 (which is the storage), as well as their Content Delivery Network (CDN) known as CloudFront.  This means that whenever a user requests to view media, such as video or audio, it is coming from a server close to their location, anywhere in the world.  This is especially important for streaming media because the latency and potential for delays during the round-trip from, say Sweden, to Sydney where the primary data centre is located, would be problematic.  CDNs are widely used by major platforms such as YouTube and Facebook, for similar reasons.

</technical stuff> 

It might seem a bit of a mismatch from our usual work in the “environment” space, and to some degree it is – however, this is where our work in the Collections space is taking us.  We have been involved in Collections for quite some time, starting from our work with the Western Australian Museum.

Our approach for the Collections stream is to investigate how other related industries can benefit from our experience, and vice versa.  As a result of our work with the WA Museum, we are able to assist groups like the Royal WA Historical Society (see our launch blog last week for more on that one).  Similarly, our work with collections has given us exposure to the Archives area, and the State Records Office of Western Australia project has helped to develop our skills in digital archive management, which is quite important in the Collections area (such as managing digital images and genetic information for a collection).

This branching out is an important part of evolving our business and to learn from others; this work means our team are now contributors to several open source packages such as Archivematica, AtoM  and CollectiveAccess, and these systems are a big part of our future, across all of the Collections areas we work in.  We’re developing new partnerships and services around these systems and will be talking more about that in the future.

Anyway, back to WANMA.  The WANMA Portal is up and running in production and you can contribute, browse the archives of performances, and find out more about the people involved, all through the Portal.  It means that the project is really living up to the aims – “finding, accessing, remembering and performing”.

Piers (and Ben!)

P.S. Feel free to leave a comment below, drop us an email, or start a conversation with us via Facebook, Twitter or LinkedIn.