Secure and cost-effective archiving of court judgments with serverless technology
The Data Transformation and Pipeline Service connects multiple systems of The National Archives together; ingesting, enriching and sharing court judgments with other systems, including the new Find Case Law service.
The client
The National Archives
Zaizi’s role
- Agile
- Engineering
- Architecture
- Process automation
- Cloud services
The client says…
“Zaizi have been very professional in their work, bringing a wealth of knowledge to this project. They worked effectively to quickly grasp our requirements and build an MVP. They then worked with us to further enhance the product to be adaptable to our future transformation needs.”
Paul Young, Digital Archiving Specialist/Researcher, TNA
The brief
The National Archives is the official archive and publisher of the UK Government. As part of its role preserving and providing access to the Public Record, it is responsible for creating a publicly accessible database of court judgments.
Zaizi was contracted to create a data transformation pipeline that verifies and processes uploaded court documents, enriches them with additional metadata and securely supplies them to a new service, Find Case Law.
This service publishes judgments to the web, making court and tribunal decisions searchable and freely available for download.
The National Archives requested an MVP solution within three months.
The delivery
Our proposal was to build a sustainable service connecting the Transfer Digital Records service to Find Case Law. We proposed a service that would:
- minimise operational overheads
- use services that required minimal patching or upgrades
- utilise serverless capabilities to save on cost and reduce environmental impact
- use technology that the market can provide skills for
Using AWS Step Functions, we designed, built, tested and documented the first iteration of the ‘Transformation Engine’ – a data-transformation and pipeline service. This service securely connects the Transfer Digital Records system with the Find Case Law system, also implementing a word to xml parser separately developed by The National Archives. This setup ensures data validation, parsing, and transformation are effectively orchestrated.
The MVP was successfully launched on schedule in April 2022. The data transformation pipeline supplied structured metadata extracted by the The National Archive’s judgment parser to the Find Case Law system, allowing users to search by:
- neutral citation
- keyword
- date
- specific court or tribunal
- judge name
- party name – for example, by claimant, defendant or other party
Following this, The National Archives extended our contract twice for two subsequent phases of work.
- The announcement phase – focusing on lower–priority tickets, including notifications and non-functional requirements such as data retention, performance and security.
- A second version of the Data Transformation and Pipeline service, bringing in further improvements. This second version:
- was based on event driven architecture to increase resilience
- was restructured to be more extensible for future data transformation needs
- saw the Continuous Delivery and Integration capability migrated to Github Actions so it aligned with the customer’s technology standards
The outcome
After our work, The National Archives can:
- securely supply court documents, with searchable metadata, to a new service
- reduce operational overheads and operating costs due to the serverless solution
- support and maintain the service after knowledge sharing from Zaizi
Since April, nearly 5,000 judgments have been processed through the data transformation pipeline we built.
Challenges
A quick delivery
The National Archives wanted the first iteration of the Data Transformation and Pipeline service ready within three months. We worked closely with their product manager to understand requirements, using Agile techniques such as sprints, story points and monitoring velocity. This helped us focus on the critical, high-priority work and deliver an MVP service on time.
The approach identified lower-priority stories and enabled the customer to visualise how we could improve the service through subsequent phases of work, as well as associated costs and timeframes.
Building experience client-side
As well as providing a solution under a tight timeframe, we left the customer with a service that they could easily operate and update. Alongside the build, we worked to transfer knowledge to the customer’s in-house team, so they had the internal capability to support and maintain the service.
Planning for efficiency
During our technical scoping, we learned that judgements only enter the system on weekdays between 9am and 6pm. This information led us to implement a serverless solution that only runs on demand, reducing operational overheads and making the service cheaper and easier to support for the customer — as the customer only pays for what they consume in the cloud.
If you’d like to find out more about our work, please get in touch.
-
Creating safe, secure and frictionless digital borders — Border Force
-
Helping local councils revamp taxi licensing to boost efficiencies and reduce costs
-
Wrap-up: Our impact on digital government services in 2023
-
From a stalled legacy system project to award-winning success
-
Working with users to improve support services for leaseholders and park home owners
-
Exploring the case for a national landlord register
