IAG's Data Mesh
Accelerating Time to
Value in the Cloud
- Decentralising data product ownership on a cloud-based data platform improves Time to Value by up to 75%.
- Migrating from an on-premises data warehouse to the cloud allowed IAG to move to their highest SLA performance tier at a fraction of the cost.
- CI/CD pipelines are essential to accelerate data product delivery times and improve data quality
- A Data Mesh built on Google Cloud streamlines scalability, reliability, security, auditability and enables AI readiness.
- Data processing performance gains of 20 times were realised by moving to a cloud-based architecture.
- The project delivered IAG’s first Data Mesh architecture currently used by over 500 engineers and data product owners.
- Audit compliance was bolstered through greater fidelity in data traceability.
The Opportunity
Over the course of nearly 100 years, Insurance Australia Group Limited (IAG) has steadily grown to become the largest general insurance company in Australia and New Zealand. With leading brands including NRMA, CGU, SGIO, NZI, State, Lumley and WFI, the company has more than 8.5 million customers and underwrites over $16 B of premiums per annum.
IAG relies on its ability to rapidly innovate and advance market offerings within a heavily regulated industry. This efficiency is necessary to maintain competitive advantage against both large players and smaller, nimble FinTech and InsurTech challengers. According to Morgan Stanley, Fintech challengers grew their market share by 2.5% in 2023.
IAG is a data-driven organisation, having built up large-scale data sets providing considerable insight into customers needs across each of their brand offerings. These data sets were historically processed and stored in an on-premises data warehouse that was managed by a centralised data team.
However, as data volumes increase within any on-premises system, so too do the levels of technical and financial risk associated with the maintenance of increased workloads. On-premises storage and compute ceilings could delay IAG’s responsiveness to operational or strategic objectives – including campaign production and delivery, payment processing, climate impact analysis, and claim response times.
Rather than being concerned by how data systems scale from a technology standpoint, IAG focused on how these systems could support their strategic objectives.
“Our strategy goals are centred around building deeper engagement with our customers and growing the number of customers in our network of brands,” said Burak Hoban, Executive Manager of Data Platforms, IAG. “We seek to create more value for our customers by making their world a safer place, and to increase the lifetime value of our customer relationships.”
To support these goals, IAG would require a data platform that was efficient, highly available, and fundamentally supported the rapid development and delivery of data products in accordance with quality and regulatory standards.
IAG recognised that moving to a decentralised architecture would be crucial in empowering the organisation’s domain teams to:
- Efficiently build solutions that deliver lifetime value for their customers.
- Leverage emerging technology trends like artificial intelligence (AI) to become more targeted and responsive to customer expectations.
“We first tried to solve this problem ourselves, but eventually we reached the limits of just how far and how much engineering we could put against running our own data platform.” – Burak Hoban
The Solution
IAG engaged Simple Machines to design and implement a scalable, performant, secure and compliant data platform that would make the development and delivery of data products more efficient, and enhance developer experience, productivity and happiness. Most importantly, the new architecture would enable IAG to leverage data more flexibly and responsively – to achieve the strategic goals outlined above by Burak Hoban.
To maximise the scalability and responsiveness of the new data architecture, Simple Machines designed a Data Mesh architecture, the first of its kind within IAG. Data Mesh is an approach to data architecture and management that is growing in popularity to address common data-related challenges in efficiency, governance and quality.
Simple Machines designed the new Data Mesh architecture to run on the Google Cloud. Known internally at IAG as the Google Data Platform (GDP), it leverages multiple Google Cloud services that provide intrinsic scalability and are not constrained by the performance ceilings that existed in the previous on-premises solution. For example, BigQuery moves the architecture from a traditional data warehouse to a massively scalable AI ready data platform with inbuilt governance.
“We first tried to solve this problem ourselves, but eventually we reached the limits of just how far and how much engineering we could put against running our own data platform,” said Dave Abrahams, Chief Technology Officer, IAG. “Performance was a key factor, and BigQuery significantly outperformed our on-prem cloud platform and the other solutions we were exploring at the same time.”
Simple Machines also worked closely with IAG to design a governance strategy that would support the decentralised goals of the Data Mesh while interfacing with the current systems and processes and data governance standards in place at IAG. This was a fundamental step in ensuring the new platform did not disrupt operational processes or compromise privacy, security or regulatory compliance.
Simple Machines supported IAG in re-engineering their development lifecycle processes. The project team adopted an Agile methodology, incorporating Continuous Integration/ Continuous Delivery (CI/CD) pipelines that enabled automated testing to accelerate the Time to Production of data products and well as improving overall data quality.
Simple Machines designed a Data Mesh architecture to help IAG deliver products that promote customer-centric design principles.
The Impact
The ultimate objective of the project was to enhance the efficiency and scalability of data products, enabling better support for the needs of IAG customers. According to Dave Abrahams the efficiency gains speak volumes: “We’ve had better than expected performance, and can now process 10 times the amount of data and volume in half the time”.
Not only was the performance efficiency of the underlying platform a major success, but the Time to Value of data products improved by up to 75%. The self-service and agile development processes introduced were key to this efficiency gain and fundamentally important to improving data quality.
It is projected that IAG will save up to $2 million in annual license costs that were previously associated with their on-premises data warehouse. Furthermore, the cost efficiencies of moving to the cloud have been multi-faceted. Moving to the Google Cloud enabled IAG to move to their highest SLA performance tier at a fraction of the cost.
These improvements extend the capabilities of IAG’s new data platform to a growing set of use cases. For example, The GDP enabled IAG’s New Zealand marketing team to significantly improve the delivery speed of marketing communications that, Abrahams said, “were just taking too long. By moving to GDP, we were able to improve the time it took for us to deliver these marketing messages from days down to hours.”
A key advantage of moving to a decentralised data architecture is the ability for domain teams to efficiently self-service the delivery of new data products or the iterative evolution of existing products. This factor is vital for IAG remain ahead in an increasingly competitive marketplace. The platform is currently used by over 400 engineers and data product owners and growing.
IAG will save up to $2 million in annual license costs that were previously associated with their on-premises data warehouse.
Lessons Learned
Small, focused, agile teams can streamline acceptance in highly controlled environments. By iterating quickly and adjusting to the best outcome, Simple Machines’ engineering team was able to ensure the project continued efficiently by building trust through proving value early and continuously.
Get on the front foot for organisational change. The new Data Mesh architecture represented a different way of working for IAG, therefore using existing tools and process where they match the new architecture was immensely valuable. Staying ahead of the game with communication and training meant that teams were ready for change.
Data decentralisation and governance must align with the organisational objectives. With a decentralisation of data comes an associated change to data governance and security. The platform design and process changes had to tread a clear and low friction path to balance these two factors without disrupting existing operational processes and systems.
Self-service for fast-tracked data product delivery. Moving the data product ownership to domain teams and away from a centralised model drives innovation. Efficiencies are seen in both Time-to-Value and cost reduction.
Data products promote customer-centric design principles. Having self-service within domain teams fosters a product mindset, which in turn enables consumer centric design principles.
