A bit late, but thought I’d squeeze in this blog that I had drafted for 2025 Trends. We’re nearly three months into 2025, and I can say that it’s going to be an interesting year, with AI as a focus at the business, executive and board level. Organisations can see and use AI personally, so they understand the benefits, but they don’t understand the risks, implications and investments in data foundations. Having run numerous round tables and those that I have attended across Australia and New Zealand – these are the challenges I’m seeing in both countries:
- The COVID digital boom is now being evaluated with consumer and business behaviour adjusting back to pre-covid levels
- Lack of foundation for teams and cloud is leading to risk, higher cost to serve and realise value
- Cloud costs have come to the forefront, and organisations are finding it difficult to predict costs
- Datasets are fragmented and lack quality, leading to failed projects
- Workforce readjustment as digital programs have slowed down and the high cost to serve technology can be sustained
- A focus on productivity and creating adjustment in working from home
AI sprawl without foundations creating risk
With the rise of LLMs, it’s easier to start using AI effectively. Many developers and business SMEs are leveraging LLMs for external or internal applications. However, without a background in data, privacy, governance, or risk management, this can lead to failed concepts or risky apps. Organisations need strong data governance foundations. Here are my predictions:
- Data governance framework will evolve to support AI to help steer organisations to be wiser and understand the risks involved.
- Data products, or what I see as the gold layer structure, will be important. Getting organisations to establish governance between silver and gold layer will lead to better data products, self-services and AI outcomes. This does need the business to get involved in the journey.
- I also see AI governance as a big growth area. In simple terms, data and AI governance have a large overlap, and combining them will reduce overhead and confusion.
- ISO and NIST standards are becoming easier to implement. They sound scary, but you can start the journey with a cloud.
- I see a boom in data quality, catalogue and glossary software and products. More organisations will invest, and vendors will need their solution at a faster rate.
Data Engineering and Integration are starting to converge more, but practices will be different
Being an architect, I always had to define design integration patterns and what I normally find a healthy debate with integration and data engineering. What I find is now technology is converging, so technical expertise is the same, but how you approach it is very different. I’m predicting the following:
- Change data capture at the source is the new growth area to manage data sprawl, master data, reduce duplication and processing costs.
- Being a Hadoop guy and finding some of its practices, like medallion architecture, a.k.a Bronze, Silver and Gold architecture, enables better data governance and data product. I see this to be the norm for organisations if they are still built on flat structures, then I think organisations will be left behind in the long run, especially with AI.
- I see data engineering pipelines feeding back into applications. Also known as reverse data engineering, like integration.
- I see the importance of CI/CD with practice for data pipelines to help establish production-grade solutions and remove the human bottlenecks if staff changes over.
- I see this battle of Iceberg and Parquet, Delta, ongoing, but technology experts vs. business value will drive it. I think both are great, but what is simpler to look after and easier to implement to get to value creation at speed?
AI and Software domain are starting to converge
With the rise of LLMs and how easy it is to build AI applications, I see the software and AI domains starting to converge, but I think the fundamentals of each domain needs to work together. Software developers excel at creating solid, well-structured code. AI and data science projects often struggle with production because they focus on hypothesis creation and data analysis, not on enterprise features like security and scalability. LLMs have blurred the lines, necessitating team collaboration to ensure data governance and enterprise-grade solutions.
From this I’m predicting the following:
- LLM is pushing the AI boom faster as it is usable out of the box, however it comes with risk of how you interpret and use the data
- Retrieval Augment Generation (RAG) will become more important for use cases after a proof of concept. This will help reduce cost, manage the quality of output and have better control of the application, this is a framework, not just about selecting a vector database and saying it’s RAG.
- Agentic Agents will help with automation, but adoption will be driven by vendors and similar to business integration tools like Power Automate and Zapier. Organisations will learn about the value but will need to manage the risks of the tool and the outcome.
- Machine learning / Artificial intelligence Ops (MLOps/AIOps) will be important for organisations to manage risk and production grade outcomes. This will need data and software teams to work together. This isn’t a lone soldier doing everything end-to-end, leading to knowledge risk if they leave.
- RPA, iPaaS and AI will converge, especially with the Agentic agent. This will be driven by the simplicity of the new tools.
- Vision AI will evolve faster, and it’s an inflection point for video. The challenges will be managing cost, and there will be a drive to have hybrid cloud solutions for this as networking will be an issue for real-time use cases.
Executive will focus on Foundation with Velocity and Value
For the past few years, we have seen data projects succeed and fail. This is probably a separate topic as data is a big word and can become or create distractions as it is a fast-evolving domain. What I see now is that organisations moving away from large programs of work to incremental value-driven projects with foundations laid on the way. To give you context, a squad for 4 -5 resources can cost a business over $500 – $600k per year, so returns need to align to investment. The key trends are:
- Data foundations will be so important from data security, processes, accountability and architecture.
- Data architecture will be less big bang, but foundations that you can build upon.
- Teams will be measured for value and releases with business feedback.
- Product plans will become an important artefact to understand value before build for product teams.
- Incremental value will be driven by two speed (DataOps) delivery model, with pilots at the beginning to understand value, and then productization which cost more.
- Agile and waterfall (Wagile) are going to be pushed by organisations to manage risk and velocity better.
