Future data trends: real-time analytics & hybrid models

Justin Borgman, Cofounder and CEO of Starburst, has outlined key trends he anticipates in data, artificial intelligence, and storage approaching 2025.

Borgman highlights the increasing priority businesses are placing on real-time analytics. "Businesses will prioritize real-time analytics, delivering insights within minutes to keep pace with intensifying customer and market demand and competition. This shift will enable faster decision-making across departments, from marketing to customer service, giving organizations a competitive edge. Real-time data will become essential for companies aiming to act on insights immediately, transforming analytics from an ad hoc, retrospective tool to a proactive business driver," he explained.

Another trend identified by Borgman is the acceleration and scaling of AI facilitated by well-defined data products. "Well-defined data products become a prerequisite for scaling AI workflows like RAG. We all know that your AI is only as good as the data you feed it, and the importance of quality and governance will become more important than ever. Furthermore, data products INCLUDE business context, which is so critical to your AI applications," Borgman stated.

Discussing data storage trends, Borgman pointed to the rise of the hybrid lakehouse model. "The resurgence of on-prem data architectures will see lakehouses expanding into hybrid environments, merging cloud and on-premises data storage seamlessly. The hybrid lakehouse model offers scalability of cloud storage and secure control of on-premises, delivering flexibility and scalability within a unified, accessible framework," Borgman commented.

Borgman also noted SQL's returning popularity in data lakes, driven by table formats like Apache Iceberg which simplify data access. "SQL is experiencing a comeback in the data lake as table formats like Apache Iceberg simplify data access, enabling SQL engines to outpace Spark. SQL's renewed popularity democratizes data across organizations, fostering data-driven decision-making and expanding data literacy across teams. SQL's accessibility will make data insights widely available, supporting data empowerment," he remarked.

Furthermore, Borgman suggests a shift in how modern data-driven SaaS applications are developed, favouring lakes over traditional databases or data warehouses.

"New data applications will be built on the lake rather than traditional databases or data warehouses," he said

"The reason is simple: SaaS companies care deeply about gross margins in the products that they offer and data lakes offer significantly better TCO and no vendor lock-in. Building an application on an object storage lake allows companies to leverage open formats like Iceberg for storage and open engines like Trino for compute. The end result is an application stack that won't break the bank and is proven to handle Internet scale," Borgman elaborated.