HudHud Tech — Tech & AI news platform
Monday, 23 November 2026
Breaking
Software & Apps

Apache Iceberg V4 Introduced at QCon San Francisco 2026

20 November 2026 3 min read هدهد تك | Hudhud Tech

Apache Iceberg V4 was introduced as a significant redesign of table metadata aimed at enhancing scalability and accommodating more frequent writes. This new version also introduces advanced capabilities for data management and schema evolution. Daniel Weeks, co-creator of Apache Iceberg and a member of the technical staff at Databricks, presented these updates during a session at QCon San Francisco.

The new version addresses the challenges associated with managing very large tables and expands the format to support new data types and workloads. Weeks described Iceberg V4 as a major reimagining of the table format rather than a mere incremental update. Key changes include a new metadata tree, more efficient commit processes, typed column statistics, relative file paths, and support for file and vector data types.

Weeks emphasized that while previous versions maintained much of the metadata structure, V4 tackles issues that arise at larger scales and introduces support for new use cases. He stated, “V4 is a major reimagining of a table format. We didn’t get everything right in V3 and prior, so now we’re taking the opportunity to fix some of the biggest things that are challenges as we scale up to even larger data sets and we bring on new use cases.”

Apache Iceberg is an open table format designed for managing large analytical datasets across various data processing engines. Initially developed at Netflix in 2017, it was open-sourced in 2018 and became an Apache top-level project in 2020. Competing open table formats include Delta Lake, created by Databricks and now a Linux Foundation project, Apache Hudi, originally developed at Uber, and Apache Paimon, which emerged from the Flink community.

In his presentation, Weeks highlighted three main areas of change in Iceberg V4: metadata and scalability, new table features, and new data types. The metadata enhancements include the Adaptive Metadata Tree, single-file commits, improved column statistics, relative paths, and column-level updates. New table features consist of expression-based default values and constraints, while new data types, particularly file and vector types, cater to workloads such as AI and search.

Weeks explained that the improved commit process allows for much faster data streaming. He noted, “This gets us into use cases where we can stream data much faster. We can do commits at the 2nd level or subsecond level, where right now we largely recommend that you don’t do commits under 2 minutes.”

In Iceberg V3, adding a data file required at least four I/O operations, including manifest creation and rewriting. V4 simplifies this by allowing the metadata tree’s root to point directly to data files, reducing the operation to a single metadata write. Weeks indicated that this change could enable commits every second or even at subsecond intervals, a significant improvement over the current recommendation of at least two minutes between commits.

Iceberg V3, introduced in 2024, added features such as deletion vectors, row lineage, new data types, and encryption.

 

Source: InfoQ

What do you think of this story?

Share this story: