If you’re using SQLMesh alongside Apache Spark and Apache Iceberg, I have some exciting news for you! Starting from version 0.57.0, SQLMesh applies the Write-Audit-Publish (WAP) pattern when executing models using Apache Spark and the Apache Iceberg data format. The best part? No user action is required to enable this behavior - it’s enabled by default. By Iaroslav Zeigerman.
Further in the article:
- What is write-audit-publish pattern
- Why Apache Iceberg
The unique characteristics of the Apache Iceberg architecture enable SQLMesh to deliver a seamless WAP experience to its users. For example, every write into an Iceberg table creates a new snapshot, akin to a Git commit. Furthermore, Iceberg enables the creation of branches within the table and supports cherry-picking of snapshots from one branch to another as a simple metadata operation. Changes made to one branch are not visible in other branches. Good read!
[Read More]