Introducing WAP pattern support with Apache Iceberg

Click for: original source

If you’re using SQLMesh alongside Apache Spark and Apache Iceberg, I have some exciting news for you! Starting from version 0.57.0, SQLMesh applies the Write-Audit-Publish (WAP) pattern when executing models using Apache Spark and the Apache Iceberg data format. The best part? No user action is required to enable this behavior - it’s enabled by default. By Iaroslav Zeigerman.

Further in the article:

  • What is write-audit-publish pattern
  • Why Apache Iceberg

The unique characteristics of the Apache Iceberg architecture enable SQLMesh to deliver a seamless WAP experience to its users. For example, every write into an Iceberg table creates a new snapshot, akin to a Git commit. Furthermore, Iceberg enables the creation of branches within the table and supports cherry-picking of snapshots from one branch to another as a simple metadata operation. Changes made to one branch are not visible in other branches. Good read!

[Read More]

Tags analytics big-data app-development apache devops