, ,

What Are Deletion Vectors (DLV)?

Deletion Vectors are a soft‑delete mechanism in Delta Lake that enables Merge‑on‑Read (MoR) behavior, letting update/delete/merge operations mark row positions as removed without rewriting the underlying Parquet files. This contrasts with the older Copy‑on‑Write (CoW) model, where even a single deleted record triggers rewriting of entire files YouTube+8docs.delta.io+8Medium+8.

Supported since Delta Lake 2.3 (read-only), full deletion vector support for DELETE/UPDATE/MERGE appeared in later versions: DELETE in 2.4, UPDATE/MERGE in Delta 3.x Miles Cole+4docs.delta.io+4delta.io+4.

✅ Why Use Deletion Vectors?

  • Faster small changes: Only binary bitmap metadata is written, rather than rewriting large Parquet files.

  • Write efficiency: Particularly efficient when changes affect sparse rows across many files Medium+11delta.io+11Towards AI+11.

  • ACID semantics preserved: Readers still get the correct view by merging with DLV metadata at read time.

However:

  • Read-time overhead: Filtering DLV metadata adds overhead during queries.

  • Maintenance needed: Unapplied deletion markers build up until compaction or purge Medium+6japila-books+6Towards AI+6.

⚙️ How They Work Under the Hood

Enabling

sql
ALTER TABLE my_table
SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);

On Databricks, settings can also auto-enable deletion vectors for new tables YouTube+8Databricks Documentation+8Data in Action+8.

Write-Time

When you delete rows (e.g. DELETE WHERE), Delta writes a deletion vector file or log entry—a bitmap of row positions using RoaringBitmap—without touching Parquet data files Databricks Documentation+8delta.io+8Medium+8.

Read-Time

Delta reads the base Parquet + DLV metadata, applying filters to hide soft-deleted rows during query execution.

Physical Cleanup

To remove soft-deleted rows physically:

  • Run OPTIMIZE to compact affected Parquet files.

  • Or use REORG TABLE ... APPLY (PURGE) to force rewriting files with DLVs applied Medium+8docs.delta.io+8Towards AI+8.

  • Then VACUUM can remove old files beyond retention threshold.

📊 Copy-on-Write vs Merge-on-Read with Deletion Vectors

Feature Copy‑on‑Write Merge‑on‑Read (Deletion Vectors)
Write/Delete cost Rewrites entire files Writes small metadata only
Read-time cost Minimal Additional filtering overhead
Best for frequent updates ❌ ✅
Best for read-heavy workloads ✅ ⚠️ (needs compaction)
Compaction required Less often Yes (OPTIMIZE/REORG)

That heatmap in the image above illustrates how deletion vectors provide faster DELETE performance as file count and changes grow Microsoft Learn+2docs.delta.io+2Databricks Documentation+2Data in ActionMedium+6delta.io+6delta.io+6Towards AI+4delta.io+4Medium+4.

💡 Best Practices for Data Engineers

  1. Enable on high-write tables (Bronze/Silver layers) where DML is frequent Data in Action+7Reddit+7delta.io+7.

  2. Plan regular maintenance: schedule OPTIMIZE, REORG TABLE APPLY PURGE, and VACUUM.

  3. Monitor read latency: as DLV files accumulate, query performance may degrade.

  4. Be cautious with compatibility: older clients or Delta sharing may not support DLV-enabled tables Medium+5Databricks Documentation+5Microsoft Learn+5MediumDatabricks Documentation.

  5. Enable workspace auto‑enable options on Databricks to enforce default usage for new tables Databricks Documentation+2Databricks Documentation+2Microsoft Learn+2.

  6. Consider using Photon + Predictive I/O for further acceleration of deletion/update workloads in Databricks Runtime 14.x+ YouTube+7Databricks Documentation+7Microsoft Learn+7.

🔧 Sample Code

sql
-- Enable deletion vectors
ALTER TABLE events
SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
-- Delete rows
DELETE FROM events WHERE date < '2025‑01‑01';
-- Compact files
OPTIMIZE events;
VACUUM events RETAIN 168 HOURS;
-- Or force purge:
REORG TABLE events APPLY (PURGE);
VACUUM events RETAIN 168 HOURS;

🧾 Summary

Deletion Vectors mark a big leap forward in efficient data mutation workflows within Delta Lake. They give you low-latency deletes and updates without immediate file rewrites, ideal for high-change tables—but they need thoughtful maintenance to keep query performance sharp. Choose wisely: use them where writes dominate and compaction is part of your data ops routine.