Delta Lake

Delta Lake is an open source project that enables building a Lakehouse architecture on top of data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing on top of existing data lakes, such as S3, ADLS, GCS, and HDFS.

Delta Lake offers the following:

  • ACID transactions in Spark: The serializable isolation levels ensure that readers never see inconsistent data.
  • Scalable Metadata Management: Leverages Spark distributed processing power to handle all the metadata for petabyte-scale tables with billions of files at ease.
  • Streaming and Batch Processing: A table in Delta Lake is a batch table as well as a streaming source and sink. Streaming data ingest, batch historic backfill, and interactive queries all just work out of the box.
  • Schema Enforcement: Automatically manages schema variations to prevent the insertion of bad records during data ingestion.
  • Time Travel: Data versioning enables rollbacks, full historical audit trails, and reproducible machine learning experiments.
  • Upserts and deletes: Supports merge, update, and delete operations to enable complex use cases like change-data-capture, slowly-changing-dimension (SCD) operations, streaming upserts, and so on.
Delta Lake VersionApache Spark Version
3.3.x3.5.x
3.2.x3.5.x
3.1.x3.5.x
3.0.x3.5.x
2.4.x3.4.x
2.3.x3.3.x
2.2.x3.3.x
2.1.x3.3.x
2.0.x3.2.x
1.2.x3.2.x
1.1.x3.2.x

Spark Shell

Initialize the Spark shell with configurations optimized for working with Delta Lake.

Bash
Copy

Import Statements

Import the necessary libraries for Delta Lake to enable data manipulation and querying.

Bash
Copy

Create Table, Insert Data, and Read Data

Example code to create a Delta Lake table, insert data into it, and perform queries to retrieve the data.

Bash
Copy

Update Data and Read Data

Show how to update records in a Delta Lake table.

Bash
Copy

Merge Data and Read Data

Demonstrate merging data from one Delta Lake table into another and reading the resultant data.

Bash
Copy

Delete Data

Example code for deleting specific records from a Delta Lake table, demonstrating data management capabilities.

Bash
Copy

Time Travel Query

Use Delta Lake's time travel feature to query data from different historical versions of a table.

Bash
Copy

For more details, see Delta Lake Documentation.

Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard
  Last updated on Aug 15, 2025
Next to read:
Installing NiFi
On This Page
Delta LakeSpark ShellImport StatementsCreate Table, Insert Data, and Read DataUpdate Data and Read DataMerge Data and Read DataDelete DataTime Travel Query