Apache Spark 4.1.1

Apache Spark 4.1.1 is a major milestone in the Spark 4.x series, delivering a comprehensive set of improvements across streaming, SQL, Python, connectivity, and usability.

This page consolidates all key features with working code examples and configuration references.

Area	Features
Connectivity	Spark Connect, Spark ML on Connect GA
Structured Streaming	Real-Time Mode, Arbitrary Stateful Processing V2
SQL	ANSI Mode, SQL Scripting GA, VARIANT GA + Shredding, Recursive CTE, Approximate Sketches, Collation
PySpark	Arrow-native UDFs/UDTFs, Python Worker Logging, Python Data Source with Filter Pushdown, pandas 2.x
Data Sources	Python Data Source APIs, XML Connector, Data Source V2 (DSV2), Delta Lake 4.0
Custom Functions	SQL UDFs, Python UDTFs, Arrow UDFs, Python UDFs, UDF Profiler
Usability	Structured Logging, Error Class Framework, Behavior Change Process

For details, access the following pages:

Connectivity
Structured Streaming
Feature Summary
Spark Reference
SQL Features
PySpark Improvements
Data Sources and Extensions
Custom Functions
Usability

Last updated on May 14, 2026

Was this page helpful?