Feature Summary

Spark Connect

Connectivity

All

spark.connect.enabled

Spark ML on Connect GA

Connectivity

Python

—

Real-Time Mode (RTM)

Streaming

Scala

.trigger(continuous=...)

Stateful Processing V2

Streaming

Python/Scala

—

ANSI Mode

SQL

SQL

spark.sql.ansi.enabled

SQL Scripting GA

SQL

SQL

Enabled by default

VARIANT GA + Shredding

SQL

SQL/Python

—

Recursive CTE

SQL

SQL

—

KLL + Theta Sketches

SQL

SQL

—

Collation Support

SQL

SQL

—

Arrow UDFs (@arrow_udf)

PySpark

Python

—

Arrow UDTFs (@arrow_udtf)

PySpark

Python

—

Python Worker Logging

PySpark

Python

spark.sql.pyspark.worker.logging.enabled

Filter Pushdown (Data Source)

PySpark

Python

—

Python UDTFs

PySpark

Python

—

pandas 2.x Support

PySpark

Python

spark.sql.execution.arrow.pyspark.enabled

UDF Profiler

PySpark

Python

spark.python.profile

Python Data Source APIs

Data Sources

Python

—

XML Connector

Data Sources

Python/Scala

—

DSV2 Extension

Data Sources

Python/Scala

—

Delta Lake 4.0

Data Sources

SQL/Python/Scala

—

SQL UDFs

Functions

SQL/Python

—

Python UDFs

Functions

Python

—

Structured Logging

Usability

All

spark.log.structuredLogging.enabled

Error Class Framework

Usability

All

—

Behavior Change Process

Usability

All

—