Usability

Structured Logging Framework

Spark 4.1.1 outputs logs in structured JSON format for easy ingestion by log aggregation tools (ELK, Splunk, Datadog).

spark.log.structuredLogging.enabled=true

spark = SparkSession.builder \ .config("spark.log.structuredLogging.enabled", "true") \ .appName("Structured Logging") \ .getOrCreate()

Error Class Framework

Standardized, actionable error messages that explain what failed, why, and how to fix it.

Before (Spark 3.x):

AnalysisException: cannot resolve 'user_id' given input columns: [id, name, value]

After (Spark 4.1.1):

[UNRESOLVED_COLUMN.WITH_SUGGESTION] Column 'user_id' does not exist. Did you mean one of the following? [id, name, value]

Example error message for boolean expression misuse:

[INVALID_BOOL_EXPRESSION] Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.

Behavior Change Process

Spark 4.1.1 follows a formal Behavior Change Process, documenting all breaking changes with opt-out flags where applicable.

Key behavior changes from Spark 3.x → 4.1.1:

Change	Spark 3.x Default	Spark 4.1.1 Default	Opt-Out Flag
ANSI SQL mode	false	`true`	spark.sql.ansi.enabled=false
Integer overflow	Silent	Exception	via ANSI flag
Null handling	Permissive	Stricter	via ANSI flag
Structured logging	Disabled	Enabled	spark.log.structuredLogging.enabled=false

Last updated on May 14, 2026

Was this page helpful?