Usability

Structured Logging Framework

Spark 4.1.1 outputs logs in structured JSON format for easy ingestion by log aggregation tools (ELK, Splunk, Datadog).

Bash
    
xxxxxxxxxx
 
spark.log.structuredLogging.enabled=true
Copy

Bash
    
xxxxxxxxxx
 
spark = SparkSession.builder \    .config("spark.log.structuredLogging.enabled", "true") \    .appName("Structured Logging") \    .getOrCreate()
Copy

Error Class Framework

Standardized, actionable error messages that explain what failed, why, and how to fix it.

Before (Spark 3.x):

Bash
    
xxxxxxxxxx
 
AnalysisException: cannot resolve 'user_id' given input columns: [id, name, value]
Copy

After (Spark 4.1.1):

Bash
    
xxxxxxxxxx
 
[UNRESOLVED_COLUMN.WITH_SUGGESTION] Column 'user_id' does not exist.Did you mean one of the following? [id, name, value]
Copy

Example error message for boolean expression misuse:

Bash
    
xxxxxxxxxx
 
[INVALID_BOOL_EXPRESSION] Cannot convert column into bool: please use '&' for 'and','|' for 'or', '~' for 'not' when building DataFrame boolean expressions.
Copy

Behavior Change Process

Spark 4.1.1 follows a formal Behavior Change Process, documenting all breaking changes with opt-out flags where applicable.

Key behavior changes from Spark 3.x → 4.1.1:

Change	Spark 3.x Default	Spark 4.1.1 Default	Opt-Out Flag
ANSI SQL mode	false	`true`	spark.sql.ansi.enabled=false
Integer overflow	Silent	Exception	via ANSI flag
Null handling	Permissive	Stricter	via ANSI flag
Structured logging	Disabled	Enabled	spark.log.structuredLogging.enabled=false

Last updated on May 14, 2026

Was this page helpful?