Title
Create new category
Edit page index title
Edit category
Edit link
Prerequisites Capacity Planning
Capacity Planning
Control Plane
A minimum of 3 nodes is required for Kubernetes control-plane functionality.
Installed Software Scope
The following applications drive capacity requirements:
Apache Spark - batch and streaming compute
Trino - distributed SQL query engine
JupyterHub - interactive notebook sessions
Concurrency and Data Volume Targets
Collect the following metrics for your environment before sizing nodes:
Metric | What to Measure |
|---|---|
Concurrent Spark jobs | Peak and average running jobs |
Concurrent Trino queries | Maximum sustained query load |
Active JupyterHub sessions | Simultaneous notebook users |
Data scan volume | Terabytes scanned per hour/day |
Processing throughput | Required gigabytes per second |
Memory vs Disk Spill
In-Memory Processing
Application | Recommendation |
|---|---|
Spark | Allocate 60--70% of executor memory for direct processing |
Trino | Size memory pools to match anticipated query complexity |
JupyterHub | Set per-user memory limits (typically 2 GB -- 16 GB) |
Disk Spill Configuration
Use Local SSD storage for shuffle spill operations.
NVMe SSDs with >= 50,000 IOPS are recommended.
Provision 2--3x the expected maximum spill volume per node.
Use multiple mount points in a JBOD configuration.
Storage Infrastructure
Local Disk
Requirement | Specification |
|---|---|
Type | NVMe SSD (mandatory for shuffle operations) |
Mount protocol | JBOD -- |
Persistent Storage
Use existing object storage (S3, GCS, Azure Blob) as the data lake repository. This is also required for hosting Spark History Server event logs.
Scaling Strategies
Dynamic Scaling
Kubernetes Cluster Autoscaler for automated node-level adjustments.
Spark Dynamic Allocation for adaptive executor provisioning.
Static Scaling
Fixed node count for environments with predictable workloads.
Lower operational complexity and more accurate cost forecasting.
Network Bandwidth
Path | Minimum | Recommended |
|---|---|---|
Inter-node communication | 25 Gbps | 100 Gbps |
Storage network | Dedicated high-bandwidth link to object storage | -- |
Consider segregating data-plane and control-plane traffic onto separate networks.
For additional help, contact our Support Team!
©2026, Acceldata Inc — All Rights Reserved.