Title
Create new category
Edit page index title
Edit category
Edit link
Apache Pinot Sizing Guide
Apache Pinot cluster hardware sizing depends on several factors, including:
Data ingestion rate (real-time or batch)
Query concurrency and latency requirements (Read and Write QPS)
Data volume (raw + segment size post ingestion)
Query complexity (filters, aggregations, joins)
Use of features like star-tree indexing, inverted indexes, etc.
Here’s a general guideline for hardware sizing across key Pinot components:
Real-time Ingestion Nodes (Server Nodes)
These handle consuming from Kafka (or another stream), indexing, and serving queries.
Parameter | Guideline |
|---|---|
CPU | 8–32 vCPUs per node (more cores for high ingest/query workloads) |
Memory (RAM) | 32–128 GB (based on segment size in memory, indexes) |
Disk (SSD recommended) | 1–2 TB NVMe SSD (ensure high IOPS; Pinot is I/O intensive) |
Network | ≥10 Gbps (especially for high ingest rate and query throughput) |
# of Nodes | Start with 3–5, scale based on data size and QPS |
Offline Ingestion/Storage Nodes
Used for querying large volumes of historical data (HDFS/S3 segments loaded onto these nodes).
Parameter | Guideline |
|---|---|
CPU | 8–16 vCPUs |
Memory | 32–64 GB |
Disk (SSD preferred) | 1–4 TB (based on segment storage needs) |
# of Nodes | 3–10+ (depends on data volume and retention) |
Broker Nodes
These handle query routing and aggregation across servers.
Parameter | Guideline |
|---|---|
CPU | 4–8 vCPUs |
Memory | 16–32 GB |
# of Nodes | 2–4 (scale based on concurrency and latency targets) |
Controller Nodes
Coordinate cluster metadata, segment assignment, and retention policies.
Parameter | Guideline |
|---|---|
CPU | 2–4 vCPUs |
Memory | 8–16 GB |
Disk | 100–200 GB |
# of Nodes | 2 (HA via active-standby setup) |
Example: Sizing for 10 TB of data with moderate real-time ingestion and ~100 QPS
Role | #Nodes | CPU | RAM | Disk (SDD) |
|---|---|---|---|---|
Server | 5 | 16 vCPUs | 64 GB | 2 TB |
Broker | 3 | 8 vCPUs | 32 GB | 500 GB |
Controller | 2 | 4 vCPUs | 16 GB | 100 GB |
Additional Notes
Pinot memory usage is influenced by segment loading and query execution.
Disk: Prefer SSDs for segment scan speed, especially for star-tree or sorted indexes.
Co-location of real-time and offline servers is possible but avoid it for production if latency is critical.
Use auto-scaling in Kubernetes or YARN environments based on CPU and memory.
For more details about the Pinot Sizing, see Capacity Planning in Apache Pinot - Part 1 | StarTree.