Worker Tags

Worker tags allow fine-grained routing of shuffle jobs to specific worker subsets. Common use cases include:

Resource Isolation: Dedicate workers to specific tenants or applications
Hardware Segmentation: Route jobs to SSD-equipped nodes vs. HDD nodes
Environment Separation: Separate production and staging worker pools
Rolling Upgrades: Tag workers by version for controlled, safe upgrades

Enable Worker Tags

Bash
    
xxxxxxxxxx
 
celeborn.tags.enabled=true
Copy

Tag Expression Syntax

Bash
    
xxxxxxxxxx
 
# AND condition: all tags must matchenv=production,region=us-east,high-io # Advanced TagsQL mode celeborn.tags.useTagsQL=trueenv:production region:{us-east,us-west} env:!sandbox
Copy

dynamicConfig.yaml — Tag Example

Bash
    
xxxxxxxxxx
 
- level: SYSTEM   config:     celeborn.tags.preferClientTagsExpr: false     celeborn.tags.tagsExpr: 'env=production'   tags:     env=production:       - 'worker1:9094'       - 'worker2:9094'     env=staging:       - 'worker3:9094'     high-io:       - 'worker1:9094'   - tenantId: analytics   level: TENANT   config:     celeborn.tags.tagsExpr: 'env=production,high-io'
Copy

Last updated on Apr 23, 2026

Was this page helpful?