Cloud Deployment
The primary delivery vehicle for Acceldata Torch is Kubernetes. The following section describes the various deployment modes and deployment architecture when deploying Torch into managed Kubernetes(K8S) clusters in any cloud vendor.
In the cloud environment, Kubernetes is available as a managed service. Different cloud vendors provide Kubernetes as a built-in offering. For Example,
- AWS has Elastic Kubernetes Service (EKS) as Kubernetes Offering
- GCP has Google Kubernetes Engine (GKE) as Kubernetes Offering
- Azure has Azure Kubernetes Service (AKS) as Kubernetes Offering.
Torch can be deployed on Kubernetes Clusters managed by any of the above K8S Offerings.
Torch can be deployed in the existing managed K8S cluster and spark jobs can be executed on existing spark cluster or in Kubernetes itself. Based on the existence of a Spark cluster, this deployment mode can be classified as follows:
Cloud Kubernetes Infra + No Existing Spark Cluster
Torch business services and the data processors run on the same K8S environment. The K8S environment is managed by the user and must provide a namespace with proper resource quota in an existing cloud managed Kubernetes environment. The Spark jobs also run in the same K8S cluster.
Kubernetes from customer’s infrastructure and Spark job running on the same Kubernetes cluster
Cloud Kubernetes Infra + Existing Spark Cluster
The Torch Business services and the Data Processors run on the same K8S environment. The K8S environment is managed by the user and must provide a namespace with proper resource quota in an existing cloud managed Kubernetes environment. These resources are required for running the Torch services. Spark jobs run on Livy enabled Spark clusters managed by the user. Also, the user must provide queues with proper resource allocation and Torch submits the spark jobs only into those queues.
Kubernetes from customer’s infrastructure and Spark job running on the external Spark cluster