Install Ozone 2

This documentation covers Ozone2 setup with ODP, managed by ambari as an mpack. The document also provides ozone configuration and functionality details to work with different hadoop-related services, s3, with or without security enabled in form of authentication (Kerberos) and/or authorization (Ranger).

Ozone2 Storage Elements

Ozone, being an object storage for Hadoop, consists of volumes, buckets, and keys:

• Volume: Volumes are similar to user accounts. Only administrators can create or delete volumes. Ozone creates s3v volume during initialization to support S3 Protocol.
• Bucket: Buckets are similar to directories and reside in volume. It resembles Amazon S3 Buckets in terms of hierarchy and ACL. A bucket can contain any number of keys, but buckets cannot contain other buckets. OzoneManager supports two metadata bucket layout formats - Object Store (OBS) and File System Optimized (FSO). By default, Ozone sets all buckets layout to FILE_SYSTEM_OPTIMIZED but buckets under s3v volume are ofOBJECT_STORE layout.
• Key: Keys are similar to files.

Environment Details

Ozone2 env directories

• PID - /var/run/ozone2
• log - /var/log/ozone2
• conf - /etc/ozone2/conf
• om conf - /etc/ozone2/conf/ozone.om
• lib - /usr/odp/{stack_version}/ozone2/share/ozone/lib
• ozone home - /usr/odp/{stack_version}/ozone2

Configurations

Metadata directories

Ozone2 Manager (OM)

Storage Container Manager (SCM)

Ozone2 Datanode (DN)

Recon Service

Ozone2 Service Endpoints

Ozone2 Manager (OM)

Storage Container Manager (SCM)

Ozone2 Datanode (DN)

S3 Gateaway (S3G)

Recon Service

Ozone2 Installation

Ozone2 integration with ODP is available as Ambari Mpack. Download the ozone2-mpack tar on your ambari-server node and install mpack.

xxxxxxxxxx

# Install mpack

ambari-server install-mpack --mpack=ambari-mpacks-ozone2.tar.gz --verbose

# Restart ambari

ambari-server restart

Navigate to the Ambari UI.

• Ambari UI > Services > Add Service

• Service Wizard > select Ozone2

• Click Next and configure Ozone2 component nodes and properties as per use case (Choose 3 nodes for OM, SCM and Datanode to maintain HA).

For a secure cluster with Kerberos, ozone2 enables Kerberos authentication by default at installation time.

To enable SSL on Ozone2, configure properties as shown Ozone2 Installation.

• Service installed successfully. Click Ok.

SSL Enablement

Update the following properties as per your SSL configurations for respective host and component.

Ambari UI > Ozone2 > Configurations > Advanced ozone2-env: Check the following properties

To ozone2-site, add below configs:

Then, configure trustore and keystore in Advanced ozone2-ssl-client, Advanced ssl-client-datanode, Advanced ssl-client-om, Advanced ssl-client-recon, Advanced ssl-client-s3g, Advanced ssl-client-scm, Advanced ssl-server-datanode, Advanced ssl-server-om, Advanced ssl-server-recon, Advanced ssl-server-s3g, Advanced ssl-server-scm.

Kerberos Configuration

Ozone2 service principal and keytab for service and spengo for UI, will be configured with Ambari automation. If you have an SPNEGO enabled Ozone2 cluster and want to disable it for all Ozone2 components, update the following properties as shown :

This mpack supports Ozone2 with kerberos security only on fresh installation of ozone2 in a kerberized ODP cluster, considering development limitations.

Ranger Configuration

Enable/disable Ranger authorization from Ambari UI > Ranger > Configs > Ozone Ranger Plugin and restart service to implement changes.

Before enabling plugin, verify/add ozone-filesystem-hadoop3-2.1.0.3.3.6.2-104.jar to given path

cp /usr/odp/current/ozone2-client/share/ozone/lib/ozone-filesystem-hadoop3-1.4.0.* /usr/odp/{stack_version}/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/ozone/

cp /usr/odp/current/ozone2-client/share/ozone/lib/bcprov-jdk15on-1.67.jar /usr/odp/(stack_version}/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/ozone/

Configure a Resource-based Service: Ozone

How to add Ozone2 service.

Procedure

1. On the Service Manager page, click the Add icon () next to Ozone.

The Create Service page appears.

2. Enter the following information on the Create Service page:

Table 3: Service Details

Table 4: Configuration Properties

3. Click Test Connection.
4. Click Add.

Configure a Resource-based Policy: Ozone

How to add a new policy to an existing Ozone service.

About this task

Through configuration, Apache Ranger enables both Ranger policies and Ozone permissions to be checked for a user request. When the Ozone Manager receives a user request, the Ranger plugin checks for policies set through the Ranger Service Manager. If there are no policies, the Ranger plugin checks for permissions set in Ozone, as per Ozone ACL.

We recommend that permissions be created at the Ranger Service Manager, and to have restrictive permissions at the Ozone level.

Procedure

1. On the Service Manager page, select an existing Ozone service. The List of Policies page appears.
2. Click Add New Policy. The Create Policy page appears.

3. Complete the Create Policy page as follows:

Table 25: Policy Details

Table 26: Allow Conditions

4. You can use the Plus (+) symbol to add additional conditions. Conditions are evaluated in the order listed in the policy. The condition at the top of the list is applied first, then the second, then the third, and so on.
5. Click Add.

The Ranger permissions corresponding to the Ozone2 operations are as follows:

Ozone2 Command Line Interface

This storage Ozone2 shell is the primary interface to interact with Ozone2 from the command line.

Volume operations

xxxxxxxxxx

# CREATE VOLUME

# authenticate user with kinit, in case of kerberized cluster

$ ozone2 --config /etc/ozone2/conf/ozone.om sh volume create testvol

OR

$ ozone2 --config /etc/ozone2/conf/ozone.om sh volume create o3://${om_service_id}/testvol

# VOLUME INFO

$ ozone2 --config /etc/ozone2/conf/ozone.om sh volume info testvol

24/05/02 07:41:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

24/05/02 07:41:31 INFO client.ClientTrustManager: Loading certificates for client.

{

  "metadata" : { },

  "name" : "testvol",

  "admin" : "ambari-qa",

  "owner" : "ambari-qa",

  "quotaInBytes" : -1,

  "quotaInNamespace" : -1,

  "usedNamespace" : 0,

  "creationTime" : "2024-05-02T05:10:09.564Z",

  "modificationTime" : "2024-05-02T05:10:09.564Z",

  "acls" : [ {

    "type" : "USER",

    "name" : "ambari-qa",

    "aclScope" : "ACCESS",

    "aclList" : [ "ALL" ]

  }, {

    "type" : "GROUP",

    "name" : "ambari-qa",

    "aclScope" : "ACCESS",

    "aclList" : [ "ALL" ]

  }, {

    "type" : "GROUP",

    "name" : "hadoop",

    "aclScope" : "ACCESS",

    "aclList" : [ "ALL" ]

  } ],

  "refCount" : 0

}

xxxxxxxxxx

# LIST VOLUME

$ ozone2 --config /etc/ozone2/conf/ozone.om sh volume list --all

# DELETE VOLUME

$ ozone2 --config /etc/ozone2/conf/ozone.om sh volume delete testvol

24/05/02 07:44:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

24/05/02 07:44:15 INFO client.ClientTrustManager: Loading certificates for client.

Volume testvol is deleted

If the volume contains any buckets or keys, we can delete the volume recursively. This will delete all keys and buckets within the volume, and then delete the volume itself. After running this command there is no way to recover deleted contents.

xxxxxxxxxx

$ ozone2 sh volume delete -r testvol

This command will delete volume recursively.

There is no recovery option after using this command, and no trash for FSO buckets.

Delay is expected running this command.

Enter 'yes' to proceed': yes

Volume testvol is deleted

Bucket operation

xxxxxxxxxx

# CREATE BUCKET

# authenticate user with kinit before accessing below commands, in case of kerberized cluster

$ ozone2 --config /etc/ozone2/conf/ozone.om sh bucket create /testvol/bucket

OR

$ ozone2 --config /etc/ozone2/conf/ozone.om sh bucket create o3://${om_service_id}/testvol/bucket

By default, all buckets are created to support file system (FILE_SYSTEM_OPTIMIZED).

If the bucket contains any keys, we can delete the bucket recursively. This will delete all the keys within the bucket, and then the bucket itself. After running this command there is no way to recover deleted contents.

Key operation

If the key is in an FSO bucket it will be moved to the trash when deleted. Location of trash folder /<volume>/<bucket>/.Trash/<user>. If the key is in an OBS bucket it will be permanently deleted.

Working with Ozone File System

Prerequisites

To enable ofs support with applications, configure applications to use necessary jars and ozone2-site.xml.

• Add the ozone-filesystem-hadoop3.jar to the application classpath.

• Add the following configs to core-site.xml

• Add following configs from ozone2-site.xml to hdfs-site.xml on Ambari.

• Restart hadoop services

HDFS with OFS

Access hdfs dfs operations with ozone2 storage :

Here are some examples :

• List files

• Create directory

• Upload file

• Reading file

YARN with Ozone2

Yarn can be used to run jobs with jobs accessing data from or writing into ozone file system.

• Add ozone-filesystem-hadoop3-1.4.0_.jar to _mapreduce.application.classpath* in mapred-site.xml .
• If ranger authorization is enabled, provide necessary permissions to ofs buckets, hdfs path, yarn queues, to perform necessary operations as per job requirements.
• Perform respective user kerberos authentication in case of secure cluster.
• submit job

Here is a sample job, doing wordcount on data from file in ofs, and storing the output file with wordcount result in ofs.

Configure mapreduce job to use ozone-site.xml. Alternatively, you can pass configs during runtime:

HIVE with Ozone

Although HIVE installation and operations use HDFS as default file system, ozone can be configured to be parallel file system for HIVE operations.

Configure Hive to work with Ozone :

• Navigate to the Ambari UI > Hive > Configs > Advanced Hive-env and add

• Restart Hive and Tez.
• If Ranger authorization is enabled, grant the necessary permissions to OFS (Ozone File System) buckets, HDFS paths, and Hive URL to allow the required operations as per the job requirements.
• Authenticate users with Kerberos credentials when operating in a secure cluster.

If queries are failing with below error when run queries as end user is enabled in hive org.apache.hadoop.security.authorize.AuthorizationException: User: hive is not allowed to impersonate ... https://issues.apache.org/jira/browse/HDDS-664

Ambari UI > Ozone > Configurations > Custom Core-site: add the following configs and restart services :

Store tables in OFS

To create tables in OFS add LOCATION '<OFS_URI> to CREATE TABLE command. This will make Hive tables reside at the specified location in ozone. All data changes her after will be in effect at table in given OFS_URI.

Here are sample hive operations with Hive accessing OFS :

• Connect to Beeline
• Create new table in ofs

• Validate new table

• Add values to table

• Validate newly added values

SPARK with Ozone2

Apache Spark can access data from Apache Ozone2 and perform tasks. To access Apache Ozone2, configure spark :

• Configure spark shell to use /usr/odp/current/ozone2-client/share/ozone/lib/ozone-filesystem-hadoop3-client-2.1.0.3.3.6.2-104.jar.

Accessing Apache Ozone data in Apache Spark3

• Creating sample data to be read by Spark3

• Upload the employee.csv file to Ozone2

• Provide necessary permissions under ozone policies, for spark user to access respective bucket and file, if ranger authorization is enabled.
• Allow spark user to submit yarn applications.
• Launch spark-shell

• Accessing csv file content in ozone as spark df

Custom PySpark Job

To run spark job using ofs use following command:

For a secure cluster, add --keytab <keytab> --principal <principal> values to above command.

Here is a sample custom job that functions to access Ozone data with Apache Spark and write output to Ozone.

• Custom Pyspark application using ofs to access data and write output

• Uploading sample input file to ofs

• Provide necessary permissions to spark user to access respective bucket and key, in case of ranger authorization enabled.
• Running PySpark app in secure cluster

• Validate output in ofs

Using Ozone2 S3 Gateway

Ozone provides S3 compatible REST interface through Ozone2 S3 Gateway to use the object store data with any S3 compatible tools. Although Ozone2 S3 Gateway is an additional to the regular Ozone2 components, in Acceldata’s ODP mpack Ozone2 S3 Gateway is installed and started as part of Ozone2 service. S3 buckets are stored under the /s3v volume.

Prerequisites

To use an S3 endpoint, configuring access key and secret for aws compatible tools is required. Here, taking example of awscli.

• Generate Access Key and Secret for aws : If security is not enabled, you can use any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. If security is enabled, you can get the key and the secret with the ozone2 s3 getsecret command (Kerberos based authentication is required)

• Export these credentials on your S3 endpoint. Here, I’m updating the credentials as a new profile.

Alternatively, you may create a new profile with ozone related credentials and use ozone profile to run

S3 utility tasks on awscli.

• Verify your S3 endpoint from S3 Gateway UI.

Ozone2 S3 Gateway to work with AWS CLI

Ozone S3 Gateway supports various bucket and object operations that the Amazon S3 API provides. Amazon Web Services (AWS) command-line interface (CLI) is one such utility tool, used to interact with S3 Gateway and work with various Ozone storage elements.

Examples of using AWS CLI for Ozone S3 Gateway :

• Create new bucket

• Upload key to new bucket

• Confirm key upload

• Verify file content through ozone

SSL enabled Ozone2 S3 Gateway to work with AWS CLI

In case of SSL-enabled Ozone, S3 Gateway has https endpoint. Python SSL supported with AWS CLI honors certificates in the PEM format. Hence, convert your CA certificate to PEM if using any other format, on all required client nodes.

Pass the certificate in PEM file format to the aws s3api commands to perform S3 utility tasks. For example :

• Create new bucket

• Upload key to new bucket

• Confirm key upload

• Verify file content through ozone

Revoke access to generated aws credentials

Revoke access to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY once your usecase is completed.

Known Issues / Limitations

• Only a single Ozone installation is supported per cluster.
• Currently, Ozone Manager Java Heap size is set to half of the machine RAM. Adjust the Java heap size as required at the time of installation.
• Ozone mpack supports secure ozone only on fresh ozone installations on kerberized clusters.
• In the new Ozone2 mpack, the S3 Gateway (S3G) web utility has been split across two separate ports, which may result in warning alerts being displayed. https://issues.apache.org/jira/browse/HDDS-7307. The new ozone S3G port for jmx operations is 19878 (default), while the S3 Gateway endpoint for static content is 9878 (default).
• HA and non-HA ozone installation is only applicable on fresh installation. Non-HA to HA migration is yet to be certified.