Working with Ozone File System

Prerequisites

To enable ofs support with applications, configure applications to use necessary jars and ozone2-site.xml.

• Add the ozone-filesystem-hadoop3.jar to the application classpath.

xxxxxxxxxx

# hadoop-env

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/odp/current/ozone2-client/share/ozone/lib/ozone-filesystem-hadoop3-*.jar

• Add the following configs to core-site.xml

xxxxxxxxxx

<property>

    <name>fs.ofs.impl</name

    <value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value>

</property>

• Add following configs from ozone2-site.xml to hdfs-site.xml on Ambari.

• Restart hadoop services

HDFS with OFS

Access hdfs dfs operations with ozone2 storage :

xxxxxxxxxx

$ hdfs dfs [options] OFS_URI

Here are some examples :

• List files

$ hdfs dfs -ls ofs://omservice/

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/odp/3.2.2.0-1/hadoop/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/odp/3.2.2.0-1/hadoop-hdfs/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]

24/03/20 16:15:08 INFO client.ClientTrustManager: Loading certificates for client.

Found 2 items

drwxrwxrwx   - hdfs hadoop          0 2024-03-20 17:58 ofs://omservice/s3v

drwxrwxrwx   - hdfs hadoop          0 2024-03-20 18:37 ofs://omservice/testvol

• Create directory

$ hdfs dfs -mkdir ofs://omservice/testvol/testbucket

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/odp/3.3.0.0-1/hadoop/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/odp/3.3.0.0-1/hadoop-hdfs/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/odp/3.3.0.0-1/tez/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]

24/05/02 13:18:57 INFO client.ClientTrustManager: Loading certificates for client.

24/05/02 13:18:58 INFO rpc.RpcClient: Creating Bucket: testvol/testbucket, with bucket layout FILE_SYSTEM_OPTIMIZED, ambari-qa as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1

• Upload file

xxxxxxxxxx

$ vi /tmp/README.md

hi,

this is README.md

$ hdfs dfs -put /tmp/README.md ofs://omservice/testvol/testbucket/

$ hdfs dfs -ls ofs://omservice/testvol/testbucket/

Found 1 items

-rw-rw-rw-   3 ambari-qa ambari-qa         22 2024-04-30 13:31 ofs://omservice/testvol/testbucket/README.md

• Reading file

xxxxxxxxxx

$ hdfs dfs -cat ofs://omservice/testvol/testbucket/README.md

24/05/02 13:37:40 INFO client.ClientTrustManager: Loading certificates for client.

24/05/02 13:37:40 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties

24/05/02 13:37:40 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).

24/05/02 13:37:40 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started

hi,

this is README.md

YARN with Ozone2

Yarn can be used to run jobs with jobs accessing data from or writing into ozone file system.

• Add ozone-filesystem-hadoop3-1.4.0_.jar to _mapreduce.application.classpath* in mapred-site.xml .
• If ranger authorization is enabled, provide necessary permissions to ofs buckets, hdfs path, yarn queues, to perform necessary operations as per job requirements.
• Perform respective user kerberos authentication in case of secure cluster.
• submit job

Here is a sample job, doing wordcount on data from file in ofs, and storing the output file with wordcount result in ofs.

Configure mapreduce job to use ozone-site.xml. Alternatively, you can pass configs during runtime:

HIVE with Ozone

Although HIVE installation and operations use HDFS as default file system, ozone can be configured to be parallel file system for HIVE operations.

Configure Hive to work with Ozone :

• Navigate to the Ambari UI > Hive > Configs > Advanced Hive-env and add

• Restart Hive and Tez.
• If Ranger authorization is enabled, grant the necessary permissions to OFS (Ozone File System) buckets, HDFS paths, and Hive URL to allow the required operations as per the job requirements.
• Authenticate users with Kerberos credentials when operating in a secure cluster.

If queries are failing with below error when run queries as end user is enabled in hive org.apache.hadoop.security.authorize.AuthorizationException: User: hive is not allowed to impersonate ... https://issues.apache.org/jira/browse/HDDS-664

Ambari UI > Ozone > Configurations > Custom Core-site: add the following configs and restart services :

Store tables in OFS

To create tables in OFS add LOCATION '<OFS_URI> to CREATE TABLE command. This will make Hive tables reside at the specified location in ozone. All data changes her after will be in effect at table in given OFS_URI.

Here are sample hive operations with Hive accessing OFS :

• Connect to Beeline
• Create new table in ofs

• Validate new table

• Add values to table

• Validate newly added values

SPARK with Ozone2

Apache Spark can access data from Apache Ozone2 and perform tasks. To access Apache Ozone2, configure spark :

• Configure spark shell to use /usr/odp/current/ozone2-client/share/ozone/lib/ozone-filesystem-hadoop3-client-2.1.0.3.3.6.2-104.jar.

Accessing Apache Ozone data in Apache Spark3

• Creating sample data to be read by Spark3

• Upload the employee.csv file to Ozone2

• Provide necessary permissions under ozone policies, for spark user to access respective bucket and file, if ranger authorization is enabled.
• Allow spark user to submit yarn applications.
• Launch spark-shell

• Accessing csv file content in ozone as spark df

Custom PySpark Job

To run spark job using ofs use following command:

For a secure cluster, add --keytab <keytab> --principal <principal> values to above command.

Here is a sample custom job that functions to access Ozone data with Apache Spark and write output to Ozone.

• Custom Pyspark application using ofs to access data and write output

• Uploading sample input file to ofs

• Provide necessary permissions to spark user to access respective bucket and key, in case of ranger authorization enabled.
• Running PySpark app in secure cluster

• Validate output in ofs