Update Real-time Monitoring Parameters of HDFS

This page describes how to enable and configure real-time monitoring of HDFS in Pulse.

Step 1: Edit Configuration File

On the Pulse node, navigate to the following path and edit the configuration file:

$AcceloHome/config/acceldata_<clustername>.conf

Step 2: Configure Real-Time Monitoring

By default, the following parameter is enabled:

fsanalytics.inotify.enabled=true

This setting enables real-time monitoring of HDFS using the inotify mechanism.

You may update other related parameters in the configuration file as needed, based on your environment and monitoring requirements.

{ name = "fsanalytics-connector" trigger = "scheduled" fsanalytics.off.heap.storage = "/etc/fsanalytics" fsanalytics.metastore { # HIVE METASTORE is ENABLED during the setup enabled = true encrypted = true DAYS = 7 } hive.schema.prefix = "hive." # Inotify configuration fsanalytics.inotify.enabled = true # Enable/Disable Inotify feature fsanalytics.inotify.poll.interval.ms = 1000 # Time between consecutive polls to HDFS Inotify fsanalytics.inotify.poll.timeout.ms = 1000 # Sets poll timeout fsanalytics.inotify.es.flush.interval.ms = 1 fsanalytics.inotify.es.batch.size = 100 fsanalytics.inotify.es.buffer.capacity = 10000 slick-mysql { profile = "slick.jdbc.MySQLProfile$" db { dataSourceClass = "slick.jdbc.DriverDataSource" properties = { driver = "org.mariadb.jdbc.Driver" # Check the server timezone # Check & validate the metastore DB Name url = "jdbc:mysql://odp311.acceldata.dvl:3306/hive?serverTimezone=UTC&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&useUnicode=true" user = "hive" password = "ColrOFzIvF+y0RK0807jpg==" } } } }

Info

Limitation: Snapshot Processing Snapshot functionality will NOT be available in the real-time system as HDFS inotify does not provide snapshot-related events & nodeId. The system focuses exclusively on regular file system operations (fsimage processing).

Limitation: Non-HDFS Operations Permission changes or file system operations performed outside standard HDFS APIs (i.e., not executed via HDFS commands or NameNode-backed interfaces) will not generate inotify events and will not be captured in real time. Such changes will only be reflected during subsequent fsimage processing.

Last updated on

Was this page helpful?