HDFS
The Hadoop Distributed File System (HDFS) is a distributed file system that uses a NameNode and DataNode structure to provide high-performance access to data across Hadoop clusters that are extremely scalable. Hadoop's primary data storage system is HDFS.
HDFS distributes data across several servers and replicates it to enhance data durability and high availability for parallel applications.
HDFS Dashboard
Enabling HDFS monitoring on Pulse gives you an in-depth overview of the Hadoop file system within your cluster. You can also observe and monitor the overall health of your HDFS cluster pertaining to the CPU and memory usage.
The statistics displayed are from the last 24 hours by default. To view statistics from a custom date range, click the icon in the top-right corner of the page and select the required time frame and timezone.
The following table provides the details of the Name Nodes and Data Nodes details of your cluster:
| Name | Description |
|---|---|
| Name Nodes Active | The number of active name nodes in the cluster. |
| Num Live Data Nodes | The number of data nodes that are currently live. |
| Num Dead Data Nodes | The number of data nodes in a dead state. |
| Num Stale Data Nodes | The number of stale data nodes. |
| Num Decom Live Data Nodes | The number of decommission data nodes that are currently live. |
The following table provides the details of the capacity of your cluster:
| Name | Description |
|---|---|
| Capacity Remaining | The amount of HDFS storage available on the cluster, in MB. This is calculated as Capacity Total - (Capacity Used + Capacity used non DFS). |
| Capacity Total | The total HDFS storage on the cluster. |
| Capacity Used | The HDFS storage used on the cluster. |
| Capacity Used Non DFS | The storage used by data in the data node that is not included in DFS. |
The following table provides the details of the files:
| Chart Name | Description |
|---|---|
| Total Files | The total number of files on the HDFS cluster. |
| Files Under Construction | The number of HDFS files that are still being written. |
| Lock Queue Length | The number of threads waiting to acquire FSNameSystemLock. |
| Num Active Clients | The total number of active clients holding lease in the system. |
Charts on HDFS Dashboard
The following table provides description of the charts on the HDFS Dashboard. All charts contain aggregated values of the metrics.
To view usage by node, click Show Individuals in the chart of your choice.
| Chart Name | Description |
|---|---|
| HDFS NameNode Process | Live tracking for active state of HDFS Namenode process, during the selected time period. |
| HDFS DataNode Process | Live tracking for active state of HDFS DataNode process, during the selected time period. |
| HDFS DataNode State Timeline | The number of Live DataNodes, Stale DataNode, Dead DataNodes. DecomLiveDataNodes on the cluster for the selected timeline. |
| CPU Usage Name Node | The rate of change of CPU utilization by a NameNode on the host, during the selected time period. |
| CPU Usage Data Node | The rate of change of CPU utilization by a DataNode on the host, during the selected time period. |
| Physical Memory NameNode | The average memory (RSS - resident set size) usage by NameNode process running on a host. |
| Physical Memory DataNode | The average memory (RSS - resident set size) usage by DataNode process running on a host. |
| Heap Memory NameNode | The amount of heap memory used/committed/maximum by the NameNode process (in MB or GB). |
| Heap Memory DataNode | The amount of heap memory used/committed/maximum by the DataNode process (in MB). |
| File Summary Trend | The file summary trend showing the Total Files, Files Under Construction, LockQueueLength, and numActiveClients. |
| Active Connections | The 95th percentile value of number of active clients holding lease in the system. |
| NN Running Threads | The number of RUNNABLE threads for NameNode process. |
| DN Running Threads | The number of RUNNABLE threads for DataNode process. |
| NN Waiting Threads | The number of WAITING and TIMED_WAITING threads for the NameNode process. |
| DN Waiting Threads | The number of WAITING and TIMED_WAITING threads for the DataNode process. |
| HDFS Usage Trend | The HDFS Usage summary trend showing CapacityTotal, CapacityUsed, and CapacityRemaining.
|
| Block Summary Trend | Blocks are the smallest units of storage on the host where data or files are broken down into chunks and stored in continuous manner. HDFS distributes these blocks across the Hadoop cluster. The block summary trend shows the following: Blocks Total: Total number of blocks in the cluster. Missing Blocks: Number of blocks having no replicas in the Hadoop cluster. Pending Replication Blocks: Number of blocks that are not yet replicated. Under Replicated Blocks: Number of blocks having replication factor less than the specified value. Corrupt Blocks - number of blocks with corrupt replicas. |
| Data Skewness | The Total Capacity and Capacity Used for each DataNode. |
| Top Operations | The most number of HDFS file operation commands running on the HDFS. |
| Top User by Operations | The users running the most number of HDFS operations. |
| RPC Times | The time taken to complete RPC calls for the following criteria. RPC Processing Time: Time taken to process RPC calls. RPC Queue Time: Time taken for an RPC call to start, or the waiting time of an RPC call. |
| RPC Operations | The number of RPC operations in the Hadoop cluster for the following types. Processed Ops: The number of processed RPC operations. Queued Ops: The number of RPC operations in queue and yet to start processing. |
| HDFS Usage by User | The amount of HDFS storage used by each user in the cluster. |
Dashplots style is reflected on the dashboards. Ensure to install Dashplots to view the diagrams in the dashboard.