You can dynamically change the log levels of Hadoop services (such as YARN and HDFS) using the hadoop daemonlog command without needing to restart the services.
Key Notes
Ports: Replace <host:port> with the admin/HTTP port of the service (not the web UI port).
Class Names: Use the fully qualified class name of the service’s main class (e.g., org.apache.hadoop.yarn.server.resourcemanager.ResourceManager).
Log Levels: Supported levels: TRACE, DEBUG, INFO, WARN, ERROR, FATAL.
Ensure that the mentioned commands on this page are executed by the user running the respective process. For NameNode-related operations, specifically use the NN service keytab and the hdfs user to initialize the Kerberos token before running the command.
Persistence: Changes are temporary and reset after service restart. For permanent changes, modify log4j.properties and restart the service.
YARN Services
Resource Manager
hadoop daemonlog -getlevel <resourcemanager_host>:8088 \\
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
The example output is as follows.
Connecting to <http://rm-host:8088/logLevel?log=org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
Submitted Class Name: org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
Log Class: org.apache.commons.logging.impl.Log4JLogger
Effective Level: INFO
hadoop daemonlog -setlevel <resourcemanager_host>:8088 \\
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager DEBUG
The example output is as follows.
Connecting to http://rm-host:8088/logLevel?log=org.apache.hadoop.yarn.server.resourcemanager.ResourceManager&level=DEBUG
Submitted Class Name: org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
Log Class: org.apache.commons.logging.impl.Log4JLogger
Submitted Level: DEBUG
Setting Level to DEBUG ...
Effective Level: DEBUG
Node Manager
hadoop daemonlog -getlevel <nodemanager_host>:8042 \\
org.apache.hadoop.yarn.server.nodemanager.NodeManager
The example output is as follows.
Connecting to http://trial04.acceldata.dvl:8042/logLevel?log=org.apache.hadoop.yarn.server.nodemanager.NodeManager
Submitted Class Name: org.apache.hadoop.yarn.server.nodemanager.NodeManager
Log Class: org.apache.commons.logging
hadoop daemonlog -setlevel <nodemanager_host>:8042 \\
org.apache.hadoop.yarn.server.nodemanager.NodeManager WARN
The example output is as follows.
Connecting to http://NM-Host:8042/logLevel?log=org.apache.hadoop.yarn.server.nodemanager.NodeManager&level=WARN
Submitted Class Name: org.apache.hadoop.yarn.server.nodemanager.NodeManager
Log Class: org.apache.commons.logging.impl.Log4JLogger
Submitted Level: WARN
Setting Level to WARN ...
Effective Level: WARN
HDFS Services
NameNode
hadoop daemonlog -getlevel <namenode_host>:50070 \\
org.apache.hadoop.hdfs.server.namenode.NameNode
The example output is as follows.
Connecting to http://NN-HOST:50070/logLevel?log=org.apache.hadoop.hdfs.server.namenode.NameNode
Submitted Class Name: org.apache.hadoop.hdfs.server.namenode.NameNode
Log Class: org.apache.commons.logging.impl.Log4JLogger
Effective Level: INFO
hadoop daemonlog -setlevel <namenode_host>:50070 \\
org.apache.hadoop.hdfs.server.namenode.NameNode ERROR
The example output is as follows.
Connecting to http://NN-HOST:50070/logLevel?log=org.apache.hadoop.hdfs.server.namenode.NameNode&level=ERROR
Submitted Class Name: org.apache.hadoop.hdfs.server.namenode.NameNode
Log Class: org.apache.commons.logging.impl.Log4JLogger
Submitted Level: ERROR
Setting Level to ERROR ...
Effective Level: ERROR
DataNode
hadoop daemonlog -getlevel <datanode_host>:50075 \\
org.apache.hadoop.hdfs.server.datanode.DataNode
The example output is as follows.
Connecting to http://DN-HOST:50075/logLevel?log=org.apache.hadoop.hdfs.server.datanode.DataNode
Submitted Class Name: org.apache.hadoop.hdfs.server.datanode.DataNode
Log Class: org.apache.commons.logging.impl.Log4JLogger
Effective Level: INFO
hadoop daemonlog -setlevel <datanode_host>:50075 \\
org.apache.hadoop.hdfs.server.datanode.DataNode DEBUG
The example output is as follows.
Connecting to http://DN-HOST:50075/logLevel?log=org.apache.hadoop.hdfs.server.datanode.DataNode&level=DEBUG
Submitted Class Name: org.apache.hadoop.hdfs.server.datanode.DataNode
Log Class: org.apache.commons.logging.impl.Log4JLogger
Submitted Level: DEBUG
Setting Level to DEBUG ...
Effective Level: DEBUG
Other Services
MapReduce JobHistory Server
hadoop daemonlog -setlevel <historyserver_host>:19888 \\
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer DEBUG
The example output is as follows.
Connecting to http://MR-HOST:19888/logLevel?log=org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer&level=DEBUG
Submitted Class Name: org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
Log Class: org.apache.commons.logging.impl.Log4JLogger
Submitted Level: DEBUG
Setting Level to DEBUG ...
Effective Level: DEBUG
Timeline Server
hadoop daemonlog -setlevel <timelineserver_host>:8188 \\
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer WARN
The example output is as follows.
Connecting to http://Timeline-Host:8188/logLevel?log=org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer&level=WARN
Submitted Class Name: org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer
Log Class: org.apache.commons.logging.impl.Log4JLogger
Submitted Level: WARN
Setting Level to WARN ...
Effective Level: WARN
Verification
tail -f /var/log/hadoop-yarn/yarn-*-resourcemanager-*.log
hadoop daemonlog -getlevel <host:port> <classname>
Example Summary
Service | Command (Set to Debug) |
|---|
ResourceManager | hadoop daemonlog -setlevel rm-host:8088 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager DEBUG
|
NodeManager | hadoop daemonlog -setlevel nm-host:8042 org.apache.hadoop.yarn.server.nodemanager.NodeManager DEBUG
|
NameNode | hadoop daemonlog -setlevel nn-host:9870 org.apache.hadoop.hdfs.server.namenode.NameNode DEBUG
|
For HTTPS clusters, add -protocol https to the command.
