Analyzing the HDFS Audit Logs
The HDFS Audit tab lets you view and search HDFS audit logs for security and compliance purposes. These logs contain information on activities recorded from every IP address plugged into the node.
To access the HDFS Audit Logs page:
- Log in to the Pulse UI and navigate to the Logs page in the left menu.
- Click the HDFS Audit tab.
The HDFS Audit logs appear on the screen.
Pulse provides the following functionalities to search or filter the logs displayed.
| Functionality | Description |
|---|---|
| Filters | Turn on the Filters functionality to filter the logs based on the service, host, and log level. |
| Timestamp | Choose an option to select a time period (For example, Today, Last 12 hours, Last 3 Months, etc.) or choose a “custom date and time” of your choice and click Apply. |
| Search Type | Select the Search type:
|
| Search Logs |
Filters or Keywords: Ugi, cms, src, dst, perm, allowed, ip, proto, callerContext, Host, Log level, Service For more details, see Query Examples. |
Searching the HDFS Audit Logs
You can search for audit logs associated with services installed in your cluster. These logs are displayed within the timeline you select.
- On the HDFS Audit tab, select the Time period and Search Type. Select one of the following search types to search for logs:
- Standard Search: Select filters like Ugi, cms, src, dst, perm, allowed, ip, proto, callerContext, Host, Log level, and Service, add an operator, and enter the details to search logs.
- Elastic QS Search: Create a query using keywords like Ugi, cms, src, dst, perm, allowed, ip, proto, callerContext, Host, Log level, and Service, etc. along with operators and corresponding details. For more details, see Query Examples.
The query string accepts only boolean operators.
- Click Enter.
The Logs are displayed for the query you have created within the timeline selected.
Filtering the HDFS Audit Logs
You can filter the audit logs associated with services installed in your cluster. These logs are displayed within the timeline you select.
- On the HDFS Audit tab, turn on the filter option (if turned off), and select the timeperiod.
- On the left panel under Filters, you can apply the following filters for searching a log file.
- Service: Search and select the service to filter logs. These services display the time stamp at which the associated logs were last captured at.
- Host: Select a host from the list. Displays the list of logs of applications running on the host.
- Log level: Select a Log level. Displays the list of logs that belong to the log level selected.
- Cmd: Search and select the command to filter logs.
- Proto: Select a protocol. Displays the list of logs belonging to the protocol selected.
- Allowed: Select true if the operation is allowed and false if the operation is not allowed.
You can select multiple filters at a time to filter the logs as needed.
The histogram, dashboard, and message details get updated automatically upon selecting a service, host, or log-level details.
Visual Log Insights
Time Histograms
This panel displays the logs in the form of a histogram where you can see the number of records for a particular time frame. You can view the histogram by Severity and Service. Click the drop-down located in the top right of the Time Histograms tile and choose a view.
The Severity view displays the number of records of the following: Error, Info, Warn, Debug, Trace. The following image is an example of the Severity view.
Log Details Panel
| Column Name | Description |
|---|---|
| Log Time | The time at which the log was captured. |
| UGI | The User Group Information code. This value is displayed along with the authorization level given to the UGI. |
| CMD | The command used at that particular time. To copy the command, click |
| SRC | The source path. To copy the path, click |
| DST | The destination path. To copy the path, click |
| PREM | The permissions given to the user, either read, write, or execute. To copy the permission, click |
| Allowed | The permission status of an operation, true if operation allowed and false if operation not allowed. |
| IP | The IP address of the user performing the operation. |
| Proto | The protocol used. |
| Caller Context | The tracking id of the application. To copy the ID, click |
The logs are divided in the following severity levels:
- Error
- Info
- Warn
- Debug
- Trace
You can click a label to view the log of that severity level.
Features
Grouping Logs
You can group the audit logs by Trace, Severity, or Host. Click the Group By drop-down label in the top left corner of the log details panel and select a grouping criteria.
Downloading Logs
To download the logs, click the icon in the top right corner of the table. The logs can be downloaded in the .xlsx or .logs format. Pulse allows you to specify the number of rows to download.