Search
Acceldata Open Source Data Platform
ODP 3.2.3.2-3
What is ODP
Installation
Advanced Installation
Configuration and Management
Upgrade
USER GUIDES
Security
Uninstall ODP
Page
Code Steps
Category
Label
Link
Separator
Title
Message
Page icon
No icon. This page keeps the plain index row.
orSVG or PNG, square, at least 32×32.
Create new category
What is the title of your new category?
Edit page index title
What is the title of the page index?
Edit category
What is the new title of your category?
Edit link
What is the new title and URL of your link?
Known Limitations
Summarize Page
Copy Markdown
Open in ChatGPT
Open in Claude
Connect to Cursor
Connect to VS Code
Known Limitations
This section consists of known limitations we are aware of, persisting in this release.
During the ODP upgrade, where Ambari-2.7.8.0 has already been upgraded, the following services will experience downtime. This downtime will interrupt the ODP stack upgrade when using express or rolling upgrade methods.
Affected Services:
Kafka
Ranger
Ranger KMS
Kafka
After completing the Ambari upgrade, during the rolling or express upgrade, Kafka needs to be up and running in a healthy mode. However, Kafka is in a stopped state due to the following errors:
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'kafka-connect-distributed' was not found in configurations dictionary!
These issues occur because Kafka Connect and MirrorMaker were added in this release.
Solution
To start Kafka, add the following configurations in Ambari:
Custom kafka-mirrormaker2-common
replication.factor =1
Custom kafka-mirrormaker2-destination
replication.enabled=truedestination.cluster.name=desttopics.whitelist=.*
Custom kafka-mirrormaker2-source
replication.enabled=truesource.cluster.name=sourcetopics.whitelist=.*
Custom kafka-connect-distributed
bootstrap.servers=localhost:6667
Proceed with restarting the Kafka service. It should start smoothly.
Ranger
During the cluster upgrade (after Ambari is upgraded to 2.7.8.0 but before the ODP stack is upgraded), the Ranger service will not start due to the following errors:
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'admin-logback' was not found in configurations dictionary!
resource_management.core.exceptions.Fail: Configuration parameter 'tagsync-logback' was not found in configurations dictionary!
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'usersync-logback' was not found in configurations dictionary!
Solution
Log in to the backend database for Ambari-server.
In this case, MySQL is used as the Ambari-server’s backend database.
Execute the following SQL query to identify the required configurations:
SELECT * FROM (
SELECT
cc.config_id,
cc.type_name,
service_name,
sc.version,
sc.note,
(SELECT MAX(version) FROM ambari.serviceconfig s
WHERE s.service_name = sc.service_name) AS max_version
FROM
ambari.serviceconfig sc
INNER JOIN
ambari.serviceconfigmapping scm
ON sc.service_config_id = scm.service_config_id
INNER JOIN
ambari.clusterconfig cc
ON cc.config_id = scm.config_id
WHERE
sc.service_name IN ("RANGER_KMS", "RANGER")
AND sc.version = (SELECT MAX(version) FROM ambari.serviceconfig s
WHERE s.service_name = sc.service_name)
) AS res
WHERE type_name IN ("kms-log4j", "admin-log4j", "tagsync-log4j", "usersync-log4j");
Expected Response:
+-----------+----------------+--------------+---------+----------------------------------------+-------------+
| config_id | type_name | service_name | version | note | max_version |
+-----------+----------------+--------------+---------+----------------------------------------+-------------+
| 232 | kms-log4j | RANGER_KMS | 4 | Enabling Kerberos for added components | 4 |
| 218 | admin-log4j | RANGER | 4 | | 4 |
| 228 | tagsync-log4j | RANGER | 4 | | 4 |
| 229 | usersync-log4j | RANGER | 4 | | 4 |
+-----------+----------------+--------------+---------+----------------------------------------+-------------+
Update the required configurations:
Ranger-admin service: Update
admin-log4jtoadmin-logback
UPDATE ambari.clusterconfig
SET type_name = "admin-logback"
WHERE config_id = 218;
Ranger tagsync service: Update
tagsync-log4jtotagsync-logback
UPDATE ambari.clusterconfig
SET type_name = "tagsync-logback"
WHERE config_id = 228;
Ranger usersync service: Update
usersync-log4jtousersync-logback
UPDATE ambari.clusterconfig
SET type_name = "usersync-logback"
WHERE config_id = 229;
Restart the Ambari-server:
ambari-server restart
Proceed with the Ranger service restart from the Ambari UI. It should start smoothly.
Ranger KMS
During the cluster upgrade (after Ambari is upgraded to 2.7.8.0 but before the ODP stack is upgraded), the Ranger-KMS service will not start due to the following error:
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'kms-logback' was not found in configurations dictionary!
Actually, kms-log4j is replaced with kms-logback starting from version 3.2.3.x.
Solution
Log in to the backend database for the Ambari server.
In this case, MySQL is the backend database for the Ambari server.
select * from (SELECT
cc.config_id,
cc.type_name,
service_name,
sc.version,
sc.note,
(SELECT MAX(version) FROM ambari.serviceconfig s
WHERE s.service_name = sc.service_name) AS max_version
FROM
ambari.serviceconfig sc
INNER JOIN
ambari.serviceconfigmapping scm
ON
sc.service_config_id = scm.service_config_id
INNER JOIN
ambari.clusterconfig cc
ON
cc.config_id = scm.config_id
WHERE
sc.service_name IN ("RANGER_KMS", "RANGER") and sc.version=(SELECT MAX(version) FROM ambari.serviceconfig s
WHERE s.service_name = sc.service_name)) as res where type_name in ("kms-log4j", "admin-log4j", "tagsync-log4j", "usersync-log4j");
Response:
+-----------+----------------+--------------+---------+----------------------------------------+-------------+
| config_id | type_name | service_name | version | note | max_version |
+-----------+----------------+--------------+---------+----------------------------------------+-------------+
| 232 | kms-log4j | RANGER_KMS | 4 | Enabling Kerberos for added components | 4 |
| 218 | admin-log4j | RANGER | 4 | | 4 |
| 228 | tagsync-log4j | RANGER | 4 | | 4 |
| 229 | usersync-log4j | RANGER | 4 | | 4 |
+-----------+----------------+--------------+---------+----------------------------------------+-------------+
Ranger-Admin Service Required Change
Update kms-log4j to kms-logback:
UPDATE ambari.clusterconfig
SET type_name = "kms-logback"
WHERE config_id = 232;
From the Ambari-Server host, perform the following steps:
Edit the File:
Path: /var/lib/ambari-server/resources/stacks/ODP/3.0/services/RANGER_KMS/package/scripts/kms.py
Locate the following lines:
File(os.path.join(params.kms_conf_dir, "kms-log4j.properties"),
owner=params.kms_user,
group=params.kms_group,
content=InlineTemplate(params.kms_log4j),
mode=0644
)
Comment them out as shown below:
"""
File(os.path.join(params.kms_conf_dir, "kms-log4j.properties"),
owner=params.kms_user,
group=params.kms_group,
content=InlineTemplate(params.kms_log4j),
mode=0644
)
"""
Restart the Ambari-Server:
ambari-server restart
Proceed with restarting the Ranger-KMS service from the Ambari UI. It should start smoothly.
Update the Ranger Configuration
Log in to the Ambari UI and navigate to
RANGER→CONFIGS->ADVANCED->Advanced admin-logback, update the following content, and save.
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<appender name="xa_log_appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{admin_log_dir}}/ranger-admin-${hostname}-${user}.log</file>
<append>true</append>
<encoder>
<pattern>%date [%thread] %level{5} [%file:%line] %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger-admin-${hostname}-${user}.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="sql_appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{admin_log_dir}}/ranger_admin_sql.log</file>
<append>true</append>
<encoder>
<pattern>%d [%t] %-5p %C{6} \(%F:%L\) %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger_admin_sql.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="perf_appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{admin_log_dir}}/ranger_admin_perf.log</file>
<append>true</append>
<encoder>
<pattern>%d [%t] %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger_admin_perf.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="patch_logger" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<append>true</append>
<file>{{admin_log_dir}}/ranger_db_patch.log</file>
<encoder>
<pattern>%d [%t] %-5p %C{6} \(%F:%L\) %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger_db_patch.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<logger name="xa" additivity="false" level="info">
<appender-ref ref="xa_log_appender"/>
</logger>
<logger name="jdbc.connection" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="com.mchange" additivity="false" level="warn" />
<logger name="org.apache.ranger.perf" additivity="false" level="info">
<appender-ref ref="perf_appender"/>
</logger>
<logger name="jdbc.audit" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.apache.ranger.patch" additivity="false" level="info">
<appender-ref ref="patch_logger"/>
</logger>
<logger name="jdbc.resultset" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.springframework" additivity="false" level="warn">
<appender-ref ref="patch_logger"/>
</logger>
<logger name="jdbc.sqltiming" additivity="false" level="warn">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.hibernate.SQL" additivity="false" level="warn">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.apache.ranger" additivity="false" level="info">
<appender-ref ref="xa_log_appender"/>
</logger>
<logger name="jdbc.sqlonly" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<root level="warn">
<appender-ref ref="xa_log_appender"/>
</root>
</configuration>
On the Ambari UI, navigate to
RANGER→CONFIGS->ADVANCED->Advanced tagsync-logback,update with the following content, and save.
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<appender name="logFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{tagsync_log_dir}}/tagsync.log</file>
<encoder>
<pattern>%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %L %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{tagsync_log_dir}}/tagsync.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<root level="info">
<appender-ref ref="logFile"/>
</root>
</configuration>
On the Ambari UI, navigate to
RANGER→CONFIGS->ADVANCED->Advanced usersync-logback,update with the following content, and save.
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<appender name="logFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{usersync_log_dir}}/usersync-${hostname}-${user}.log</file>
<encoder>
<pattern>%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{usersync_log_dir}}/usersync-${hostname}-${user}.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<root level="info">
<appender-ref ref="logFile"/>
</root>
</configuration>
Restart the Ranger service from the Ambari UI.
Update the Ranger KMS Configuration
Log in to the Ambari UI and navigate to
RANGER→CONFIGS->ADVANCED->Advanced admin-logback,update the following content, and save.
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration scan="true">
<appender name="kms-audit" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<Append>true</Append>
<File>${kms.log.dir}/kms-audit-${hostname}-${user}.log</File>
<encoder>
<pattern>%d{ISO8601} %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${kms.log.dir}/kms-audit-${hostname}-${user}.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="kms-metric" class="ch.qos.logback.core.FileAppender">
<!--See also http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<Append>false</Append>
<File>${kms.log.dir}/ranger_kms_metric_data_for_${metric.type}.log</File>
<encoder>
<pattern>%m%n</pattern>
</encoder>
</appender>
<appender name="kms" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<File>${kms.log.dir}/ranger-kms-${hostname}-${user}.log</File>
<Append>true</Append>
<encoder>
<pattern>%d{ISO8601} %-5p [%t] %c{1} \(%F:%L\) - %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${kms.log.dir}/ranger-kms-${hostname}-${user}.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<logger name="com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator" level="OFF"/>
<logger name="kms-audit" additivity="false" level="INFO">
<appender-ref ref="kms-audit"/>
</logger>
<logger name="org.apache.hadoop" level="INFO"/>
<logger name="org.apache.hadoop.conf" level="INFO"/>
<logger name="org.apache.hadoop.crypto.key.kms.server.KMSMetricUtil" level="INFO">
<appender-ref ref="kms-metric"/>
</logger>
<logger name="org.apache.ranger" level="INFO"/>
<root level="WARN">
<appender-ref ref="kms"/>
</root>
</configuration>
Restart the Ranger KMS service from the Ambari UI.
Other Known Limitations
Ozone
Ozone service check fails when Ranger is enabled. Workaround: Create a new policy for
ambari-qauser to access only ambarismokevolume as an admin.Ozone mpack supports Secure Ozone only on fresh Ozone installations on Kerberized clusters.
Ozone 1.4.0 limits Ranger Audit due to open bugs related to performance.
Last updated on Jan 15, 2025
Was this page helpful?
Next to read:
Fixed CVEsnull
Discard Changes
Do you want to discard your current changes and overwrite with the template?
Archive Synced Block
Message
Create new Template
What is this template's title?
Delete Template
Message