Search
Finalize Upgrade Pre-Check” Step Fails During Stack Express Upgrade
Description: The “Finalize Upgrade Pre-Check” step may fail during stack express upgrade if Kafka add-ons (Cruise Control, Kafka Connect, Kafka MirrorMaker) are installed.
Workaround:
- Remove all Kafka add-ons before starting the upgrade.
- Proceed with the upgrade.
- Reinstall compatible Kafka add-ons after upgrade completion (if required).
Upgrade Service Configs Step Fails
- Workaround (Option 1): Run Ambari Upgrade Helper Script
xxxxxxxxxxgit clone https://github.com/acceldata-io/ce-utils.git cd ce-utils cd ./odp-upgrade-to-3_3_6_4_1/ # #Run pre-ambari upgrade script bash ./upgrade_ambari_336.sh- Workaround (Option 2): Select "Ignore and proceed."
Below instructions are applied for ODP-3.3.6.1-1 related upgrades
Oozie Restart Fails with “No Port in URL” for Non-SSL Clusters
- Stage: Post-Upgrade Validation / Service Restart
Error Observed:No Port in URLRoot Cause:
- The configuration property
oozie.https.enabledwas newly introduced in 3.3.6.2-1. - For existing non-SSL clusters, this property is missing, causing Oozie to fail during startup.
Workaround:
- In Ambari, navigate to Configs → Oozie → Custom oozie-site.
- Add the following property:
xxxxxxxxxxoozie.https.enabled = false- Restart the Oozie service.
Druid Shuts Down After Ambari Upgrade to 3.0.0
Description: After upgrading Ambari to 3.0.0, the Druid service may shut down or fail to start.
Root Cause: The existing druid-env configuration is incompatible with Ambari 3.0.0. Required environment variables are missing or not exported correctly during service startup.
Workaround / Fix: Update Advanced → druid-env → druid-env template with the following content:
x
# Set DRUID specific environment variables here. # Define paths for the log files and heap dumps log_dir="/var/log/druid" jaas_file="{druid_jaas_file}" {% if java_version == 8 %} # Java 8 JVM options export druid.broker.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:${log_dir}/broker.gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=10m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/broker.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.coordinator.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:${log_dir}/coordinator.gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=10m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/coordinator.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.middlemanager.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:${log_dir}/middlemanager.gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=10m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/middlemanager.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.historical.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:${log_dir}/historical.gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=10m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/historical.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.overlord.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:${log_dir}/overlord.gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=10m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/overlord.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.router.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:${log_dir}/router.gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=10m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/router.hprof -Djava.security.auth.login.config=${jaas_file}" {% else %} # Java 11 JVM options export druid.broker.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -Xlog:gc*,gc+heap=debug,gc+phases=debug:file=${log_dir}/broker.gc.log:time,level,tags:filecount=50,filesize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/broker.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.coordinator.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -Xlog:gc*,gc+heap=debug,gc+phases=debug:file=${log_dir}/coordinator.gc.log:time,level,tags:filecount=50,filesize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/coordinator.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.middlemanager.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -Xlog:gc*,gc+heap=debug,gc+phases=debug:file=${log_dir}/middlemanager.gc.log:time,level,tags:filecount=50,filesize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/middlemanager.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.historical.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -Xlog:gc*,gc+heap=debug,gc+phases=debug:file=${log_dir}/historical.gc.log:time,level,tags:filecount=50,filesize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/historical.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.overlord.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -Xlog:gc*,gc+heap=debug,gc+phases=debug:file=${log_dir}/overlord.gc.log:time,level,tags:filecount=50,filesize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/overlord.hprof -Djava.security.auth.login.config=${jaas_file}" export druid.router.jvm.opts="-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -Xlog:gc*,gc+heap=debug,gc+phases=debug:file=${log_dir}/router.gc.log:time,level,tags:filecount=50,filesize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${log_dir}/router.hprof -Djava.security.auth.login.config=${jaas_file}" {% endif %} # The java implementation to use. export JAVA_HOME={{java8_home}} export PATH=$JAVA_HOME/bin:$PATH export DRUID_PID_DIR={{druid_pid_dir}} export DRUID_LOG_DIR={{druid_log_dir}} export DRUID_CONF_DIR={{druid_conf_dir}} export DRUID_LIB_DIR={{druid_home}}/lib export HADOOP_CONF_DIR={{hadoop_conf_dir}} ##export ssl password variables: export DRUID_CLIENT_HTTPS_TRUSTSTOREPASSWORD=`{{password_command}} "druid.client.https.truststorepassword" ` export DRUID_SERVER_HTTPS_KEYSTOREPASSWORD=`{{password_command}} "druid.server.https.keystorepassword"` ##export metadata password variable: export DRUID_METADATA_STORAGE_CONNECTOR_PASSWORD=`{{password_command}} "druid.metadata.storage.connector.password"`Recommended to move configuration files from /etc/druid/conf/* on druid hosts, and restart Druid service.
Was this page helpful?