You can perform the following steps to upgrade the Ambari Server.
The following steps have to be performed on the installed Ambari Server node.
Stop Infra-Solr Service
Stop the Infra-solr Service from the Ambari UI by navigating to Ambari > Infra Solr > Actions > Stop.
Stop Ambari Server and Agents
Stop the Ambari server and agents by running the following command.
xxxxxxxxxxambari-server stopambari-agent stopConfiguration Backup
Take a backup of the Ambari-server configs.
xxxxxxxxxxcp -r /etc/ambari-server/config /etc/ambari-server/conf_bkpBack up Ambari Database
xxxxxxxxxxmysqldump --databases ambari > /tmp/ambari-server_db_bckp/ambari_mysql_bckp.sqlErase the Ambari Server
For RHEL:
xxxxxxxxxxyum erase ambari-server -yFor Ubuntu:
xxxxxxxxxxapt remove ambari-server -yAdd Ambari Repo on Cluster Nodes
Back up the Old Ambari Repository
For RHEL:
xxxxxxxxxxmv /etc/yum.repos.d/ambari.repo /tmp/For Ubuntu:
xxxxxxxxxxmv /etc/apt/sources.list.d/ambari.list /tmp/Update the Repository Files
Retrieve the latest Ambari repository details from Ambari Repositories and add it to all the cluster nodes.
Update ambari.repo (RedHat) or ambari.list (Ubuntu).
For RHEL:
xxxxxxxxxxvim /etc/yum.repos.d/ambari.repoxxxxxxxxxx[ambari]async = 1baseurl = https://mirror.odp.acceldata.dev/ODP/rhel/Ambari-3.0.0.1-1/gpgcheck = 0name = ambari Version - ambari-3.0.0.1-1For Ubuntu:
xxxxxxxxxxvim /etc/apt/sources.list.d/ambari.listxxxxxxxxxxdeb https://mirror.odp.acceldata.dev/ODP/ubuntu20/Ambari-3.0.0.1-1/ ODP main deb https://mirror.odp.acceldata.dev/ODP/ubuntu20/Ambari-3.0.0.1-1/ ODP-UTILS mainList the New Packages
Verify the latest Ambari package for the version - 3.0.0.1-1.
For RHEL:
xxxxxxxxxxyum clean allyum list ambari-*For Ubuntu 20:
xxxxxxxxxxapt clean allapt list ambari-*Install the Updated Ambari Packages
For RHEL:
xxxxxxxxxxyum clean allyum install ambari-server -yFor Ubuntu:
xxxxxxxxxxapt cleanapt install ambari-server -yRestore the Configuration Files
Copy ambari.properties and password.dat to /etc/ambari-server/conf/.
Once the installation is successful, make sure to set up the following versions.
Set up Java 17
Make sure to install and update the default Java version as 17 on all the cluster nodes.
For RHEL:
xxxxxxxxxxyum install java-17-openjdk.x86_64yum install java-17-openjdk-devel.x86_64For Ubuntu:
xxxxxxxxxxapt-get install openjdk-17-jdk openjdk-17-jdk-headlessEnsure not to remove the jdk8 packages at this state.
xxxxxxxxxxupdate-alternatives --config java There are 2 choices for the alternative java (providing /usr/bin/java). Selection Path Priority Status------------------------------------------------------------* 0 /usr/lib/jvm/java-17-openjdk-amd64/bin/java 1111 auto mode 1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode 2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode Press <enter> to keep the current choice[*], or type selection number: 1Set up Python 3.11
xxxxxxxxxxyum install python3.11Make sure to install and update the default Python version as 3.11 on all the cluster nodes.
xxxxxxxxxxupdate-alternatives --config python There are 4 programs which provide 'python'. Selection Command-----------------------------------------------* 1 /usr/libexec/no-python 2 /usr/bin/python3 3 /usr/bin/python2 + 4 /usr/bin/python3.11Enter to keep the current selection[+], or type selection number: 4Update JSVC
Make sure to Instal the latest version of the bigtop-jsvc-1.2.4 on all the cluster nodes.
For RHEL:
xxxxxxxxxxyum install bigtop-jsvc* -yFor Ubuntu:
xxxxxxxxxxapt install bigtop-jsvcRestore Config Files
Restore the ambari.properties and password.dat files, which are backed up at this step: Configuration Backup on this page.
Restore the ambari.properties file
xxxxxxxxxxcp /etc/ambari-server/conf_bkp/ambari.properties /etc/ambari-server/conf/Please override the file by prompting 'y'.
Restore the password.dat file
xxxxxxxxxxcp /etc/ambari-server/conf_bkp/password.dat /etc/ambari-server/conf/Update the java.home and stack.java.home
In this case, set JAVA_HOME to /usr/lib/jvm/java-17-openjdk
File path - /etc/ambari-server/conf/ambari.properties | grep -i java.home
xxxxxxxxxxjava.home=/usr/lib/jvm/java-17-openjdkstack.java.home=/usr/lib/jvm/java-17-openjdkambari.java.home=/usr/lib/jvm/java-17-openjdkmpacks-v2.staging.path=/var/lib/ambari-server/resources/mpacks-v2Update the Ambari Database with New Tables and Columns
MySQL
xxxxxxxxxxmysql -u ambari -pambari USE ambari; CREATE TABLE IF NOT EXISTS registries(id BIGINT NOT NULL,registy_name VARCHAR(255) NOT NULL,registry_type VARCHAR(255) NOT NULL,registry_uri VARCHAR(255) NOT NULL,CONSTRAINT PK_registries PRIMARY KEY (id)); CREATE TABLE IF NOT EXISTS mpacks(id BIGINT NOT NULL,mpack_name VARCHAR(255) NOT NULL,mpack_version VARCHAR(255) NOT NULL,mpack_uri VARCHAR(255),registry_id BIGINT,CONSTRAINT PK_mpacks PRIMARY KEY (id),CONSTRAINT uni_mpack_name_version UNIQUE(mpack_name, mpack_version),CONSTRAINT FK_registries FOREIGN KEY (registry_id) REFERENCES registries(id)); ALTER TABLE stack ADD COLUMN mpack_id BIGINT NULL;ALTER TABLE stack ADD CONSTRAINT FK_mpacks FOREIGN KEY (mpack_id) REFERENCES mpacks(id); ALTER TABLE upgrade ADD COLUMN upgrade_package_stack VARCHAR(255) NOT NULL;MariaDB
xxxxxxxxxxmysql -u ambari -pambari USE ambari; CREATE TABLE IF NOT EXISTS registries(id BIGINT NOT NULL,registy_name VARCHAR(255) NOT NULL,registry_type VARCHAR(255) NOT NULL,registry_uri VARCHAR(255) NOT NULL,CONSTRAINT PK_registries PRIMARY KEY (id)); CREATE TABLE IF NOT EXISTS mpacks(id BIGINT NOT NULL,mpack_name VARCHAR(255) NOT NULL,mpack_version VARCHAR(255) NOT NULL,mpack_uri VARCHAR(255),registry_id BIGINT,CONSTRAINT PK_mpacks PRIMARY KEY (id),CONSTRAINT uni_mpack_name_version UNIQUE(mpack_name, mpack_version),CONSTRAINT FK_registries FOREIGN KEY (registry_id) REFERENCES registries(id)); ALTER TABLE stack ADD COLUMN mpack_id BIGINT NULL;ALTER TABLE stack ADD CONSTRAINT FK_mpacks FOREIGN KEY (mpack_id) REFERENCES mpacks(id); ALTER TABLE upgrade ADD COLUMN upgrade_package_stack VARCHAR(255) NOT NULL;xxxxxxxxxx-- 0) Use Ambari schemaUSE ambari; -- 1) (Optional) Confirm current FK names on affected tablesSELECT TABLE_NAME, CONSTRAINT_NAMEFROM information_schema.TABLE_CONSTRAINTSWHERE CONSTRAINT_SCHEMA = DATABASE() AND CONSTRAINT_TYPE = 'FOREIGN KEY' AND TABLE_NAME IN ('servicecomponentdesiredstate', 'hostcomponentdesiredstate', 'hostcomponentstate')ORDER BY TABLE_NAME, CONSTRAINT_NAME; -- 2) Drop child FKs first, then parent FKALTER TABLE hostcomponentdesiredstate DROP FOREIGN KEY hstcmpnntdesiredstatecmpnntnme;ALTER TABLE hostcomponentstate DROP FOREIGN KEY hstcomponentstatecomponentname;ALTER TABLE servicecomponentdesiredstate DROP FOREIGN KEY srvccmponentdesiredstatesrvcnm; -- 3) Alter service_name columns to VARCHAR(255)ALTER TABLE servicecomponentdesiredstate MODIFY COLUMN service_name VARCHAR(255) NOT NULL;ALTER TABLE hostcomponentdesiredstate MODIFY COLUMN service_name VARCHAR(255) NOT NULL;ALTER TABLE hostcomponentstate MODIFY COLUMN service_name VARCHAR(255) NOT NULL; -- 4) Recreate FKsALTER TABLE servicecomponentdesiredstate ADD CONSTRAINT srvccmponentdesiredstatesrvcnm FOREIGN KEY (service_name, cluster_id) REFERENCES clusterservices (service_name, cluster_id); ALTER TABLE hostcomponentdesiredstate ADD CONSTRAINT hstcmpnntdesiredstatecmpnntnme FOREIGN KEY (component_name, service_name, cluster_id) REFERENCES servicecomponentdesiredstate (component_name, service_name, cluster_id); ALTER TABLE hostcomponentstate ADD CONSTRAINT hstcomponentstatecomponentname FOREIGN KEY (component_name, service_name, cluster_id) REFERENCES servicecomponentdesiredstate (component_name, service_name, cluster_id); -- 5) Verify column definitionsSHOW COLUMNS FROM servicecomponentdesiredstate LIKE 'service_name';SHOW COLUMNS FROM hostcomponentdesiredstate LIKE 'service_name';SHOW COLUMNS FROM hostcomponentstate LIKE 'service_name';Oracle 19C
xxxxxxxxxxsqlplus / as sysdba ALTER SESSION SET CONTAINER=<DB Name>; ALTER SESSION SET CURRENT_SCHEMA = <UserName>; CREATE TABLE registries( id NUMBER(19) NOT NULL, registy_name VARCHAR2(255) NOT NULL, registry_type VARCHAR2(255) NOT NULL, registry_uri VARCHAR2(255) NOT NULL, CONSTRAINT PK_registries PRIMARY KEY (id)); CREATE TABLE mpacks( id NUMBER(19) NOT NULL, mpack_name VARCHAR2(255) NOT NULL, mpack_version VARCHAR2(255) NOT NULL, mpack_uri VARCHAR2(255), registry_id NUMBER(19), CONSTRAINT PK_mpacks PRIMARY KEY (id), CONSTRAINT uni_mpack_name_version UNIQUE(mpack_name, mpack_version), CONSTRAINT FK_registries FOREIGN KEY (registry_id) REFERENCES registries(id)); ALTER TABLE stack ADD mpack_id NUMBER(19) NULL;ALTER TABLE stack ADD CONSTRAINT FK_mpacks FOREIGN KEY (mpack_id) REFERENCES mpacks(id);Postgres
Ambari Server Upgrade Fails on PostgreSQL Due to Missing ambari Schema
Summary:
When Ambari is backed by PostgreSQL and ambari-server upgrade is executed, the Java SchemaUpgradeHelper process fails immediately because it attempts to set the PostgreSQL search path to a schema named ambari, but that schema does not exist. All Ambari tables are instead located in the default public schema.
This issue is specific to PostgreSQL schema behavior.
Other database platforms do not encounter this problem because:
- MySQL treats the database and schema as the same object, so no separate schema creation is required.
- Oracle automatically creates a schema for each database user.
- PostgreSQL requires schemas to be explicitly created. Unless a schema is manually created beforehand, tables are created in the default
publicschema.
Error:
The following error appears during ambari-server upgrade execution and in /var/log/ambari-server/ambari-server.log:
xxxxxxxxxxSchemaUpgradeHelper:483 - Exception occurred during upgrade org.apache.ambari.server.AmbariException: ERROR: schema "ambari" does not exist Caused by: org.postgresql.util.PSQLException: ERROR: schema "ambari" does not exist at AbstractUpgradeCatalog.changePostgresSearchPath(AbstractUpgradeCatalog.java:397)A secondary Python-level error may also appear:
xxxxxxxxxxambari_simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)This is not the root cause. The Java upgrade process terminated unexpectedly and returned empty output, causing the Python wrapper to fail while attempting to parse the response.
Root Cause
The Ambari ambari.properties contains the property:
xxxxxxxxxxserver.jdbc.postgres.schema=ambariThis causes AbstractUpgradeCatalog.changePostgresSearchPath() to execute:
xxxxxxxxxxSET search_path TO ambari;The upgrade fails because the ambari schema does not exist. The Ambari PostgreSQL DDL was run before the schema was created, so PostgreSQL placed all Ambari tables in the default public schema.
The database should have been initialized with:
xxxxxxxxxxCREATE SCHEMA ambari AUTHORIZATION ambari;ALTER ROLE ambari SET search_path TO ambari, public;Diagnosis
- Read the connection details from ambari.properties
xxxxxxxxxxDB_HOST=$(grep server.jdbc.hostname /etc/ambari-server/conf/ambari.properties | cut -d= -f2)DB_PASS=$(cat /etc/ambari-server/conf/password.dat) echo "Host: $DB_HOST"- Check which schemas exist
xxxxxxxxxxPGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c "\dn" -- Expected (broken state):-- Name | Owner-- --------+--------------------- public | pg_database_owner -- Expected (correct state after fix):-- Name | Owner-- --------+---------- ambari | ambari-- public | pg_database_owner- Check where the tables are located
xxxxxxxxxxPGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c " SELECT table_schema, count(*) AS table_count FROM information_schema.tables WHERE table_schema NOT IN ('information_schema','pg_catalog') GROUP BY table_schema;" ##-- Broken state: public | 113 ##-- Fixed state: ambari | 113Fix Procedure
Before making changes, take a full backup:
pg_dump -U ambari -h $DB_HOST ambari > /tmp/ambari_backup_ $(date +%Y%m%d).sql
- Set the DB connection variables
Set connection variables and verify connectivity:
xxxxxxxxxxDB_HOST=$(grep server.jdbc.hostname /etc/ambari-server/conf/ambari.properties | cut -d= -f2) DB_PASS=$(cat /etc/ambari-server/conf/password.dat) # Verify connection PGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c "SELECT 1;"- Create the ambari schema and move all objects
Create the schema, move tables and sequences, and set the search path:
xxxxxxxxxxPGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 << 'EOF' -- Create schema CREATE SCHEMA IF NOT EXISTS ambari AUTHORIZATION ambari; -- Move all tables from public to ambari DO $$ DECLARE r RECORD; BEGIN FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'public' LOOP EXECUTE 'ALTER TABLE public.' || quote_ident(r.tablename) || ' SET SCHEMA ambari'; RAISE NOTICE 'Moved table: %', r.tablename; END LOOP; END; $$; -- Move all sequences from public to ambari DO $$ DECLARE r RECORD; BEGIN FOR r IN SELECT sequence_name FROM information_schema.sequences WHERE sequence_schema = 'public' LOOP EXECUTE 'ALTER SEQUENCE public.' || quote_ident(r.sequence_name) || ' SET SCHEMA ambari'; RAISE NOTICE 'Moved sequence: %', r.sequence_name; END LOOP; END; $$; -- Set permanent search path for ambari role ALTER ROLE ambari SET search_path TO ambari, public; EOF- Verify
Verify all tables are now in the ambari schema:
xxxxxxxxxx# All tables should now be in ambari schema PGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c " SELECT table_schema, count(*) AS table_count FROM information_schema.tables WHERE table_schema NOT IN ('information_schema','pg_catalog') GROUP BY table_schema;" # Expected output: # table_schema | table_count# --------------+-------------# ambari | 113 # Verify search path is set PGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c "SHOW search_path;" # Expected: ambari, public- Retry the upgrade
xxxxxxxxxxambari-server upgrade -vWhy does this only affect PostgreSQL?
| Database | Schema behavior | Ambari impact |
|---|---|---|
| MySQL / MariaDB | Database and schema are the same concept | No issue |
| Oracle 19c | The schema is automatically created per user | No issue |
| PostgreSQL | The schema must be created manually | Upgrade fails if the schema is missing |
Prevention
For new Ambari PostgreSQL deployments, create the schema before running the Ambari DDL:
xxxxxxxxxxCREATE DATABASE ambari; CREATE USER ambari WITH PASSWORD 'your_password'; \c ambari CREATE SCHEMA ambari AUTHORIZATION ambari; ALTER ROLE ambari SET search_path TO ambari, public;Then run the Ambari DDL:
xxxxxxxxxxpsql -U ambari -d ambari -f Ambari-DDL-Postgres-CREATE.sqlCorrect Setup Checklist:
Before running Ambari PostgreSQL DDL, confirm that:
- Database
ambariexists. - User
ambariexists. password.datcontains the correct password.- Schema
ambariexists. - Schema
ambariis owned by userambari. ALTER ROLE ambari SET search_path TO ambari, publichas been run.- Ambari tables are created in the
ambarischema, notpublic.
Upgrade the Ambari Server Database Schema
xxxxxxxxxxambari-server upgrade -vUpon running the command, the response will be as shown in the screenshot.
Make sure to take the database backup as mentioned earlier and enter 'y' here.
Start the Ambari Server
Once started, you are able to access the Ambari UI using the same url as it was earlier.
xxxxxxxxxxambari-server startAccess the Ambari UI
Use your browser to access the Ambari UI.
Check the Upgraded Version
To verify the upgraded Ambari version, navigate to Admin → About in the top-right corner of the Ambari UI.
All the services also must be working normally without any downtime.
Known Issues
Pip dependency missing
Error:
xxxxxxxxxx[root@ce19 ~]# ambari-server upgradeUsing python /usr/bin/python3Upgrading ambari-serverTraceback (most recent call last): File "/usr/sbin/ambari-server.py", line 28, in <module> from ambari_commons.exceptions import FatalException, NonFatalException File "/usr/lib/ambari-server/lib/ambari_commons/init.py", line 21, in <module> from ambari_commons.os_check import OSCheck, OSConst File "/usr/lib/ambari-server/lib/ambari_commons/os_check.py", line 25, in <module> import distroModuleNotFoundError: No module named 'distro'Workaround:
xxxxxxxxxxpip3 install distroIf still hitting the distro module not found error, follow the below steps:
Step 1:
- Download the
get-pip.pyscript:
xxxxxxxxxxcurl https://bootstrap.pypa.io/get-pip.py -o get-pip.py- Install
pipfor Python 3.11:
xxxxxxxxxx/usr/bin/python3.11 get-pip.pyStep 2: Install the distro module using pip3.11
Once `pip` is installed for Python 3.11, you can now use it to install the missing `distro` module:
xxxxxxxxxx/usr/bin/python3.11 -m pip install distroStep 3: Verify the installation.
After installing the distro module, verify that it’s correctly installed:
xxxxxxxxxx/usr/bin/python3.11 -c "import distro"The ambari-server upgrade -v upgrade is successfully done.
Mpacks uninstallation not done
xxxxxxxxxxTraceback (most recent call last): File "/usr/lib/ambari-server/lib/ambari_simplejson/scanner.py", line 49, in _scan_once nextchar = string[idx]IndexError: string index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 1083, in <module> mainBody() File "/usr/sbin/ambari-server.py", line 1053, in mainBody main(options, args, parser) File "/usr/sbin/ambari-server.py", line 999, in main action_obj.execute() File "/usr/sbin/ambari-server.py", line 78, in execute self.fn(*self.args, **self.kwargs) File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 262, in upgrade retcode = run_schema_upgrade(args) File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 162, in run_schema_upgrade upgrade_response = json.loads(stdout) File "/usr/lib/ambari-server/lib/ambari_simplejson/init.py", line 538, in loads return _default_decoder.decode(s) File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 378, in decode obj, end = self.raw_decode(s) File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 408, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) File "/usr/lib/ambari-server/lib/ambari_simplejson/scanner.py", line 93, in scan_once return _scan_once(string, idx) File "/usr/lib/ambari-server/lib/ambari_simplejson/scanner.py", line 51, in _scan_once raise JSONDecodeError(errmsg, string, idx)ambari_simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)Workaround: Uninstall all the ambari-mpacks.
xxxxxxxxxxambari-server uninstall-mpack --mpack-name=<installed mpack>