Upgrading to HDB 2.3.x
The HDB 2.3 upgrade procedure is described below. This procedure describes how to update an existing HDB 2.1.x or 2.2.x installation, upgrading Ambari, HDP, the Ambari HAWQ Plug-in, HDB, and PXF components to supported versions. For example, use this procedure to upgrade from HDB 2.1.1 to HDB 2.3.0, or to any later maintenance release version.
This procedure uses HDB 2.earlier to refer to your currently-installed HDB version and HDB 2.3.current to refer to the release to which you are upgrading.
Note: Direct upgrades from HDB 2.0.x are not supported.
Pre- and post- upgrade component versions are identified in the table below.
|Component||Pre-Upgrade Version||Post-Upgrade Version|
|Ambari HAWQ Plug-in||2.1.0, 2.1.1, 2.1.2||2.3.0|
|HDB||2.1.0, 2.1.1, 2.1.2, 2.2.0||2.3.0|
|PXF||3.1.0, 3.1.1, 3.2.0, 3.2.1||3.3.0|
This upgrade procedure must be performed in the exact order presented:
Part 1: Upgrade the Stack
Part 2: Upgrade HAWQ
- Step 3: Add New HDB Software Repositories
- Step 4: Upgrade Ambari HAWQ Plug-in
- Step 5: Back up Existing HDB Installation
- Step 6: Upgrade HDB Software
- Step 7: Upgrade PXF
- Step 8: Ambari Post-Upgrade Actions for PXF
- Step 9: HDB Post-Upgrade Actions
If you use Ambari to manage your cluster and your Ambari version is older than 2.4.2:
Download Ambari 2.4.2.
Follow the instructions in Upgrading Ambari to 2.4.2 in the Hortonworks Pivotal HDP documentation.
Perform the following steps to upgrade HDP in your Ambari-managed installation:
Start all services by selecting Start All from the Ambari service pane Actions drop down.
Perform service checks by clicking on each service in the service pane and selecting Run Service Check from the Service Actions drop down.
Navigate to HDFS service Configs > Advanced > Custom hdfs-site and set the
false. (Alternatively, you can enter this property in the HDFS Configs search bar.)
Do not yet restart HDFS and dependent services.
Stop the HAWQ service due to the incompatible
dfs.allow.truncatesetting. First, click on the service in the left pane and then select the Stop action from the Service Action dropdown.
Restart HDFS and dependent services. Do not restart HAWQ. Perform service checks on these services.
Upgrade HDP to version 2.5.3. Follow the instructions in Upgrading HDP in the Hortonworks Pivotal HDP documentation.
Navigate back to HDFS service Configs > Advanced > Custom hdfs-site and set the
dfs.allow.truncateproperty back to
true. Restart the HDFS and any dependent services.
Verify all services are running (green).
Follow the instructions in Setting up HDB Repositories to set up local
yum HDB repositories on the
repo-node you choose to host the HDB software. This system must be accessible to all nodes in your HAWQ cluster.
After you set up the repos, each HAWQ host can obtain the HDB software from the
repo-node HDB repositories.
If you use Ambari to manage your cluster, perform the following steps to upgrade the Ambari HAWQ plug-in software.
Note: The Ambari plug-in for HAWQ is included in the HDB software installation package.
Log in to the Ambari server system as the
$ ssh root@<ambari-server>
Install the HAWQ Ambari plug-in. The RPM package will be installed from the HDB repository set up on
repo-nodein the previous section.
root@ambari-server$ yum install hawq-ambari-plugin
Scripts and template files are installed to
Ensure that the Ambari server is running. If not, start it:
root@ambari-server$ /usr/sbin/ambari-server status root@ambari-server$ /usr/sbin/ambari-server start
add-hawq.pyAmbari HAWQ set-up script, providing the HDB repository URL (replace
ywith the appropriate HDB maintenance version number) and HDP stack version; you will be prompted for your Ambari admin credentials if you do not provide
--passwordoptions. For example:
root@ambari-server$ /var/lib/hawq/add-hawq.py -s HDP-2.5 --hawqrepo http://<repo-node>/hdb-2.3.y.0
--hawqrepooption is not required if you have set up the HDB repositories on your Ambari server host.
Restart the Ambari server:
root@ambari-server$ /usr/sbin/ambari-server restart
Stop the PXF and HAWQ services. First, click on the service in the left pane and then select the Stop action from the Service Action dropdown.
Unless directed to by these instructions, do not invoke any service actions in the Ambari management console until the upgrade procedure is complete.
Backing up your HDB installation before a minor upgrade is recommended.
Log in to the HDB 2.earlier HAWQ master node and set up the environment. For example, in an HDB 2.1.0 installation:
$ ssh gpadmin@<master> gpadmin@master$ . /usr/local/hawq/greenplum_path.sh gpadmin@master$ hawq version HAWQ version 22.214.171.124 build 2490
(Optional) Clean out old server log files from your master and segment data directories. Cleaning out old log files is not required, but will reduce the size of the HAWQ files that are backed up.
Follow the instructions in Backing Up and Restoring HAWQ to back up your existing databases.
Backing up an HDB 2.earlier binary installation is not necessary; HDB version 2.0.1 and newer support upgrade in place.
Stop the PXF Agent on each node if you have not done this from the Ambari UI in a previous step:
hawq-node$ sudo service pxf-service stop
Stop the HAWQ cluster if you have not done this from the Ambari UI in a previous step:
gpadmin@master$ hawq stop cluster
Back up data directories on the HAWQ master, standby master, and all segment nodes. You can determine the location of the data directories by examining the
hawq_segment_directoryconfiguration values in
For example, to copy the master node data directory to the local system:
master$ mkdir -p /save/hawq-backup/data master$ cp -r /data/hawq/master /save/hawq-backup/data/
On a segment node:
segment$ cp -r /data/hawq/segment /save/hawq-backup/data/
Copy or preserve any additional folders or files (such as backup folders) that you have added in the HAWQ data directories.
Perform the following steps on the HAWQ master, HAWQ standby master, and on each HAWQ segment node. You may choose to update your cluster in parallel using your choice of tool such as
Log in to the HAWQ node as the
$ ssh root@<hawq-node>
Install the HDB software on the node:
root@hawq-node$ yum upgrade hawq
The HAWQ software is installed to
ywith the appropriate HDB maintenance version number).
Create a symbolic link to the new HAWQ installation directory:
root@hawq-node$ ln -s /usr/local/hawq_2_3_y_0 /usr/local/hawq
If you do not use Ambari to manage your HAWQ cluster, copy the HDB 2.earlier
etc/configuration directory to the new HDB 2.3.y installation. For example, for HDB 2.1.2:
hawq-node$ cp -rf /usr/local/hawq_2_1_2_0/etc /usr/local/hawq/ hawq-node$ chown -R gpadmin:gpadmin /usr/local/hawq
Perform steps 1-4. above on the HAWQ masters and each segment node in your HAWQ cluster.
Restart the HAWQ cluster:
a. If you use Ambari to manage your cluster, start the HAWQ service from the Ambari console.
b. If you do not use Ambari, log in to the HAWQ master node and restart the HAWQ cluster:
$ ssh gpadmin@<master> gpadmin@master$ . /usr/local/hawq/greenplum_path.sh gpadmin@master$ hawq start cluster
Display the HAWQ version to verify the HDB software upgrade. For example:
gpadmin@master$ hawq version HAWQ version 2.3.y.0 build NNNN
Perform the following steps on each PXF node in your HAWQ cluster to upgrade the PXF software:
Log in to the PXF node as the
$ ssh root@<pxf-node>
Stop the PXF service on each node if you have not done this from the Ambari UI in a previous step:
root@pxf-node$ service pxf-service stop
Back up the PXF 3.earlier configuration files found in the
/etc/pxf/conf/directory. For example:
root@pxf-node$ mkdir -p /save/pxf-conf root@pxf-node$ cp /etc/pxf/conf/* /save/pxf-conf/
Uninstall the PXF 3.earlier RPMs:
root@pxf-node$ yum erase -y pxf*
Remove the PXF service instance:
root@pxf-node$ rm -rf /var/pxf /etc/pxf*
Upgrade the PXF software to version 3.3.current:
root@pxf-node$ yum install -y pxf
This command installs the PXF agent software and all PXF plug-ins: HDFS, Hive, HBase, JDBC, JSON. The PXF software is installed to
/etc/pxf-3.3.y.0, with links created from
If you updated any PXF configuration files in your original installation, propagate these changes to the new PXF installation. Specifically, manual changes to
pxf-public.classpathfiles will need to be ported over.
Initialize the PXF service; ensure that you have set
root@pxf-node$ service pxf-service init
After upgrading PXF on each node, start the PXF service:
a. If you use Ambari to manage your cluster, start the PXF service via the Ambari console.
b. If you do not use Ambari, start the PXF service from the command line on each node:
root@pxf-node$ service pxf-service start
If you manage your HAWQ cluster with Ambari and use PXF, follow the instructions in Ambari Post-Install Procedure for PXF to manually update certain PXF profile configurations.
After you have verified your HDB 2.3.current installation was upgraded successfully and your HAWQ cluster has been running without issues, uninstall HDB 2.earlier by performing the following operation on each node in your HAWQ cluster. For example, to uninstall HDB 2.1.1:
root@hawq-node$ yum erase hawq_2_1_1_0