Pivotal HDB 2.0.1 Release Notes

Supported Platforms

The supported platform for running for Pivotal HDB 2.0 comprises:

Each Pivotal HDB host machine must also meet the Apache HAWQ (Incubating) system requirements. See Apache HAWQ System Requirements for more information.

Product Support Matrix

The following table summarizes Pivotal HDB product support for current and previous versions of HDB, Hadoop, HAWQ, Ambari, and operating systems.

Pivotal HDB Version HDP Version Requirement (Pivotal HDP and Hortonworks HDP) Ambari Version Requirement HAWQ Ambari Plug-in Requirement MADlib Version Requirement RHEL/CentOS Version Requirement SuSE Version Requirement
2.0.1.0 2.4.0, 2.4.2 2.2.2, 2.4 2.0.1 1.9, 1.9.1 6.4+ (64-bit) n/a
2.0.0.0 2.3.4, 2.4.0 2.2.2 2.0.0 1.9, 1.9.1 6.4+ (64-bit) n/a
1.3.1.1 2.2.6 2.0.x 1.3.1 1.7.1, 1.8, 1.9, 1.9.1 6.4+ SLES 11 SP3
1.3.1.0 2.2.6 2.0.x 1.3.1 1.7.1, 1.8, 1.9, 1.9.1 6.4+ SLES 11 SP3
1.3.0.3 2.2.4.2 1.7 1.2 1.7.1, 1.8, 1.9, 1.9.1 6.4+ SLES 11 SP3
1.3.0.2 2.2.4.2 1.7 1.2 1.7.1, 1.8, 1.9, 1.9.1 6.4+ SLES 11 SP3
1.3.0.1 2.2.4.2 1.7 1.1 1.7.1, 1.8, 1.9, 1.9.1 6.4+ n/a
1.3.0.0 n/a n/a n/a 1.7.1, 1.8, 1.9, 1.9.1 n/a n/a

Note: RHEL/CentOS 7 is not supported.

Note: If you are using Ambari 2.4.0 or 2.4.1 and you want to install both HDP and HAWQ at the same time, see Installing HDP and HDB with Ambari 2.4.0 or 2.4.1 before you begin.

Procedural Language Support Matrix

The following table summarizes component version support for Procedural Languages available in Pivotal HDB 2.x. The versions listed have been tested with HDB. Higher versions may be compatible. Please test higher versions thoroughly in your non-production environments before deploying to production.

Pivotal HDB Version PL/Java Java Version Requirement PL/R R Version Requirement PL/Perl Perl Version Requirement PL/Python Python Version Requirement
2.0.1.0 1.7 3.3.1 5.10.1 2.6.2
2.0.0.0 1.6, 1.7 3.1.0 5.10.1 2.6.2

AWS Support Requirements

Pivotal HDB is supported on Amazon Web Services (AWS) servers using either Amazon block level Instance store (Amazon uses the volume names ephemeral[0-23]) or Amazon Elastic Block Store (Amazon EBS) storage. Use long-running EC2 instances with these for long-running HAWQ instances, as Spot instances can be interrupted. If using Spot instances, minimize risk of data loss by loading from and exporting to external storage.

Pivotal HDB 2.0.1 Features and Changes

Pivotal HDB 2.0.1 is based on Apache HAWQ (Incubating), and includes the following new features and changes in behavior as compared to Pivotal HDB 2.0:

  • Quicklz compression has been removed. See HDB 2.0 to 2.0.1 Migration for related migration details.

    Note: MADlib uses Quicklz compression by default. In order to use MADlib with HDB 2.0.1, you must execute a script during the MADlib installation to remove Quicklz compression. See MADlib Compression below.

  • Snappy compression is now supported for append-only tables.

  • The hawq register command has been added to register existing data files or folders as HAWQ internal tables. This enables HAWQ to directly read the files instead of loading or copying them. HAWQ can also leverage the benefits of HAWQ internal table processing for transaction support and higher performance. Typical usage scenarios for hawq register include:

    • Integrating external Parquet files that were generated by systems such as Hive or Spark.
    • Importing data from a secondary, backup cluster for data recovery.

    See hawq register.

  • HDB 2.0.1 includes a beta release of the PXF JSON plug-in.

  • The Pivotal HDB product tarball names have changed, as has the procedure for installing the tarballs to your yum repository. See Installing HAWQ Using Ambari.

  • The Ambari plug-in for HAWQ is no longer a separate download, but now included in the HDB product tarball.

  • PL/Perl is now an installable procedural language extension.

  • PL/Java, PL/Perl, and pgcrypto are now embedded in the Pivotal HDB installation package. The gppkg command is no longer used to install these packages and is deprecated. The .gppkg package format is no longer used in HDB.

  • PL/R installation has changed:

    • R is no longer included in the PL/R installation package. It will be installed from the EPEL (Extra Packages for Enterprise Linux) repo as a dependency when installing PL/R. When installed in this manner, R components are located in system locations (i.e. /usr/lib64/R) and not in $GPHOME/ext.
    • PL/R is now installed via an .rpm package.

Ambari 2.4 Features and Changes

Ambari 2.4 includes the following features and changes related to installing and managing HAWQ. See also the Ambari 2.4 Release Notes for information about general features and changes introduced with Ambari 2.4.

Apache Jira Description
AMBARI-16074 Added a new PXF widget to the Ambari dashboard.
AMBARI-17473 Exposed additional hawq-site.xml configuration parameters in the Ambari UI.
AMBARI-17719 Ambari now automatically sets many of the required HDFS properties during HAWQ installation.
AMBARI-16745 Ambari calculates the recommended value for hawq_rm_memory_limit_perseg during HAWQ installation.
AMBARI-16992 Ambari calculates the recommended value for hawq_rm_nvcore_limit_perseg during HAWQ installation.
AMBARI-18014 Added the new pxf-json profile in pxf-profiles.xml to support the JSON connector.
AMBARI-17024 Updated the name of the PXF component to PXF Agent.
AMBARI-16419 The PXF component is co-located with the NameNode even if the NameNode is moved to another host.
AMBARI-17937 Ambari creates a new gpadmin database during HAWQ installation or initialization.
AMBARI-16917 Ambari uses the postgres database instead of the template1 database during service checks for HAWQ.
AMBARI-16827 Ambari exposes the vm.overcommit_ratio parameter in from hawq-sysctl-env on the HAWQ config page.
AMBARI-16646 Ambari uses the recommended value for vm.overcommit_memory during HAWQ installation.
AMBARI-16624 Ambari provides a checkbox on the HAWQ config page for setting hawq_ssh_exkeys.
AMBARI-17023 Ambari displays or hides relevant properties in the UI based on the selected mode for Resource Management.
AMBARI-17070 Changed the default HAWQ DFS Url from hawq_default to hawq_data.
AMBARI-16885 Changed the default location of HAWQ temporary directories to avoid using /tmp.
AMBARI-16694 Removed unused configuration parameters from hawq-site.xml.

Installing HDP and HDB with Ambari 2.4.0 or 2.4.1

If you are using Ambari 2.4.0 or 2.4.1 and you want to install both HDP and HAWQ at the same time, special care must be taken if you want to install the very latest version of the HDP stack instead of the default version. Follow these steps:

  1. After installing Ambari, start the Cluster Install Wizard and proceed until you reach the Select Version screen.
  2. On the Select Version screen, select HDB-2.4 from the list of available stack versions.
  3. While still on the Select Version screen, copy the Base URL values for the HDP-2.4 and HDP-UTILS-1.1.0.20 repositories that are listed for your operating system. Paste these values into a temporary file; you will need to restore these Base URL values later.
  4. Use the drop-down menu for HDP-2.4 to select the stack option, HDP-2.4 (Default Version Definition). Verify that the hdb-2.0.1.0 and hdb-add-ons-2.0.1.0 repositories now appear in the list of Repositories for your operating system.
  5. To install the very latest version of HDP, replace the Base URL values for the HDP-2.4 and HDP-UTILS-1.1.0.20 repositories with the values you pasted into the text file in Step 3.
  6. Click Next to continue, and finish installing the new HDP cluster.
  7. Install and configure Pivotal HDP as described in Installing HAWQ Using Ambari.

Note: This workaround may not be required with later versions of Ambari 2.4.

HDB 2.0 to HDB 2.0.1 Migration

  • Quicklz append-only table compression is not supported in HDB 2.0.1. If you have existing AO tables compressed with quicklz, you must migrate these tables to another compression type before upgrading. Pivotal recommends loading the data of each such table into either:
    1. A parquet table with snappy compression, or
    2. A new AO table with zlib compression. After upgrading, you can perform similar operations to load the migrated data to an AO table with snappy compression.
  • The HDB 2.0 to 2.0.1 Upgrade guide provides specific details on upgrading your HAWQ installation. Note: If you are upgrading an HDB version prior to 2.0, refer to the HDB 2.0 documentation.

Differences Compared to Apache HAWQ (Incubating)

Pivotal HDB 2.0.1 includes all of the functionality in Apache HAWQ (Incubating), and adds several bug fixes described below.

Resolved Issues

The following HAWQ issues were resolved in HDB 2.0.1 and Ambari 2.4.

Apache Jira Component Summary
AMBARI-16192 Ambari 2.4 The Ambari Restart All action did not ensure that the HAWQ master, segments, and standby master were started in the correct order. This problem has been resolved.
AMBARI-16236 Ambari 2.4 When a HAWQ standby master was stopped, the Ambari UI would disable controls to stop the HAWQ service or to perform a HAWQ service check. This problem was resolved so that the HAWQ service can be stopped and checked even when the standby master is stopped.
AMBARI-16386 Ambari 2.4 Fixed HAWQ Password handling in Ambari UI.
AMBARI-16750 Ambari 2.4 Using Ambari to activate a standby master could fail on a retry of the operation. This problem has been resolved.
AMBARI-17207 Ambari 2.4 Ambari did not export the PGHOST variable before executing HAWQ Master or Standby Master custom commands. This problem has been resolved.
AMBARI-17996 Ambari 2.4 The HAWQ service advisor could show incorrect recommendations in response to some edge cases. This problem has been resolved.
HAWQ-712 Core Added configuration parameter gp_vmem_idle_resource_timeout to do timeout for gang release in idle sessions.
HAWQ-715 libyarn hawq_rm_yarn_address does not take effect if libyarn is in HA mode.
HAWQ-737 Resource Manager The last action time is initialized as created time instead of 0.
HAWQ-738 Core HAWQ does not allocate a query resource twice in a function call through jdbc.
HAWQ-746 n/a addHostNameAlternatives in the HAWQ ssh utility do not use the short hostname.
HAWQ-749 Fault Tolerance Segment registration considers the same IP across nodes.
HAWQ-755 Resource Manager Fixed a CPU/Memory ratio comparison issue.
HAWQ-759 n/a Fixed an issue where a query could not be terminating using pg_cancel_backend or pg_terminate_backend.
HAWQ-767 n/a Made HAWQ code work with the upstream libesmtp.
HAWQ-800 Core, Dispatcher, Query Execution, Resource Manager Fixed an issue where fewer tuples were inserted because data locality information was not refreshed and dispatched in a prepared statement.
HAWQ-812 n/a Fixed an issue where activating a standby master failed after creating a new database.
HAWQ-815 Query Execution Fixed a KeyError that occurred while compiling a PLPython function due to the deletion of a non-existent record from Python global dictionary.
HAWQ-825 n/a Fixed a problem where HAWQ component reload showed a misleading error message on stdout.
HAWQ-835 Core, Query Execution Fixed an issue where HAWQ could not retrieve a tuple from a temp table that was created in function.
HAWQ-839 libyarn Fixed a libyarn coredump that occurred during failover to a standby RM.
HAWQ-845 n/a Parameterized the kerberos principal name for HAWQ.
HAWQ-850 Core Planner supports the refineCachedPlan interface to correctly recalculate resources for PBE (prepare bind execution) and cursor.
HAWQ-853 Standby master Master standby avoid incomplete split operations.
HAWQ-855 Resource Manager The resource manager asks for a cluster report quickly when it detects that YARN is down.
HAWQ-861 Core HAWQ errors out when the bucket number of a result hash table is less than the number of locations for gpfdist.
HAWQ-862 Core Fixed user defined function get_ao_distribution.
HAWQ-871 Command Line Tools hawq check looks for incorrect YARN HA parameters.
HAWQ-872 Command Line Tools hawq check fails if --kerberos is ommitted when kerberos is actually enabled.
HAWQ-880 Command Line Tools Corrected output of hawq stop --reload/-u’`.
HAWQ-887 Core Query continues when the bucket number of the result hash table is larger than the number of segments for the external table.
HAWQ-901 Command Line Tools Added retry to standby master during HAWQ cluster start.
HAWQ-912 Fault Tolerance HAWQ skips temporary directory checking for master/standby.
HAWQ-918 Query Execution Fixed a memtuple forming bug when the null-saved size is larger than 32763 bytes.
HAWQ-940 libyarn Fixed an issue where a Kerberos ticket expired for libyarn operations.
HAWQ-961 n/a HAWQ dispatches session user info instead of BOOTSTRAP_SUPERUSERID, from master to segments.
HAWQ-966 libyarn Refined libyarn logging.
HAWQ-970 libyarn HAWQ provides more accurate information when libyarn meets an exception.
HAWQ-978 n/a Fixed a deadlock in the signal handler which calls asynchronous unsafe functions elog().
HAWQ-979 Resource Manager Resource Broker reconnects Hadoop Yarn when it fails to get a cluster report.
HAWQ-980 n/a HAWQ now handles configuration parameter values that contain spaces.
HAWQ-984 Command Line Tools Performance improvement for hawq config.
HAWQ-990 Parser, Planner Fixed a failure that occurred when a UNION clause was used with a DISTRIBUTED BY clause.
HAWQ-999 Core An error is logged when the file count is not in proportion to the bucket number of a hash table.
HAWQ-1009 n/a Removed the requirement for using the environment value, MASTER_DATA_DIRECTORY.
HAWQ-1030 Core Ported a PostgreSQL spin lock performance enhancement to HAWQ.
HAWQ-1032 Core The bucket number of a newly-added partition is now consistent with the parent hash table.
HAWQ-1077 Storage Fixed an issue where inserting into an AO table with snappy compression could hang.

Known Issues and Limitations

MADlib Compression

Pivotal HDB 2.0.1 is compatible with MADlib 1.9 and 1.9.1. However, you must download and execute a script in order to remove the MADlib Quicklz compression, which is not supported in HDB 2.0.1. Run this script if you are upgrading to HDB 2.0.1, or if you are installing MADlib on HDB 2.0.1.

If you are upgrading an HDB 2.0 system that contains MADlib:

  1. Complete the Pivotal HDB 2.0.1 upgrade procedure as described in Upgrading HDB Components.

  2. Download and unpack the MADlib 1.9.1 binary distribution from the Pivotal HDB Download Page on Pivotal Network.

  3. Execute the remove_compression.sh script in the MADlib 1.9.1 distribution, providing the path to your existing MADlib installation:

    $ remove_compression.sh --prefix <madlib-installation-path>
    

    Note: If you do not include the --prefix option, the script uses the location ${GPHOME}/madlib.

For new MADlib installations, complete these steps after you install Pivotal HDB 2.0.1:

  1. Download and unpack the MADlib 1.9.1 binary distribution from the Pivotal HDB Download Page on Pivotal Network.

  2. Install the MADlib .gppkg file:

    $ gppkg -i <path-to>/madlib-ossv1.9.1_pv1.9.6_hawq2.0-rhel5-x86_64.gppkg
    
  3. Execute the remove_compression.sh script, optionally providing the MADlib installation path:

    $ remove_compression.sh --prefix <madlib-installation-path>
    

    Note: If you do not include the --prefix option, the script uses the location ${GPHOME}/madlib.

  4. Continue installing MADlib using the madpack install command as described in the MADlib Installation Guide. For example:

    $ madpack –p hawq install
    

Management Tools

  • The hawq extract utility includes the Bucketnum attribute for randomly-distributed tables, even though this attribute applies only to hash-distributed tables. Remove the attribute definition for any YAML files that you generate for a randomly-distributed table.
  • If you use the hawq register utility with the --force option and provide a YAML configuration file, then:

    • File definitions in the YAML file for any existing table files must appear before the definitions for newly-added table files, and
    • File definitions in the YAML file for any existing table files must appear in numerical order.

    If the YAML file does not observe both of the above restrictions, then the hawq register command completes but queries against the registered table result in the error:

    ERROR:  hdfs file length does not equal to metadata logic length! (cdbdatalocality.c:1102)
    

Operating System

  • Some Linux kernel versions between 2.6.32 to 4.3.3 (not including 2.6.32 and 4.3.3) have a bug that could introduce a getaddrinfo() function hang. To avoid this issue, upgrade the kernel to version 4.3.3+.

PXF

  • PXF in a Kerberos-secured cluster requires YARN to be installed due to a dependency on YARN libraries.
  • In order for PXF to interoperate with HBase, you must manually add the PXF HBase JAR file to the HBase classpath after installation. See Post-Install Procedure for Hive and HBase on HDP.
  • HAWQ-974 - When using certain PXF profiles to query against larger files stored in HDFS, users may occasionally experience hanging or query timeout. This is a known issue that will be improved in the next HDB release.
  • After upgrading from HDB version 2.0.0, HCatalog access through PXF may fail with the following error:

    postgres=# \d hcatalog.default.hive_table
    ERROR:  function return row and query-specified return row do not match
    DETAIL:  Returned row contains 5 attributes, but query expects 4.
    

    To restore HCatalog access, you must update the PXF pxf_get_item_fields() function definition. Perform this procedure only if you upgraded from HDB 2.0.0.

    1. Log in the HAWQ master node and start the psql subsystem:

      $ ssh gpadmin@master
      gpadmin@master$ psql -d postgres
      
    2. List all but the hcatalog and template0 databases:

      postgres=# SELECT datname FROM pg_database WHERE NOT datname IN ('hcatalog', 'template0');
      
    3. Run the following commands on each database identified in Step 2 to update the pxf_get_item_fields() function definition:

      postgres=# CONNECT <database>;
      postgres=# SET allow_system_table_mods = 'dml';
      postgres=# UPDATE pg_proc
                   SET proallargtypes = '{25,25,25,25,25,25,25}',  proargmodes = '{i,i,o,o,o,o,o}',  proargnames = '{profile,pattern,path,itemname,fieldname,fieldtype,sourcefieldtype}'
                 WHERE proname = 'pxf_get_item_fields';
      
    4. Reset your psql session:

      postgres=# RESET allow_system_table_mods;
      

      Note: Use the allow_system_table_mods server configuration parameter and identified SQL commands only in the context of this workaround. They are not otherwise supported.

PL/R

The HAWQ PL/R extension is provided as a separate RPM in the hdb-add-ons-2.0.1.0 repository. When this RPM is installed, a PL/R library and script are placed in an incorrect HAWQ base directory. Perform the following steps on each node in your HAWQ cluster after PL/R RPM installation to copy the files to the correct HAWQ base install directory:

gpadmin@hawqnode$ cd /usr/local/hawq_2_0_1_0-2.0.1.0
gpadmin@hawqnode$ cp share/postgresql/contrib/plr.sql ../hawq/share/postgresql/contrib/
gpadmin@hawqnode$ cp docs/contrib/README.plr ../hawq/docs/contrib/
gpadmin@hawqnode$ cp lib/postgresql/plr.so ../hawq/lib/postgresql/
gpadmin@hawqnode$ cd ../hawq
gpadmin@hawqnode$ chown gpadmin:gpadmin share/postgresql/contrib/plr.sql docs/contrib/README.plr lib/postgresql/plr.so

YARN Integration

  • If you are using YARN mode for HAWQ resource management on Hortonworks HDP 2.3 and the timeline server is configured, you must set yarn.resourcemanager.system-metrics-publisher.enabled to false in yarn-site.xml. Otherwise, HAWQ may fail to register itself to YARN. This is a known problem with YARN as described in YARN-4452. YARN-4452 is fixed in HDP 2.4.

Ambari

  • When installing HAWQ in a Kerberos-secured cluster, the installation process may report a warning/failure in Ambari if the HAWQ configuration for resource management type is switched to YARN mode during installation. The warning is related to HAWQ not being able to register with YARN until the HDFS & YARN services are restarted with new configurations resulting from the HAWQ installation process.
  • The HAWQ standby master will not work after you change the HAWQ master port number. To enable the standby master you must first remove and then re-initialize it. See Removing the HAWQ Standby Master and Activating the HAWQ Standby Master.
  • The Ambari Re-Synchronize HAWQ Standby Master service action fails if there is an active connection to the HAWQ master node. The HAWQ task output shows the error, Active connections. Aborting shutdown... If this occurs, close all active connections and then try the re-synchronize action again.
  • The Ambari Run Service Check action for HAWQ and PXF may not work properly on a secure cluster if PXF is not co-located with the YARN component.
  • In a secured cluster, if you move the YARN Resource Manager to another host you must manually update hadoop.proxyuser.yarn.hosts in the HDFS core-site.xml file to match the new Resource Manager hostname. If you do not perform this step, HAWQ segments fail to get resources from the Resource Manager.
  • The Ambari Stop HAWQ Server (Immediate Mode) service action or hawq stop -M immediate command may not stop all HAWQ master processes in some cases. Several postgres processes owned by the gpadmin user may remain active.
  • Ambari checks whether the hawq_rm_yarn_address and hawq_rm_yarn_scheduler_address values are valid when YARN HA is enabled. In clusters that use YARN HA, these properties are not used and may get out-of-sync with the active Resource Manager. This can leading to false warnings from Ambari if you try to change the property value.
  • Ambari does not support Custom Configuration Groups with HAWQ.
  • If you install HAWQ using Ambari 2.2.2 with the HDP 2.3 stack, before you attempt to upgrade to HDP 2.4 you must use Ambari to change the dfs.allow.truncate property to false. Ambari will display a configuration warning with this setting, but it is required in order to complete the upgrade; choose Proceed Anyway when Ambari warns you about the configured value of dfs.allow.truncate. After you complete the upgrade to HDP 2.4, change the value of dfs.allow.truncate back to true to ensure that HAWQ can operate as intended.
  • A failure in the Ambari Activate Standby Wizard can leave the hawq-site.xml file inconsistent. Re-running the wizard will continue to fail with the message: Error: UnboundLocalError: local variable 'old_standby_host_name' referenced before assignment. As a workaround, exit the Activate Standby Wizard and restart the HAWQ service to push the old configuration to the cluster. Then start the Activate Standby Wizard once again.
  • Certain HAWQ server configuration parameters related to resource enforcement are not active. Modifying the parameters has no effect in HAWQ since the resource enforcement feature is not currently supported. These parameters include hawq_re_cgroup_hierarchy_name, hawq_re_cgroup_mount_point, and hawq_re_cpu_enable. These parameters appear in the Advanced hawq-site configuration section of the Ambari management interface.

Workaround Required after Moving Namenode

If you use the Ambari Move Namenode Wizard to move a Hadoop namenode, the Wizard does not automatically update the HAWQ configuration to reflect the change. This leaves HAWQ in an non-functional state, and will cause HAWQ service checks to fail with an error similar to:


2017-04-19 21:22:59,138 - SQL command executed failed: export PGPORT=5432 && source
/usr/local/hawq/greenplum_path.sh && psql -d template1 -c \\\\\"CREATE  TABLE
ambari_hawq_test (col1 int) DISTRIBUTED RANDOMLY;\\\\\"
Returncode: 1
Stdout:
Stderr: Warning: Permanently added 'ip-10-32-36-168.ore1.vpc.pivotal.io,10.32.36.168'
(RSA) to the list of known hosts.
WARNING:  could not remove relation directory 16385/1/18366: Input/output error
CONTEXT:  Dropping file-system object -- Relation Directory: '16385/1/18366'
ERROR:  could not create relation directory
hdfs://ip-10-32-36-168.ore1.vpc.pivotal.io:8020/hawq_default/16385/1/18366: Input/output error

2016-04-19 21:22:59,139 - SERVICE CHECK FAILED: HAWQ was not able to write and query from a table 2016-04-19 21:23:02,608 - ** FAILURE **: Service check failed 1 of 3 checks stdout: /var/lib/ambari-agent/data/output-281.txt

To work around this problem, perform one of the following procedures after you complete the Move Namenode Wizard.

Workaround for Non-HA NameNode Clusters:
  1. Perform an HDFS service check to ensure that HDFS is running properly after you moved the NameNode.
  2. Use the Ambari config.sh utility to update hawq_dfs_url to the new NameNode address. See the Modify configurations on the Ambari Wiki for more information. For example:

    $ cd /var/lib/ambari-server/resources/scripts/
    $ ./configs.sh set {ambari_server_host} {clustername} hawq-site
    $ hawq_dfs_url {new_namenode_address}:{port}/hawq_default
    
  3. Restart the HAWQ configuration to apply the configuration change.

  4. Use ssh to log into a HAWQ node and run the checkpoint command:

    $ psql -d template1 -c "checkpoint"
    
  5. Stop the HAWQ service.

  6. The master data directory is identified in the $GPHOME/etc/hawq-site.xml file hawq_master_directory property value. Copy the master data directory to a backup location:

    $ export MDATA_DIR=/value/from/hawqsite
    $ cp -r $MDATA_DIR /catalog/backup/location
    
  7. Execute this query to display all available HAWQ filespaces:

  8. SELECT fsname, fsedbid, fselocation FROM pg_filespace as sp,
    pg_filespace_entry as entry, pg_filesystem as fs WHERE sp.fsfsys = fs.oid
    and fs.fsysname = 'hdfs' and sp.oid = entry.fsefsoid ORDER BY
    entry.fsedbid;
    
          fsname | fsedbid | fselocation
    -------------+---------+------------------------------------------------
    cdbfast_fs_a | 0       | hdfs://hdfs-cluster/hawq//cdbfast_fs_a
    dfs_system   | 0       | hdfs://test5:9000/hawq/hawq-1459499690
    (2 rows)
    
  9. Execute the hawq filespace command on each filespace that was returned by the previous query. For example:

    $ hawq filespace --movefilespace dfs_system --location=hdfs://new_namenode:port/hawq/hawq-1459499690
    $ hawq filespace --movefilespace cdbfast_fs_a --location=hdfs://new_namenode:port/hawq//cdbfast_fs_a
    
  10. If your cluster uses a HAWQ standby master, reinitialize the standby master in Ambari using the Remove Standby Wizard followed by the Add Standby Wizard.

  11. Start the HAWQ Service.

  12. Run a HAWQ service check to ensure that all tests pass.

Workaround for HA NameNode Clusters:
  1. Perform an HDFS service check to ensure that HDFS is running properly after you moved the NameNode.
  2. Use Ambari to expand Custom hdfs-client in the HAWQ Configs tab, then update the dfs.namenode. properties to match the current NameNode configuration.
  3. Restart the HAWQ configuration to apply the configuration change.
  4. Run a HAWQ service check to ensure that all tests pass.