WANdisco Fusion Plugin Guide

1. Welcome

Welcome to the User Guide for the Fusion Plugin for Live Ranger, version 2.1.

Apache Ranger is a framework to manage data security in Hadoop deployments. It provides centralized security administration, fine-grained authorization and centralized auditing within a single cluster. Use the Fusion Plugin for Live Ranger to extend the capabilities of WANdisco Fusion to Apache Ranger across multiple Hadoop environments, and keep your security policies consistent.

1.1. Product overview

WANdisco Fusion gives you LiveData: consistent data everywhere, spanning platforms and locations, even for changing data at petabyte scale. Business critical data is guaranteed consistent, always available, and accessible from anywhere.

The Fusion Plugin for Live Ranger extends WANdisco Fusion to information managed and used by Apache Ranger. Use it to keep your security policies consistent among Hadoop deployments with WANdisco Fusion. Key features include:

Apache Ranger policy replication
Coordination of activities that modify policy definitions, including those performed via the Apache Ranger REST API, or from its administrative interface in a browser
Integration with WANdisco Fusion

1.2. Documentation guide

This guide contains the following:

Welcome: This chapter introduces this user guide and provides help with how to use it.
Release Notes: Details the latest software release, covering new features, fixes and known issues to be aware of.
Concepts: Explains how Fusion Plugin for Live Ranger through WANdisco Fusion uses WANdisco’s LiveData platform.
Installation: Covers the steps required to install and set up Fusion Plugin for Live Ranger into a WANdisco Fusion deployment.
Operation: Describes the steps required to run, reconfigure and troubleshoot Fusion Plugin for Live Ranger.
Reference: Additional Fusion Plugin for Live Ranger documentation, including documentation for the available REST API.

1.2.1. Symbols in the documentation

In the guide we highlight types of information using the following call outs:

The alert symbol highlights important information.

The STOP symbol cautions you against doing something.

Tips are principles or practices that you’ll benefit from knowing or using.

The i symbol shows where you can find more information, such as in our online Knowledge base.

1.3. Contact support

See our online Knowledge base which contains updates and more information.

If you need more help raise a case on our support website.

1.4. Give feedback

If you find an error or if you think some information needs improving, raise a case on our support website or email docs@wandisco.com.

2. Release Notes

The Fusion Plugin for Live Ranger extends WANdisco Fusion by replicating Apache Ranger. With it, WANdisco Fusion maintains a Live Data environment including Ranger content, so that applications can access, use and modify a consistent view of data everywhere, spanning platforms and locations, even at petabyte scale. WANdisco Fusion ensures the availability and accessibility of critical data everywhere.

2.1. Live Ranger 2.1.2 Build 19

05 November 2018: The 2.1.2 release of the Fusion Plugin for Live Ranger is a minor update. Implementations of WANdisco Fusion that include Ranger replication requirements should take advantage of this update.

2.1.1. Installation

The Fusion Plugin for Live Ranger supports an integrated installation process that allows it to be added to an existing WANdisco Fusion deployment. Please consult the Installation Guide for details.

2.2. Highlighted Improvements

2.2.1. WD-RPX-321 - Support for HDP 2.6.5

Added Fusion Plugin for Live Ranger support for Hortonworks Data Platform 2.6.5.

2.2.2. WD-RPX-313/IBM-REPL-79 - Live Ranger JSON Errors

JSON errors can be seen in the proxy server logs when creating or deleting a user. This has been resolved with this release.

2.3. Known Issues

WD-RPX-179 - Poor operation with Azure HD Insight configured with ADLS as primary file system.
WD-RPX-172 - No provision is made for Ranger service entity replication and repair, as service entity information can be cluster-specific.
WD-RPX-172 - Consistency checks do not check service entity configuration details as these can be cluster specific, but will check service entity details based on component name.
WD-RPX-238 - On removing 'Live Ranger' service from 'Ambari' and performing 'ambari-server restart', the cluster manager shows the following warnings:
```
'WARN - You have config(s): proxy-server-site-version1528585291378 that is(are) not mapped (in serviceconfigmapping table) to any service!'
```
This is just a warning and has no impact on the desired functionality. This is an issue with Ambari - AMBARI-20875.
WD-RPX-347 - Live Ranger plugin status on the WANdisco Fusion dashboard is incorrectly shown as 'Unknown'.

2.3.1. Resolved Issues

WD-RPX-323 - Live Ranger proxy fails with an exception due to missing Hadoop dependency.
WD-RPX-313 - Live Ranger JSON errors in proxy server logs when creating or deleting a user.

2.3.2. Supported Platforms

WANdisco Fusion

2.12

Hadoop

Hortonworks Data Platform 2.6.4 and Hortonworks Data Platform 2.6.5
Azure HDInsight 3.6

2.4. Live Ranger 2.1.1 Build 172

11 October 2018: The 2.1.1 release of the Fusion Plugin for Live Ranger is a minor update, resolving a single known issue. Implementations of WANdisco Fusion that include Ranger replication requirements should take advantage of this update.

2.4.1. Installation

The Fusion Plugin for Live Ranger supports an integrated installation process that allows it to be added to an existing WANdisco Fusion deployment. Please consult the Installation Guide for details.

2.4.2. Highlighted Improvements

This minor update to the Fusion Plugin for Live Ranger 2.1 resolves a single known issue:

WD-RPX-306/IBM-REPL-79 - Live Ranger Intermittent Replication Failures
Intermittent replication failures could occur when interacting with the Ranger UI from specific browser versions. This has been resolved with this release.

See the Live Ranger 2.1 release notes below for more information on supported platforms, system requirements and known issues.

2.5. Live Ranger 2.1 Build 170

11 September 2018: The 2.1 release of the Fusion Plugin for Live Ranger is a major update, adding functionality and resolving some of the known issues with prior releases, ensuring that it covers the features available with the prior Fusion Plugin for Live Ranger. With this release, implementations of WANdisco Fusion that include Ranger replication requirements should take advantage of the Fusion Plugin for Live Ranger.

2.5.1. Installation

The Fusion Plugin for Live Ranger supports an integrated installation process that allows it to be added to an existing WANdisco Fusion deployment. Consult the Installation Guide for details.

2.5.2. Highlighted New Features

This major update to Fusion Plugin for Live Ranger 2.1 adds some new key features and addresses limitations of the 2.0 release. Notable enhancements are:

High Availability: Fusion Plugin for Live Ranger now supports high availability in WANdisco Fusion deployments. This allows for the plugin and its proxy to operate in a HA configuration.
Reporting on non-writer Nodes: Consistency Check and Repair statuses and reports are now available on non-writer nodes.
Limit scope of Consistency Check and Repair by Entity: There is now an option to limit the scope of Consistency Check and Repair by Entity (e.g. users, groups, etc.). This allows for more control over what gets repaired and when.
Ambari-based Installation: The Fusion Plugin for Live Ranger now supports Ambari-based WANdisco Fusion deployments, which includes high availability setups.

2.5.3. Supported Platforms

WANdisco Fusion

2.12

Hadoop

Hortonworks Data Platform 2.6.4
Azure HDInsight 3.6

2.5.4. System Requirements

Before installing or upgrading, ensure that your systems, software, and hardware meet the requirements. The requirements for WANdisco Fusion are found in the User Guide at http://docs.wandisco.com/bigdata/wdfusion/2.12/#_prerequisites_checklist.

Fusion Plugin for Live Ranger is tested on a more limited number of operating systems then the main product. These are:

CentOS 6 x86_64
CentOS 7 x86_64
RHEL 7 x86_64
Ubuntu 16.04LTS

2.5.5. Resolved Issues

The following known issues have been resolved with this release.

WD-RPX-141 - Session/credentials storage on Fusion Writer change
WD-RPX-235 - UBUNTU - LiveRanger plugin uninstallation executed when fusion server is running
WD-RPX-238 - Ambari stack deployment: ambari warns that RPX config isn’t removed
WD-RPX-239 - Ambari stack deployment: stack upgrade doesn’t work
WD-RPX-242 - Users are being created with Error
WD-RPX-246 - Both /etc/wandisco/fusion/server and /etc/hadoop/conf should be included in classpath
WD-RPX-269 - Handle URL Encoding
WD-RPX-279 - Live Ranger HA mode does not support Ambari stack configuration
WD-RPX-289 - Repair for groups does not work properly
WD-RPX-290 - Users created with wrong group lists

2.5.6. Known Issues

Fusion Plugin for Live Ranger 2.1 includes a small set of known issues.

WD-RPX-179 - Poor operation with Azure HD Insight configured with ADLS as primary file system.
WD-RPX-172 - No provision is made for Ranger service entity replication and repair, as service entity information can be cluster-specific.
WD-RPX-172 - Consistency checks do not check service entity configuration details as these can be cluster specific, but will check service entity details based on component name.
WD-RPX-238 - On removing 'Live Ranger' service from 'Ambari' and performing 'ambari-server restart', the cluster manager shows the following warnings:
```
'WARN - You have config(s): proxy-server-site-version1528585291378 that is(are) not mapped (in serviceconfigmapping table) to any service!'
```
This is just a warning and has no impact on the desired functionality.
WD-RPX-278 -Character encoding support
To use the standard Chinese coded character set GB18030, some additional configurations must be made to the underlying Ranger DBMS, i.e.,
1. Replace your /etc/my.cnf with my.cnf.
2. The Ranger assets within MySQL also needed to be converted from UTF8 to UTF8MB4.
  See ranger_mysql_gb18030.sql

3. Concepts

Familiarity with the following concepts will improve your use of the Fusion Plugin for Live Ranger.

WANdisco Fusion Plugin: A plugin is used by WANdisco Fusion to extend its functionality. Plugins are loaded by the WANdisco Fusion server on startup.
Apache Ranger: Apache Ranger offers a centralized security framework for fine grained access control over Hadoop and related components (Apache Hive, HBase, Storm, Knox, Solr, Kafka and YARN). Use the Apache Ranger administration console to manage policies for accessing resources (file, folder, database, table, column, etc.) for a particular set of users and/or groups, and enforce those policies within Hadoop.

Ranger has a centralized web application that consists of policy, audit and administration modules. Authorized users can manage security policies via a web interface or the Apache Ranger REST API. Policies are enforced in Hadoop components by Ranger Plugins.
Apache Ranger Policy Server: The Policy Server maintains the policies defined by users, and responds to requests from Ranger Plugins to retrieve policy information.
Apache Ranger Audit Server: The Audit Server can be configured to send access audit logs generated by Apache Ranger Plugins to a range of destinations.
Apache Ranger Administration Portal: The Ranger Administration Portal provides a simple interface for security administrators to create and manage policies enforced by Apache Ranger.
Apache Ranger Plugin: Ranger Plugins are specific to the Hadoop component in which they enforce Ranger policies retrieved from the Ranger Policy Server. They are lightweight Java implementations that are embedded in the processes of other cluster components to intercept operations that would always execute without security policy enforcement, and apply those policies to prevent unauthorized operations. Plugins also deliver information to the Ranger Audit Server.

3.1. Product concepts

The Fusion Plugin for Live Ranger implements LiveData for Apache Ranger policies. It intercepts operations that act on policy definitions in the Apache Ranger Policy Server and ensures that they are coordinated and replicated among multiple Ranger Policy Server instances.

It consists of two key components:

Live Ranger Proxy: The Live Ranger Proxy is a server that sits between clients and the REST API and Web interface of the Ranger Policy Server. Prior to forwarding client requests to the Ranger Policy Server, the proxy first proposes them to the WANdisco Fusion server for coordination.
Live Ranger Plugin: The Live Ranger Plugin is a runtime extension for the WANdisco Fusion server. It accepts proposals for operation coordination from the Live Ranger Proxy, and leverages the LiveData capabilities of the WANdisco Fusion server to ensure that all operations are performed with guaranteed consistent outcomes among multiple Apache Ranger deployments.

This Plugin is also responsible for the execution of operations that originate from other Ranger deployments. It presents those requests to its local Apache Ranger Policy Server as though they originated locally so they can be executed.

3.2. Supported Functionality

The Fusion Plugin for Live Ranger:

provides functionality to replicate Ranger policy definitions between instances of the Apache Ranger Policy Administration Server using WANdisco Fusion
intercepts all means by which Ranger policies can be created, modified, deleted, etc. to coordinate those operations among multiple Apache Ranger instances
offers functionality for an administrator to check and report on the consistency between policy definitions across multiple Ranger instances
supports the ability to resolve inconsistencies among policies between Ranger instances
provides a selection of REST API endpoints by which its operation can be managed

Of note, the following capabilities are explicitly not performed by this product:

Synchronization of operations performed by Ranger Plugins that are specific to Hadoop components in each cluster. There is no dependency between the Fusion Plugin for Live Ranger and Ranger Plugins deployed in each cluster. Note that this means that although Ranger policies and their administration will be replicated with guaranteed consistency among Ranger instances, each cluster’s Ranger plugins will poll those policies independently, applying them independently also.
Replication of the Ranger Key Management Service. The Ranger KMS is a cryptographic key management service that supports "data at rest" encryption in HDFS.
Selective replication of Ranger policies. Ranger policy replication is enabled as a whole between clusters when using the Fusion Plugin for Live Ranger. Either all Ranger policies and repositories are replicated, or none are.

4. Installation

4.1. Pre-requisites

4.1.1. System Requirements

Along with the standard product requirements for WANdisco Fusion, you need to:

Ensure that your clusters use an Ambari-based deployment, see the release notes for your specific version for more information on which Hortonworks versions are supported.^[1]
Configure the Hadoop environment for either Simple or Kerberos security.
Use Apache Ranger for policy enforcement.

Known Issue

The GET operation for EntityCache loader fails if the Ranger Admin is not up while installing the Fusion Plugin for Live Ranger proxy.
Work around: Ensure that the Ranger Admin is active before installing the Fusion Plugin for Live Ranger proxy.

4.1.2. Replication Requirements

Replication Rule Creation
Prior to installation, establish a replication rule associated with an HDFS path that is dedicated for the use of the Fusion Plugin for Live Ranger.

This replicated rule has to be /rangerproxy.

Ranger services must match For consistency check and repair to function properly, both zones MUST have the same set of ranger services. See the Ranger service section for more information.

4.1.3. Security Requirements

There are a range of security-related preparations that must be performed directly in your environments. For each cluster, ensure that the following tasks are performed.

Add the system user wd-ranger-user in all nodes:

# useradd wd-ranger-user Enter

Create the user directory in hdfs

# hdfs dfs -mkdir /user/wd-ranger-user Enter
# hdfs dfs -chown wd-ranger-user:wd-ranger-user /user/wd-ranger-user Enter
# hdfs dfs -chmod 755 /user/wd-ranger-user Enter

On the node where the KDC server is running:

Create the principal

kadmin.local# addprinc -randkey wd-ranger-user/<hostname of the Ranger proxy server>@<REALM.COM> Enter

Create the keytab

kadmin.local# xst -norandkey -kt wd-ranger-proxy.keytab wd-ranger-user/<hostname of the Ranger proxy server>@<REALM.COM> Enter

Copy the keytab into the Ranger Proxy Server node

# scp wd-ranger-proxy.keytab root@<hostname of the Ranger proxy server>:/etc/security/keytabs Enter

Change ownership of the file on that host

# chown wd-ranger-user:wd-ranger-user /etc/security/keytabs/wd-ranger-proxy.keytab Enter

Add the wd-ranger-user and hdfs user to the underlying Ranger instance with admin roles.

Create appropriate users in Ranger

Login to the Ranger Admin UI
Navigate to Settings >> Users/Groups tab
Create wd-ranger-user user with admin role
Create hdfs user with admin role

4.2. Installation

There are two methods for Fusion Plugin for Live Ranger installation. Follow the Ambari Installation method or install through the installer.

4.2.1. Ambari Installation

The details provided here are subject to change with minor updates. Details on the availability of the Ambari Management Pack required to follow this installation process can be obtained from WANdisco support.

Obtain Installation Components

The Ambari Management Pack for the Fusion Plugin for Live Ranger can be provided by WANdisco support: fusion-ranger-proxy-hdp-<version>.stack.tar.gz.

Copy the Management Pack to the Ambari server

# scp fusion-ranger-proxy-hdp-<version>.stack.tar.gz root@<ambari-server-host>:/ Enter

Install the Management Pack in Ambari

On the Ambari server host:

# service ambari-server stop Enter
# ambari-server install-mpack --mpack=/fusion-ranger-proxy-hdp-<version>.stack.tar.gz -v Enter
...
INFO: Management pack liveranger-mpack-1.0 successfully installed! Please restart ambari-server.
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties Ambari Server 'install-mpack' completed successfully.)

Restart the Ambari server

# service ambari-server start Enter
# service ambari-server restart Enter

Before starting the installation, Fusion servers must be inducted between zones, and a replication folder (and rule) should be in place before Live Ranger installation, Fusion servers must be inducted between zones, and replication folder (and rule) should exist for that membership.

Installation via the Ambari UI

Access the Ambari user interface and follow the steps below:

Click on Actions > Add Service: Add Service

Check "Live Ranger Proxy": Click "Next"

Assign Masters: Select the node where you want to deploy Live Ranger Proxy Server.

If existing proxy server RPM files are found by Ambari, it will cause the installation to fail. If this happens, remove them and restart the installation. IMPORTANT: One installation is complete, make sure you leave them in place.

Assign Slaves: Deploy slave roles to the nodes where the WANdisco Fusion server is installed.

Configure Services: Provide the necessary configuration values.

Configure Plugin: Provide configuration for plugin items also.

Configure Server: Provide server configuration details.

Review Details: Review the configuration and click "Deploy"

Install, Start, Test: Confirm successfull deployment and client "Next"

Access Fusion Live Ranger: Click on "Quick Links" > "Live Ranger UI"

Administer Ranger: Check that Ranger service is accessible through the Live Ranger Proxy, i.e. try to connect to http://<live-ranger-proxy-server-host>:8072 - changes to Ranger which are made using this proxy server will be replicated between zones.

4.2.2. Installation with the installer

Install the Fusion Plugin for Live Ranger using a standard RPM-based installation process. Configure the plugin with simple command-line tools or manual changes to configuration files that are specific to the plugin.

This is an alternative method to the Ambari installation method described above.

Run the installer

Obtain the Fusion Plugin for Live Ranger installer from WANdisco. Open a terminal session on each WANdisco Fusion node required and run the installer as follows:

# ./live-ranger-installer.<version>.sh Enter

Now configure the plugin with simple command-line tools or manual changes to configuration files.

Configure the plugin

Change current directory to /etc/wandisco/fusion/plugins/live-ranger:

# cd /etc/wandisco/fusion/plugins/live-ranger Enter

Execute the configuration script configure-proxy-plugin and provide details of how the proxy will operate:

Kerberos: Whether or not the cluster has security enabled.
Ranger Admin Username: The username of the Ranger administrator account
Ranger Admin Password: The password for the Ranger administrator account
Cluster Name: The name of the cluster
Fusion SSL: Whether or not Fusion is SSL enabled

An example (interactive mode):

# ./configure-proxy-plugin Enter
Enter the Ranger Policy Manager URL: http://rpxy01-vm0.bdfrem.wandisco.com:6080 Enter
Is the cluster Kerberos enabled (yes/no)? : yes Enter
Enter the Ranger Admin Username: admin Enter
Please enter the password to be encrypted: ***** Enter
Enter the Cluster Name: RPXY-01 Enter
---------------------------------------------------------------------------
* Ranger details *
Ranger Policy Manager URL: http://rpxy01-vm0.bdfrem.wandisco.com:6080
Cluster Name: RPXY-01
---------------------------------------------------------------------------
Confirm the rangerproxy plugin configuration details (yes/no): yes Enter
Adding 'ranger_default_rule=true' as a additional global properties into fusion
Enter the RangerProxy replication path [/rangerproxy]: /rangerproxy Enter
Is fusion server ssl enabled? (yes/no): no Enter
Response: * About to connect() to rpxy01-vm1.bdfrem.wandisco.com port 8082 (#0)
* Trying 10.10.214.121. connected
* Connected to chen5-5.bigd.wandisco.com (10.6.214.24) port 8082 (#0)
> PUT /fusion/fs/properties/global/additionalProperties?path=/rangerproxy HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: rpxy01-vm1.bdfrem.wandisco.com:8082
> Accept: /
> Content-Type: application/xml
> Content-Length: 138
>
< HTTP/1.1 401 Authentication required
* gss_init_sec_context() failed: : Ticket expiredWWW-Authenticate: Negotiate
< Set-Cookie: hadoop.auth=; Path=/; HttpOnly
< Content-Length: 0
< Server: Jetty(6.1.26.hwx)
<
* Connection #0 to host rpxy01-vm1.bdfrem.wandisco.com left intact
* Closing connection #0
RangerProxy plugin configuration done successfully, restart fusion server to load the plugin
--------------------------------------------------------------------------------------------
Note: You can edit the configuration values anytime in: /etc/wandisco/fusion/plugins/live-ranger/proxy-plugin-site.xml
The fusion server must be restarted for the changes to take effect
--------------------------------------------------------------------------------------------

Extra step required if using rhel7

If you are using rhel7 then the following steps will have been omitted above:

Enter the RangerProxy replication path [/rangerproxy]: /rangerproxy Enter
Is fusion server ssl enabled? (yes/no): no Enter

As a workaround, run the following command to create a replicated rule:

curl -v -X PUT -H 'Content-Type: application/xml' -d '<properties><properties><entry><key>ranger_default_rule</key><value>true</value></entry></properties></properties>' 'http://<fusion-server-host-name>:8082/fusion/fs/properties/global/additionalProperties?path=/rangerproxy'

To enable/disable SSL, edit the <ssl-enabled> property in /etc/wandisco/fusion/plugins/live-ranger/proxy-plugin-site.xml.

Example proxy-plugin-site.xml:

<rangerproxy>
  <ranger>
    <policymgr-url>http://hostname:6080</policymgr-url>
    <admin-username>admin</admin-username>
    <admin-password>encrypted_password</admin-password>
    <cluster-name>CLUSTER-01</cluster-name>
    <ssl-enabled>false</ssl-enabled>
  </ranger>
</rangerproxy>

An example (non-interactive mode):

Enter configuration values in defines_tmpl.sh file, e.g.

RANGER_POLICYMGR_URL="http://rpxy01-vm0.bdfrem.wandisco.com:6080"
PROXY_PLUGIN_KERBEROS="yes"
RANGER_ADMIN_USERNAME="admin"
RANGER_ADMIN_PASSWORD="*****"
CLUSTER_NAME="RPXY-01"
REPL_PATH="/rangerproxy"
FUSION_SERVER_SSL_ENABLED="no"

Execute the configuration script

# ./configure-proxy-plugin --config=defines_tmpl.sh Enter

Once completed, the script will produce the configuration file at /etc/wandisco/fusion/plugins/live-ranger/proxy-plugin-site.xml. You can modify this file later if required. If this file is changed, restart the Live Ranger Fusion server as configuration properties are obtained on server startup only.

Configure the proxy

Change current directory to /etc/wandisco/live-ranger-proxy:

# cd /etc/wandisco/live-ranger-proxy Enter

Execute the configuration script configure-proxy-server and provide details of how the proxy will operate.

Note - the exact details required here will depend on your answers to previous questions. For example, Enter list of read-only users is only evaluated on a kerberized cluster. Here you need to list all the users which are read-only i.e. those which only want to have access to the GET operation. Usernames begin with a lowercase letter.

An example (interactive mode):

# ./configure-proxy-server Enter
Enter the RangerProxy server listen host [0.0.0.0]: rpxy01-vm1.bdfrem.wandisco.com Enter
Enter the RangerProxy server listen port [8072]: 8072 Enter
Do you want to enable ssl (yes/no)?
[If yes, you need to provide the keystore path and password]: no Enter
Is the cluster Kerberos enabled (yes/no)?
[If yes, you need to provide the principal and keytab]: yes Enter
Enter list of read-only users: ranger Enter
Enter Spnego Principal: HTTP/rpxy01-vm1.bdfrem.wandisco.com@WANDISCO.HADOOP Enter
Enter the Spnego Keytab file path: /etc/security/keytabs/spnego.service.keytab Enter
/etc/security/keytabs/spnego.service.keytab file found.
Enter the Kerberos principal: rangerproxy@WANDISCO.HADOOP Enter
Enter the Kerberos keytab file path: /etc/security/keytabs/rangerproxy.keytab Enter
/etc/security/keytabs/rangerproxy.keytab file found.
Enter the Ranger Admin Username: admin Enter
Please enter the password to be encrypted : ***** Enter
Enter the fusion server zone name: zone01 Enter
Enter the Ranger Policy Manager URL: http://rpxy01-vm0.bdfrem.wandisco.com:6080 Enter
Enter the Cluster Name: RPXY-01 Enter
-------------------------------------------------------------------------------
* RangerProxy server details
RangerProxy server listen host: rpxy01-vm1.bdfrem.wandisco.com
RangerProxy server listen port: 8072
RangerProxy server SSL: false
RangerProxy server Kerberos: true
Kerberos Principal: rangerproxy@WANDISCO.HADOOP
Kerberos Keytab path: /etc/security/keytabs/rangerproxy.keytab
Kerberos Read-Only users list: Ranger
Kerberos Spnego Principal: HTTP/rpxy01-vm1.bdfrem.wandisco.com@WANDISCO.HADOOP
Kerberos Spnego Keytab: /etc/security/keytabs/spnego.service.keytab
Kerberos Name rules: DEFAULT
Ranger details
Fusion server zone name: zone01
Ranger Policy Manager URL: http://rpxy01-vm0.bdfrem.wandisco.com:6080
Cluster Name: RPXY-01
--------------------------------------------------------------------------------
Which user should Live Ranger Proxy run as? [root]: root Enter
Which group should Live Ranger Proxy run as? [root]: root Enter
Enter the minimum memory(-Xms) for Live Ranger Proxy (in MB) [512]: 512 Enter
Enter the maximum memory(-Xmx) for Live Ranger Proxy (in MB) [1024]: 1024 Enter
-------------------------------------------------
Live Ranger Proxy environment details *
Run as User: root
Run as Group: root
Minimum memory: 512m
Maximum memory: 1024m
-------------------------------------------------
Do you confirm the details for configuration (yes/no): yes Enter
RangerProxy server configuration done successfully, restart rangerproxy-server to load the rangerproxy server
--------------------------------------------------------------------------------------------------------
Note: You can edit the configuration values anytime in: /etc/wandisco/live-ranger-proxy/proxy-server-site.xml
The rangerproxy-server must be restarted for the changes to take effect
----------------------------------------------------------------------------------------------------

An example (non-interactive mode):

Enter configuration values in defines_tmpl.sh file, e.g.

LISTEN_HOST="rpxy01-vm1.bdfrem.wandisco.com"
LISTEN_PORT="8072"
PROXY_SERVER_SSL="no"
KEY_STORE_PATH=""
KEY_STORE_PASS=""
PROXY_SERVER_KERBEROS="yes"
KERBEROS_READ_ONLY_USERS_LIST="ranger"
KERBEROS_SPNEGO_PRINCIPAL="HTTP/rpxy01-vm1.bdfrem.wandisco.com@WANDISCO.HADOOP"
KERBEROS_SPNEGO_KEYTAB="/etc/security/keytabs/spnego.service.keytab"
KERBEROS_PRINCIPAL="rangerproxy@WANDISCO.HADOOP"
KERBEROS_KEYTAB_PATH="/ertc/security/keytabs/rangerproxy.keytab"
RANGER_ADMIN_USERNAME="admin"
RANGER_ADMIN_PASSWORD="*****"
ZONE_NAME="zone01"
RANGER_POLICYMGR_URL="http://rpxy01-vm0.bdfrem.wandisco.com:6080"
CLUSTER_NAME="RPXY-01"
PROXY_SERVER_USER_DEFAULT="root"
PROXY_SERVER_GROUP_DEFAULT="root"
PROXY_SERVER_MEM_LOW_DEFAULT="512m"
PROXY_SERVER_MEM_MAX_DEFAULT="1024m"

Execute the configuration script

# ./configure-proxy-server --config=defines_tmpl.sh Enter

Once completed, the script will produce the configuration file at etc/wandisco/live-ranger-proxy/proxy-server-site.xml. You can modify this file later if required.

Start the Ranger Proxy server

# rangerproxy-server start Enter

To view the Live Ranger UI you need to use the values set for the RangerProxy server listen host and RangerProxy server listen port e.g. rpxy01-vm1.bdfrem.wandisco.com:8072. Note this is not the default Ranger Admin UI port which is 6080.

4.3. Upgrade

The release can be upgraded from prior versions with the following steps:

Stop the WANdisco Fusion server

# service fusion-server stop Enter

Stop the Ranger Proxy server

# service rangerproxy-server stop Enter

Upgrade the plugin RPM

# rpm -U fusion-ranger-plugin-hdp-2.6.4-2.0-100.noarch.rpm Enter

Upgrade the proxy RPM

# rpm -U fusion-ranger-proxy-hdp-2.6.4-2.0-100.noarch.rpm Enter

Start the WANdisco Fusion server

# service fusion-server start Enter

Start the Ranger Proxy server

# service rangerproxy-server start Enter

4.3.1. Upgrade with Ambari

SSH into the Ambari server node e.g.
```
ssh root@10.10.214.130
```

Scp the new Live Ranger stack tar file on node where Live Ranger Proxy Server is installed e.g:

scp server/build/distributions/fusion-ranger-proxy-hdp-<version>_<version>-centos.stack.tar.gz  root@10.10.214.130:/

Upgrade the Ambari stack

service ambari-server stop

ambari-server upgrade-mpack --mpack=/<new mpack stack tar file> -v

ambari-server upgrade-mpack --mpack=/fusion-ranger-proxy-hdp-<version>_<version>-centos.stack.tar.gz  --verbose

service ambari-server start

Restart Live Ranger from the Ambari UI

4.4. Validation

Once your installation is complete, you should verify that replication is working as expected before entering into a production phase. For example, create a User or Group through the Ranger UI and confirm that it is replicated.

Go to your Ranger UI - http://<Host Name>:<Live Ranger Port> e.g http://localhost:8072
In Settings → Users/Groups, add a new User.
Once the User is created, go to another zone in Ranger and confirm that the new User has been replicated.

4.5. Uninstallation

Uninstallation is simple, and can be done with the following steps.

Important!

Do not remove the Live Ranger core plugin RPMs/Jars as part of an uninstallation procedure. There may be cases where Fusion’s database continues to reference the core plugin RPMs after their removal, causing problems with Fusion restarts. Before removing core plugin RPMs/Jars you must first perform a flush of Fusion’s internal database, which would require a re-induction and configuration of your Fusion deployment. Contact WANdisco support for more information about this procedure.

Stop the WANdisco Fusion server

# service fusion-server stop Enter

Stop the Ranger Proxy server

# service rangerproxy-server stop Enter

Uninstall

# rpm -e fusion-ranger-plugin-hdp-2.6.4-2.0-100.noarch.rpm Enter
# rpm -e fusion-ranger-proxy-hdp-2.6.4-2.0-100.noarch.rpm Enter

Start the WANdisco Fusion server

# service fusion-server start Enter

5. Operation

WANdisco Fusion dashboard - plugin status unknown

The Fusion Plugin for Live Ranger 2.1 status on the WANdisco Fusion dashboard will incorrectly show as Unknown. This will be fixed in the next release of Fusion Plugin for Live Ranger.

Character encoding support

To use the standard Chinese coded character set GB18030, some additional configurations must be made to the underlying Ranger DBMS, i.e.,

Replace your /etc/my.cnf with my.cnf.
The Ranger assets within MySQL also needed to be converted from UTF8 to UTF8MB4. See ranger_mysql_gb18030.sql.

5.1. Login and login credentials

5.2. Ranger service

Ranger services must match

For consistency check and repair to function properly, both zones must have the same set of Ranger services. This includes, for example, Ranger KMS.

5.2.1. For Replication

Operations on Ranger Services are not replicated. They must be created on a per cluster basis.
Policy replication will only occur if the service name is the same between clusters. For example for c1_hdfs in cluster 1 and c2_hdfs in cluster 2, replication will occur between the clusters. However, replication will not occur between c1_hdfs and c2_hdfs1. Note that replication will also occur if the cluster name is omitted, provided the service names match, i.e. it is just called hdfs in all clusters.
Everything that is not a Ranger Service is replicated (users, groups etc).

5.2.2. For Consistency Check

Service consistency check will not consider configuration, and will check for the service name based on the cluster. For example, cluster1 name c1 will create the service as c1_<component1> and will be treated same as cluster2 name c2 service c2_<component1>.
Entities can also be used in consistency checks, e.g.
```
curl -v -s -X POST "http://<fusion-server-host-name>:8082/plugin/rangerproxy/cc?path=/rangerproxy"
```
The above call will give the consistency check for 6 entities - servicedef, service, policy, user, group and permission.

You can also specify entities, giving more than one entity by comma separated values, e.g.,
```
curl -v -s -X POST "http://<fusion-server-host-name>:8082/plugin/rangerproxy/cc?path=/rangerproxy&<entityName>
```
where <entityName> can be one of the 6 listed above in the format entityName = servicedef,policy.

5.2.3. For Repair

Service repair will not be done, this needs to be done manually.
As with replication and consistency checks, policy repair will only work if the service exists on both clusters (see above).

Once configured, restart the WANdisco Fusion server to use the configuration applied:

# service fusion-server restart Enter

Then start each Ranger Proxy server:

# service rangerproxy-server start Enter

5.3. Configuration

Configuration of the Fusion Plugin for Live Ranger proxy and server is performed with changes to the configuration files generated at installation time:

/etc/wandisco/fusion/plugins/live-ranger/proxy-plugin-site.xml
/etc/wandisco/live-ranger-proxy/proxy-server-site.xml

The Ranger Administration UI can be enabled for access via SSL. For full details of how to configure the Fusion Plugin for Live Ranger for interoperability with SSL-enabled Ranger installations, please contact WANdisco support.

5.4. Live Ranger Replication Rules

System critical rules, such as the Live Ranger plugin’s default rules are not displayed in the UI due to their sensitive nature. These rules are critical to the working of the plugin and should never be modified. For this reason the default rules are not exposed through the UI.

Default plugin replication rules will not appear in the Live Ranger UI, although, if required, you can interact with them through the REST API.

6. Reference Guide

The Fusion Plugin for Live Ranger exposes functionality using a REST API. Operations that can be performed using this API are described below.

6.1. Consistency Check

A Consistency Check is used to identify whether there are differences between the policy definitions of each participating Ranger deployment. Consistency checks can be long-lived tasks, and are associated with a task identifier that can be used to determine their progress, and to obtain results when a consistency check is complete.

Examples of consistency check operations are given below:

Start a consistency check

# curl --negotiate -u : -v -s -X POST "http://localhost:8082/plugin/rangerproxy/cc?path=/rangerproxy" Enter
HTTP/1.1 202 Accepted
Content-Location: http://localhost:8082/fusion/task/<taskId>;
Content-Length: 1221
Server: Jetty(6.1.26.hwx)

The <taskId> value returned by the operation to start a consistency check is used for subsequent operations that check on status or provide a consistency check report.

Check on status

# curl --negotiate -u : -v -s -X GET "http://localhost:8082/fusion/task/<taskId>" Enter
HTTP/1.1 200 OK
Content-Length: 1221
Content-Type: application/xml
Server: Jetty(6.1.26.hwx)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<task>
<taskId>9ee718f2-2122-11e8-a5bc-f2c1622b4ea1</taskId>
<timeCreated>1520329321377</timeCreated>
<creatorNodeId>a8446f91-083e-446b-a88e-536efd91aee8</creatorNodeId>
<timeUpdated>1520329324352</timeUpdated>
<isDone>true</isDone>
<aborted>false</aborted>
<properties>
<entry>
<key>CC_REPORT_PATH</key>
<value>/rangerproxy/.fusion/50c60f07-1c62-11e8-929c-c6059be1e476/metadata/9ee718f2-2122-11e8-a5bc-f2c1622b4ea1/cc-report</value>
</entry>
<entry>
<key>TOTAL_INCONSISTENCIES_FOUND</key>
<value>GroupDiff=0; UserDiff=11; PermModelDiff=4; ServiceDefDiff=0; ServiceDiff=21; PolicyDiff=42</value>
</entry>
<entry>
<key>TASK_TYPE</key>
<value>RANGERPROXY_CONSISTENCY_CHECK</value>
</entry>
<entry>
<key>LOCAL_COMPLETE</key>
<value>1520329324352</value>
</entry>
<entry>
<key>CC_REPORT_SUMMARY_PATH</key>
<value>/rangerproxy/.fusion/50c60f07-1c62-11e8-929c-c6059be1e476/metadata/9ee718f2-2122-11e8-a5bc-f2c1622b4ea1/cc-report-summary</value>
</entry>
<entry>
<key>LOCAL_START</key>
<value>1520329321377</value>
</entry>
<entry>
<key>CONSISTENCY_CHECK_STATUS</key>
<value>INCONSISTENT</value>
</entry>
</properties>
<previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
</task>

Obtain a consistency check report

# curl --negotiate -u : -v -s -X GET "http://localhost:8082/plugin/rangerproxy/cc/report/<taskId>?path=/rangerproxy&withconsistencyreport=true" Enter
HTTP/1.1 200 OK
Content-Length: 1221
Content-Type: application/xml
Server: Jetty(6.1.26.hwx)


{
"Totals": {
"users": {
"zone-01": 24,
"zone-02": 23
},
"groups": {
"zone-01": 9,
"zone-02": 9
},
"permissionModels": {
"zone-01": 6,
"zone-02": 6
},
"policies": {
"zone-01": 22,
"zone-02": 28
},
"services": {
"zone-01": 12,
"zone-02": 13
},
"servicedefinitions": {
"zone-01": 11,
"zone-02": 11
}
},
"Deltas": [
{
"zoneName": "zone-02",
"+users": {
"zone-01": 6
},
"-users": {
"zone-01": 5
},
"+groups": {
"zone-01": 0
},
"-groups": {
"zone-01": 0
},
"+permissionModels": {
"zone-01": 2
},
"-permissionModels": {
"zone-01": 2
},
"+policies": {
"zone-01": 18
},
"-policies": {
"zone-01": 24
},
"+services": {
"zone-01": 10
},
"-services": {
"zone-01": 11
},
"+servicedefinitions": {
"zone-01": 0
},
"-servicedefinitions": {
"zone-01": 0
}
},
{
"zoneName": "zone-01",
"+users": {
"zone-02": 5
},
"-users": {
"zone-02": 6
},
"+groups": {
"zone-02": 0
},
"-groups": {
"zone-02": 0
},
"+permissionModels": {
"zone-02": 2
},
"-permissionModels": {
"zone-02": 2
},
"+policies": {
"zone-02": 24
},
"-policies": {
"zone-02": 18
},
"+services": {
"zone-02": 11
},
"-services": {
"zone-02": 10
},
"+servicedefinitions": {
"zone-02": 0
},
"-servicedefinitions": {
"zone-02": 0
}
}
]
}

6.2. Repair

A Repair is used to resolve inconsistencies between the policy definitions of each participating Ranger deployment. Repair tasks can be long-lived, and are associated with a task identifier that can be used to determine their progress.

Examples of repair operations are given below:

Start a repair

# curl -v -s -X POST "http://localhost:8082/plugin/rangerproxy/repair/<taskId>?path=/rangerproxy&srcZone=<Source-zone-name>" Enter
HTTP/1.1 200 OK
< Content-Location: http://localhost:8082/fusion/task/cd2826ca-2124-11e8-a5bc-f2c1622b4ea1
< Content-Length: 0
< Server: Jetty(6.1.26.hwx)

Check on repair status

# curl --negotiate -u : -v -s -X GET "http://localhost:8082/fusion/task/<repair-taskId>" Enter
HTTP/1.1 200 OK
Content-Length: 1221
Content-Type: application/xml
Server: Jetty(6.1.26.hwx)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<task>
<taskId>cd2826ca-2124-11e8-a5bc-f2c1622b4ea1</taskId>
<timeCreated>1520330257985</timeCreated>
<creatorNodeId>a8446f91-083e-446b-a88e-536efd91aee8</creatorNodeId>
<timeUpdated>1520330258073</timeUpdated>
<isDone>true</isDone>
<aborted>false</aborted>
<properties>
<entry>
<key>TASK_TYPE</key>
<value>REPAIR_TASK</value>
</entry>
<entry>
<key>UPDATE_PENDING_ZONES</key>
<value/>
</entry>
<entry>
<key>REPAIR_STATUS</key>
<value>COMPLETED</value>
</entry>
<entry>
<key>LOCAL_COMPLETE</key>
<value>1520330258073</value>
</entry>
<entry>
<key>LOCAL_START</key>
<value>1520330257985</value>
</entry>
</properties>
<previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
</task>

6.3. Setup Live Ranger HA

Step 1 :
Follow the steps mentioned in the below Document for specific type of environment https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_hadoop-high-availability/content/configure_ranger_admin_ha.html

Step 2 :
After step 1 Ranger Admin will be running in HA mode. Check "http://rpxy01-vm2.bdfrem.wandisco.com:88" on browser, you should see the Ranger login page. (Assuming load balancer is installed on vm2 and Ranger Admin on vm4, vm5)

Step 3 :
Install Live Ranger on any of the more than one node (e.g. vm0 and vm1) While configuring Live Ranger Proxy and Plugin enter both the policy mgr url of both the Ranger comma separated. E.g : Enter the ranger policy manager URL : http://rpxy01-vm4.bdfrem.wandisco.com:6080,http://rpxy01-vm5.bdfrem.wandisco.com :6080

Step 4 :
Change load balancer configuration to point to Live Ranger Proxy rather than Ranger Admin

cd /usr/local/apache2/conf
vi ranger-cluster.conf

Update the following statement

<Proxy balancer://rangercluster>
    BalancerMember http://rpxy01-vm4.bdfrem.wandisco.com:6080 loadfactor=1 route=1
    BalancerMember http://rpxy01-vm5.bdfrem.wandisco.com :6080 loadfactor=1 route=2

<Proxy balancer://rangercluster>
    BalancerMember http://rpxy01-vm0.bdfrem.wandisco.com:8072 loadfactor=1 route=1 sta retry=30
    BalancerMember http://rpxy01-vm1.bdfrem.wandisco.com:8072 loadfactor=1 status=+H retry=0 route=2

Step 5 :
Run the following commands to restart the httpd server:

cd /usr/local/apache2/bin
./apachectl restart

Step 6 :
After step 5 Live Ranger will be running in HA mode. Check "http://rpxy01-vm2.bdfrem.wandisco.com:88" on browser, you should see the Ranger login page.

1. While operation is supported with Azure HDInsight 3.6, there is no automated installation process for it because its version of Ambari prevents the deployment of additional stacks.