3. Administration Guide
This Admin Guide describes how to set up and use WANdisco's WD Fusion.
3.1 Housekeeping
This section covers basic operations for
Starting up
To start WD Fusion UI:
- Open a terminal window on the server and log in with suitable file permissions.
- Run the fusion-ui-server service from the /etc/init.d folder:
rwxrwxrwx 1 root root 47 Apr 10 16:05 fusion-ui-server -> /opt/wandisco/fusion-ui-server/bin/fusion-ui-server
- Run the script with the start command:
[root@localhost init.d]# ./fusion-ui-server start
WD Fusion starts. Read more about the fusion-ui-server init.d script.
Starting fusion-ui-server:. [ OK ] - Also you can invoke the service directly. e.g.
service fusion-ui-server stop/start
Shutting down
To shut down:
- Open a terminal window on the server and log in with suitable file permissions.
- Run the WD Fusion UI service, located in the init.d folder:
rwxrwxrwx 1 root root 47 Dec 10 16:05 fusion-ui-server -> /opt/wandisco/fusion-ui-server/bin/fusion-ui-server
- Run the stop script:
[root@redhat6 init.d]# ./fusion-ui-server stop stopping fusion-ui-server: [ OK ] [root@redhat6 init.d]#The process shuts down.
The shutdown script attempts to stop proceses in order before completing, as a result you may find that (from WD Fusion 2.1.3) shutdowns may take up to a minute to complete.
init.d management script
The start-up script for persistent running of WD Fusion is in the /etc/init.d folder. Run the script with the help command to list the available commands:
[root@redhat6 init.d]#./fusion-ui-server helpusage:./fusion-ui-server(start|stop|restart|force-reload|status|uistart|uistop|repstart|repstop|version) start Start Fusion UI Server services stop Stop Fusion UI Server services restart Restart Fusion UI Server services force-reload Restart Fusion UI Server services status Show the status of Fusion UI Server services uistart Start the Fusion UI Server User Interface uistop Stop the Fusion UI Server User Interface repstart Start the Fusion UI Server Replicator repstop Stop the Fusion UI Server Replicator version Show the version of Fusion UI Server
Check the running status (with current process ID):
[root@redhat6 init.d]# ./fusion-ui-server status Checking delegate:not running [ OK ] Checking ui:running with PID 17579 [ OK ]
Check the version:
[root@redhat6 init.d]# ./fusion-ui-server version 1.0.0-83
Managing Services through the WD Fusion UI
Providing that the UI service is running, you can stop and start WD Fusion through the Fusion servers tab.
WD Fusion UI Login
The UI for managing WD Fusion can be accessed through a browser, providing your have network access and the port that the UI is listening on is not blocked.
http://<url-for-the-server>:<UI port>
http://wdfusion-static-0.dev.organisation.com:8083/ui/
You should not need to add the /ui/ at the end, you should be redirected there automatically.
Login using your Hadoop platform's manager credentials.
Login credentials
Currently you need to use the same username and password that is required for your platform manager, e.g. Cloudera Manager or Ambari. In a future release we will separate WD Fusion UI from the manager and use a new set of credentials.
Checking cluster status on the dashboard
The WD Fusion UI dashboard provides a view of WD Fusion's status. From the world map you can identify which data centers are experiencing problems, track replication between data centers or monitor the usage of system resources.
For more details on what each section of the Dashboard, see the Reference section for the Dashboard.
UI Dashboard will indicate if there are problems with WD Fusion on your cluster.
Server Logs Settings
The WD Fusion logs that we display in the WD Fusion UI are configured by properties in the ui.properties file. This is read by the Non-stop NameNode UI server on start up so you will need to restart the server for changes to take affect. The ui.properties file is not replicated between nodes so you must currently set it manually on each node.
3.2 Troubleshooting
We list any current known issues here, along with advise on fix or work around them:
Handling ThreadLocalRandom errors
When running hdfs commands it is possible that you may see the following error message:
WARN internal.ThreadLocalRandom: Failed to generate a seed from SecureRandom within 3 seconds. Not enough entrophy?
This error is caused by a deficiency in netty and will correspond in a drop in performance and is worth addressing using the following procedure. This example applies to Centos servers, you may need to adjust the procedure for other operating systems:
- Open a terminal session as root. Run
yum install rng-tools - Edit the rngd file, e.g.
vi /etc/sysconfig/rngd - add the line
EXTRAOPTIONS="-r /dev/urandom" - Run
service rngd start
to start up the random data daemon. - Run
to start the service on system boot.chkconfig rngd on
Note that when Netty 4.0.28 is released this procedure should no longer be necessary.
4. Managing Replication
WD Fusion is built on WANdisco's patented DConE active-active replication technology. DConE sets a requirement that all replicating nodes that synchronize data with each other are joined in a "membership". Memberships are co-ordinated groups of nodes where each node takes on a particular role in the replication system.
For more information about DConE and its different roles see the reference section's chapter called A Paxos Primer
Membership example
4.1 Create a membership
- Log in to the WD Fusion UI. Click on the Membership tab. Click on the + Create button. A "New Membership" window will open that will display the WD Fusion nodes organized by zone.
Create Membership1
- Configure the membership by selecting which nodes should be acceptors.
Settings
Guide to nodes
- APL
- Acceptor - the node will vote on the order in which replicated changes will play out
Proposer - the node will create proposals for changes that can be applied to the other nodes
Learner - the node will receive replication traffic that will synchronize its data with other nodes.
4.2 Replicated Folders
WD Fusion allows selected folders within your hdfs file system to replicated to other data centers in your cluster. This section covers the set up and management of replicated folders.
Create a replicated folder
Before you can configure WD Fusion UI to replicate a hdfs folder, you need to set the folder up on your cluster. This procedure is covered in the initial set up of WD Fusion server, it's covered here in more detail, specifically for creating additional replicated directories after a successful deployment.
- On each data center, create a directory in the hdfs file space. To avoid permission problems, ensure that the owning user/group are identical across your data centers. Use Hadoop's filesystem command to complete the tasks:
hadoop fs -mkdir /user/hiver hadoop fs -chown -R hiver:groupname /user/hiver
- As user
hdfs, run the following commands on each data center:hadoop fs -mkdir /usr/hive/warehouse-replicated hadoop fs -chown hiver:hiver /warehouse-replicated
This ensures that the a universal system user has read/write access to the hdfs directorywarehouse-replicatedthat will be replicated through WD Fusion.
Set up the folder in WD Fusion UI
- Once the folder is in place on all nodes, log in to WD Fusion UI on one of the WD Fusion nodes and click on the Replciated Folders tab.
Create Rule
- Select the folder that you created in step 1. from the HDFS File Tree. Select check boxes of the Zones that you wish the folder to be replicated between. Select an available Membership from the Membership dropdown. The membership determines how replication is managed, there's more information about how membership works in the Reference section A Primer on Paxos.
Replicated folders
- If you didn't complete a consistency check on the selected folder, you may do so now.
Replicate to Zones
- After the completion of a consistency check, the Consistency column will report the consistency status.
Replicated folder status
Configure Hadoop
Once WD Fusion has been installed and set up, you will need to modify your Hadoop applications so that when appropriate, they write to your replicated folder.
Configure Hadoop applications to write to the replicated file space.
- Copy DC1's
/etc/hadoop/conf/hdfs-site.xmlinto your DC1's/opt/fusion-server/directory. - Modify the hdfs-site of DC1's
/opt/fusion-server/to use "myCluster" or whatever the nameservice is forfs.defaultFS. - Copy DC1's
/etc/hadoop/conf/hdfs-site.xmlinto your DC1's/opt/fusion/'s corresponding IHC server directory. - Modify the core-site of DC1's
/opt/fusion/'s corresponding IHC server directory to use "myCluster" or whatever the nameservice is forfs.defaultFS. - Restart the DC1 Fusion server and the DC1 corresponding IHC server.
- If DC2 is also HA-enabled, repeat the process but from DC2's side.
- Log in to the WD Fusion UI. Click on the Settings tab.
- Click on Disk Monitoring at the top of the side menu.
Settings - Disk monitor
- Click Create.
Settings - Disk monitor
- Enter the required details for setting up a disk monitor.
Settings - Disk monitor
- File system path
- Enter the full path of the system directory that will be monitored for disk usage.
- Severity level
- Select a system log severity level (Severe, Warning, Info or Debug) that will correspond with the Disk Capacity Threshold.
Caution Assigning a monitor with Severe level will impact operation should its trigger Disk Capacity Threshold be met. The affected WD Fusion will immediately shut down to protect its file system from corruption. Ensure that Severe level monitors are set up with a threshold that corresponds with serious risk. Set the threshold too low and you may find WD Fusion nodes are shutdown needlessly.
- Disk Capcity Threshold (bytes)
- The maximum amount of data that can be consumed by the selected system path before the monitor sends an alert message to the log file.
- Message
- A human-readible message that will be sent to the log at the point that the Disk Capacity Threshold is reached.
- You can set a monitor to have multiple trigger points. Click + Add another severity monitor and add an additional Severity level, Disk Capacity Threshold and Message. You can have a separate monitor for each Log level.
Settings - Additional Disk monitors
- Log in to the WD Fusion UI. Click on the Settings tab.
- Click on UI Settings link on the side menu.
- Enter a new HTTP Port or HTTP SSL.
Settings - Change it
- Click Update. You may need to update the URL in your browser to account for the change you just made.
- Log in to the WD Fusion UI. Click on the Settings tab.
- Click on Email Notifications link on the side menu or the main header bar.
- Enter the details for your SMTP server.
Email Notification Settings
- SMTP Host
- The hostname or IP address for your email relay server.
- SMTP Port
- The port used by your email relay service. SMTP default port is 25.
- Connection Encryption:
- Drop-down for choosing the type of encryption that the mail server uses, None, SSL or TLS are supported. If SSL or TLS are selected you should make sure that you adjust the SMTP port value, if required.
- Authentication
- Checkbox for indicating that a username and password are required for connecting to the mail server. If you tick the checkbox additional entry fields will appear.
- SMTP Username
- A username for connecting to the email server.
- SMTP Password
- Apassword for connecting to the email server.
- Test email to
- Optional field for entering an email address that can be used for testing that the email setup will work.
- Send Test Email
- Button, click to send a sample notification email so that you can confirm that the email notification settings are working.
- Save Settings
- Button, click to store your email notification entries.
- Click Update. You may need to update the URL in your browser to account for the change you just made.
- Open a terminal to the WD Fusion. You will be making changed to configuration files so you need to be logged on as a system user with read-write permissions for system files.
- Navigate to the WD Fusion UI's properties directory, e.g.
/opt/wandisco/fusion-ui-server/properties/ - Open the
emailer.propertiesfile in an editor, after creating a backup copy. If you can't see anemail.propertiesfile there, you may be able to track it down by opening theui.propertiesfile and looking for the line:email.properties.path = path/to/email.properties - In the
email.propertiesfile, locate the message lines that correspond to those events that you want to edit:# below properties have default values: email.auth=false email.encryption=NONE email.from=namenode@example.com email.host=smtp.example.com email.password=4k9+9sWZ3/4= email.port=25 email.username= cpu_load_threshold_hit.enabled=false cpu_load_threshold_hit.message=The following issue occured: %s cpu_load_threshold_hit.subject=NonStop Name Node Alert cpu_load_threshold_hit.to= data_node_limit_count.enabled=false data_node_limit_count.message=The following issue occured: %s data_node_limit_count.subject=NonStop Name Node Alert data_node_limit_count.to= hdfs_usage_threshold_hit.enabled=false hdfs_usage_threshold_hit.message=The following issue occured: %s hdfs_usage_threshold_hit.subject=NonStop Name Node Alert hdfs_usage_threshold_hit.to= java_heap_usage_threshold_hit.enabled=false java_heap_usage_threshold_hit.message=The following issue occured: %s java_heap_usage_threshold_hit.subject=NonStop Name Node Alert java_heap_usage_threshold_hit.to= license_expiring.enabled=true license_expiring.message=The following issue occured: %s license_expiring.subject=NonStop Name Node Alert license_expiring.to=support@example.com - Enable the message block by changing the
enabledto true. Enter the messaging that you want for each notification event. You can for a new line by using a forward slash, i.e. "\".cpu_load_threshold_hit.enabled=true cpu_load_threshold_hit.message=Enter your own messaging here, allowing you to set more specific messaging for the alert. %s cpu_load_threshold_hit.subject=NonStop Name Node Alert cpu_load_threshold_hit.to=email address suitable for monitoring system failures - Check and save the changes. You do not need to restart anything for changes to take effect.
- Look to the security procedures of your particular form of Hadoop.
- For Hortonworks/Ambari see Ambari Kerberos Setup .
- For Cloudera/CDH5 see CDH5 Security Guide and Hadoop Security
- Generate Keytab for each of your WD Fusion nodes.
- On the KDC, using
kadmin.local, create new principals for WD Fusion user and generate keytab file:> addprinc -randkey fusion/${hostname}@${krb_realm} > ktadd -k fusion.keytab -norandkey fusion/${hostname}@${krb_realm} - Copy the generated keytab to
/etc/hadoop/conf/on the WD Fusion server.
- On the KDC, using
- Update the configuration on the Fusion servers, adding the following properties to
core-site.xml.<property> <name>fs.fusion.keytab</name> <value>/etc/hadoop/conf/fusion.keytab</value> </property> <property> <name>fs.fusion.principal</name> <value>fusion/{hostname}@{krb_realm}</value> </property>- fs.fusion.keytab
- Path to generated keytab.
- fs.fusion.principal
- Principal for the keytab file.
- Set up a proxy user on the NameNode, adding the following properties to
core-site.xmlon the NameNode(s). - hadoop.proxyuser.$USERNAME.hosts
- Defines hosts from which client can be impersonated.
- hadoop.proxyuser.$USERNAME.groups
- Defines groups that can be impersonated value can be a list comma separated or a wildcard (*).
- Open a terminal and navigate to
<INSTALL_DIR>/etc/wandisco/config. - Within
/configmake a new directory called ssl.
mkdir ssl
- Navigate into the new directory.
cd ssl
- Copy your private key into the directory. If you don't already have keys set up you can use JAVA's keygen utility, using the command:
keytool -genkey -keyalg RSA -keystore wandisco.ks -alias server -validity 3650 -storepass <YOUR PASSWORD>
Read more about the Java keystore generation tool in the KB article - Using Java Keytool to manage keystores
Java keytool options
Variable Name Description -genkey Switch for generating a key pair (a public key and associated private key). Wraps the public key into an X.509 v1 self-signed certificate, which is stored as a single-element certificate chain. This certificate chain and the private key are stored in a new keystore entry identified by alias. -keyalg RSA The key algorithm, in this case RSA is specified. wandisco.ks This is file name for your private key file that will be stored in the current directory. - alias server Assigns an alias "server" to the key pair. Aliases are case-insensitive. -validity 3650 Validates the keypair for 3650 days (10 years). The default would be 3 months. - storepass <YOUR PASSWORD> This provides the keystore with a password. If no password is specified on the command, you'll be prompted for it. Your entry will not be masked so you (and anyone else looking at your screen) will be able to see what you type.
Most commands that interrogate or change the keystore will need to use the store password. Some commands may need to use the private key password. Passwords can be specified on the command line (using the
-storepassand-keypassoptions).
However, a password should not be specified on a command line or in a script unless it is for testing purposes, or you are on a secure system.
The utility will prompt you for the following information
What is your first and last name? [Unknown]: What is the name of your organizational unit? [Unknown]: What is the name of your organization? [Unknown]: What is the name of your City or Locality? [Unknown]: What is the name of your State or Province? [Unknown]: What is the two-letter country code for this unit? [Unknown]: Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct? [no]: yes
Enter key password for <mykey> (RETURN if same as keystore password): - With the keystore now in place, you'll now need to add variables to the application.properties
SSL DConE Encryption Variables for application.properties
Variable Name Example Description ssl.enabled true Requires a "true" or "false" value. Clearly when the value is set to false, none of the other variables will be used. ssl.debug true Requires a "true" or "false" value. When set to true debugging mode is enabled. ssl.keystore ./properties/wandisco.ks The path to the SSL private Keystore file that is stored in the node. By default this is called "wandisco.ks". ssl.key.alias wandisco The assigned alias for the key pair. Aliases are case-insensitive. ssl.keystore.password <a password> The SSL Key password. ssl.truststore ./properties/wandisco.ks The path to the SSL private truststore file that is stored in the node. By default this is called "wandisco.ks" because, by default the keystore and truststore are one and the same file, although it doesn't have to be. ssl.truststore.password "bP0L7SY7f/4GWSdLLZ3e+ The truststore password. The password should be encrypted.
Changes in any of these values require a restart of the DConE service. Any invalid value will restart the replicator and no DConE traffic will flow. - On ALL nodes create key directories:
/etc/security/serverKeys and /etc/security/clientKeys
- On all nodes, create keystore files:
cd /etc/security/serverKeys keytool -genkeypair -alias $HOSTNAME -keyalg RSA -keysize 2048 -dname CN=$HOSTNAME,OU=Dev,O=BigData,L=SanRamon,ST=ca,C=us -keypass $PASSWORD -keystore $HOSTNAME.ks -storepass $PASSWORD
There's further explanation of what these options do, see the key Java keytool options - On all nodes export the certificate public key to a certificate file:
cd /etc/security/serverKeys keytool -exportcert -alias $HOSTNAME -keystore $HOSTNAME.ks -rfc -file $HOSTNAME.crt -storepass $PASSWORD
- On all nodes, import the certificate into truststore file:
cd /etc/security/serverKeys keytool -importcert -noprompt -alias $HOSTNAME -file $HOSTNAME.crt -keystore $HOSTNAME.trust -storepass $PASSWORD
-
Create a single truststore file containing the public key from all certificates (this will be for clients) start on node1:
cd /etc/security/serverKeys
Copy trust store file from current node to next one and redo all steps above. - From last node copy trust store, which has all certificates to all servers under
/etc/security/clientKeys/all.jks - On all nodes, copy keystore to "
service".ks (e.g. hdfs.ks) - The keystore contains private keys and certificates used by SSL servers to authenticate themselves to SSL clients. By convention, such files are referred to as keystores.
- When used as a truststore, the file contains certificates of trusted SSL servers, or of Certificate Authorities trusted to identify servers. There are no private keys in the truststore.
- Hadoop SSL requires that truststores and the truststore password be stored, in plaintext, in a configuration file that- is readable by all.
- Keystore and key passwords are stored, in plaintext, in a file that is readable only by members of the appropriate group.
- In Ambari, navigate to the HDFS service edit the configuration.
- Type SSL into the search filed to show the SSL properties.
- Make edits to the following properties:
Property Description SSL Server Keystore File Location Path to the keystore file containing the server certificate and private key. SSL Server Keystore File Password Password for the server keystore file. SSL Server Keystore Key Password Password that protects the private key contained in the server keystore. -
If you don't plan to use the default truststore, configure SSL client truststore properties:
Property Description Cluster-Wide Default SSL Client Truststore Location Path to the client truststore file. This truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. Cluster-Wide Default SSL Client Truststore Password Password for the client truststore file. -
We recommend that you also enable web UI authentication for the HDFS service, providing that you have already secuired the HDFS service. Enter web consoles in the search field to bring up Enable Authentication for HTTP Web-Consoles property. Tick the check box to enable web UI authentication.
Property Description Enable Authentication for HTTP Web-Consoles Enables authentication for hadoop HTTP web-consoles for all roles of this service. - Now the necessary edits are complete, click Save Changes.
- Follow the next section for setting up SSL for YARN/MapReduce.
- Navigate to the YARN or MapReduce service and click Configuration.
- In the search field, type SSL to show the SSL properties.
- Edit the following properties according to your cluster configuration:
Property Description SSL Server Keystore File Location Path to the keystore file containing the server certificate and private key. Enable Authentication for HTTP Web-Consoles Password for the server keystore file. SSL Server Keystore Key Password Password for the client truststore file. -
We recommend that you also enable web UI authentication for the HDFS service, providing that you have already secuired the HDFS service. Enter web consoles in the search field to bring up Enable Authentication for HTTP Web-Consoles property. Tick the check box to enable web UI authentication.
Property Description Enable Authentication for HTTP Web-Consoles Enables authentication for hadoop HTTP web-consoles for all roles of this service. - Click Save Changes.
- Navigate to the HDFS service and in the search field, type Hadoop SSL Enabled. Click the value for the Hadoop SSL Enabled property and select the checkbox to enable SSL communication for HDFS, MapReduce, and YARN.
Property Description Hadoop SSL Enabled Enable SSL encryption for HDFS, MapReduce, and YARN web UIs, as well as encrypted shuffle for MapReduce and YARN. - Restart all affected services (HDFS, MapReduce and/or YARN), as well as their dependent services.
Configure for High Availability Hadoop
If you are running Hadoop in a High Availability (HA) configuration then you should run through the following steps for WD Fusion:
4.3 Consistency Check
The consistency check mechanism lets you verify that replicated HDFS data is consistent between sites.
4.4 Deleting memberships
It is possible to delete memberships that are no longer required.
-->5. Settings
Set up a Custom Disk Monitor
Use this procedure to set up a custom monitor in WD Fusion UI's Disk Monitor tool.
The Monitoring Data tool monitors the disk usage of the WD Fusion software, providing a basic level of protection against it consuming all disk space. The tool also lets you set up your own monitors for user-selected resources.
The disk monitor is no substitute for dedicated, system-wide monitoring tools. Instead, it is intended to be a 'last stand' against possible disk space exhaustion that could lead to data loss or corruption.
Read our Recommendations for system-wide monitoring tools.
Edit a Disk Monitor
You can make changes to an existing custom monitor by clicking on the Edit
Settings - Change it
Caution You can't delete or modify the default Monitor which protects the system from disk space exhaustion from the temporary files created in the WANdisco replication directory /DConE/consensusNode
Change the UI Settings
You can change how you interact with WD Fusion UI through the browser. Use the following procedure to change either the HTTP or HTTP SSL port that is used to view the UI through a browser.
Setting up Node Location
WD Fusion is design to fit into deployments that have far-flug data centers. The Node Location setting is used to identify where in the world the data center is situated, using standard global positioning system coordinates. These coordinates will be used by any connected WD Fusion nodes to correctly place the node's location on the world map.
WD Fusion setting server location.
Set up email notifications
This section describes how to set up notification emails that will be triggered if one of the system resources that are graphed on the Dashboard breach a threshold.
SMPT Settings
Set up email messages
The notification emails content is not currently exposed to the UI. You can change it by making the following edit to the emailer.properties file.
Notification Email
Notification emails that are sent out by WD Fusion UI are made up of the following parts:
Event Details- System generated description that appears as "%s" in the emailer.properties file. It contains the specific details of the message, including NameNode name and threshold level breached.
Message- The user-defined message placed in the emailer.properties file. Footer- System generated list of node properties.
Event Details
TBD
Footer
Each email has a footer block added by the system which includes a lot of extra information useful for quickly troubleshooting problems.
HostName IP Address IP port Node Name Node Role Node's manager Node status Node's zone Node location Node latitude Node longitude Memory usage Disk usage Last update Time Now
Setting up Kerberos
If the Hadoop deployment is secured using Kerberos you need to enable Kerberos in the WD Fusion UI. Use the following procedure to enable Kerberos:
<property>
<name>hadoop.proxyuser.fusion.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.fusion.groups</name>
<value>*</value>
</property>
Setting up SSL encryption for DConE traffic
WD Fusion supports the use of Secure Socket Layer encryption (SSL) for securing its replication traffic. To enable this encryption you need to generate a keypair that must be put into place on each of your WD Fusion nodes. You then need to add some variables to the application.properties file.
Setting the server key
In the keystore, the server certificate is associate with a key. By default, we look for a key named server to validate the certificate. If you use a key for the server with a different name, enter this in the SSL settings.
SSL Troubleshooting
A complete debug of the SSL logging will be required to diagnose the problems. To capture the debugging, ensure that the variable debugSsl is set to "true".
To enable the logging of SSL implemented layer, turn the logging to FINEST for 'com.wandisco.platform.net' package.
Enable SSL for Hadoop Services
This section shows you how to enable SSL encryption for Hadoop's native services such as HDFS, Yarn or MapReduce.
Keystores are used in two ways:
While all SSL clients must have access to a truststore, it is not always necessary to create and deploy truststores across a cluster. The standard JDK distribution includes a default truststore which is pre-provisioned with the root certificates of a number of well-known Certificate Authorities. If you do not provide a custom truststore, the Hadoop daemons load this default truststore. Therefore, if you are using certificates issued by a CA in the default truststore, you do not need to provide custom truststores. However, you must consider the following before you decide to use the default truststore:
If you choose to use the default truststore, it is your responsibility to maintain it. You may need to remove the certificates of CAs you do not deem trustworthy, or add or update the certificates of CAs you trust. Use the keytool utility to perform these actions.
Security Considerations
keystores contain private keys. truststores do not. Therefore, security requirements for keystores are more stringent:
These considerations should guide your decisions about which keys and certificates you will store in the keystores and truststores that you will deploy across your cluster.
Keystores should contain a minimal set of keys and certificates. Ideally you should create a unique keystore for each host, which would contain only the keys and certificates needed by the Hadoop SSL services running on the host. Usually the keystore would contain a single key/certificate entry. However, because truststores do not contain sensitive information you can safely create a single truststore for an entire cluster. On a production cluster, such a truststore would often contain a single CA certificate (or certificate chain), since you would typically choose to have all certificates issued by a single CA.
SSL roles for Hadoop Services
| Service | SSL Role |
|---|---|
| HDFS | server and client |
| MapReduce | server and client |
| YARN | server and client |
| HBase | server |
| Oozie | server |
| Hue | client |
SSL servers load the keystores when starting up. Clients then take a copy of the truststore and uses it to validate the server's certificate.
Configure SSL for HDFS, YARN and MapReduce
Before you begin
Ensure keystores/certificates are accessible on all hosts running HDFS, MapReduce or YARN. As these services also run as clients they also need access to the truststore. (As mentioned, it's okay to put the truststores on all nodes as you can't always determine which hosts will be running the relevant services.)
keystores must be owned by the hadoop group and have permissions 0440 (readable by owner and group). truststores must have permission 0444 (readable by all).
You'll need to specify the absolute paths to keystore and truststore files - these paths need to be valid for all hosts - this translates into a requirement for all keystore file names for a given service to be the same on all hosts.
Multiple daemons running on a host can share a certificate. For example, in case there is a DataNode and an Oozie server running on the same host, they can use the same certificate.