WD Fusion Deployment

2.8 Manual Installation

The following section shows how you can manually install WD Fusion into your cluster without using the provided installer or orchestration script. We strongly recommend that you use the installer if you can, or the ochestration script if you must. However, we're aware that many large organizations follow strict policies for package management and require a set of manual steps so that they can deploy WD Fusion themselves.

Install Fusion Servers

  1. Install the fusion server in Zone 1 (e.g. cdh 5.3.0) 
    ssh -q -t -o StrictHostKeyChecking=no wdfusion2.example2.host.com /bin/rm -f /tmp/fusion-cdh-5.3.0-server-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/cdh/5.3.0/fusion-cdh-5.3.0-server-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion2.example2.host.com:/tmp
ssh -q -t -o StrictHostKeyChecking=no wdfusion2.example2.host.com yum install -y /tmp/fusion-cdh-5.3.0-server-2.0.1_SNAPSHOT-908.noarch.rpm
  2. Install the fusion server in Zone 2 (e.g. hdp 2.2.0) 
    ssh -q -t -o StrictHostKeyChecking=no wdfusion8.example2.host.com /bin/rm -f /tmp/fusion-hdp-2.2.0-server-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/hdp/2.2.0/fusion-hdp-2.2.0-server-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion8.example2.host.com:/tmp
ssh -q -t -o StrictHostKeyChecking=no wdfusion8.example2.host.com yum install -y /tmp/fusion-hdp-2.2.0-server-2.0.1_SNAPSHOT-908.noarch.rpm

    Install Fusion Clients

  3. Install the fusion client on all client nodes in zone 1 (all node managers, and client CM gateway roll Ambari HDFS client).

    Zone 1 Fusion Sever

    ssh -q -t -o StrictHostKeyChecking=no wdfusion2.example2.host.com /bin/rm -f /tmp/fusion-cdh-5.3.0-client-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/cdh/5.3.0/fusion-cdh-5.3.0-client-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion2.example2.host.com:/tmp
ssh -q -t -o StrictHostKeyChecking=no wdfusion2.example2.host.com yum install -y /tmp/fusion-cdh-5.3.0-client-2.0.1_SNAPSHOT-908.noarch.rpm

    Zone 1 NameNode(s)

    ssh -q -t -o StrictHostKeyChecking=no wdfusion2.example2.host.com /bin/rm -f /tmp/fusion-cdh-5.3.0-client-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/cdh/5.3.0/fusion-cdh-5.3.0-client-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion2.example2.host.com:/tmp
ssh -q -t -o StrictHostKeyChecking=no wdfusion2.example2.host.com yum install -y /tmp/fusion-cdh-5.3.0-client-2.0.1_SNAPSHOT-908.noarch.rpm

    For each Zone1 DataNode

    ssh -q -t -o StrictHostKeyChecking=no wdfusion2.example2.host.com /bin/rm -f /tmp/fusion-cdh-5.3.0-client-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/cdh/5.3.0/fusion-cdh-5.3.0-client-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion2.example2.host.com:/tmp
ssh -q -t -o StrictHostKeyChecking=no wdfusion2.example2.host.com yum install -y /tmp/fusion-cdh-5.3.0-client-2.0.1_SNAPSHOT-908.noarch.rpm
    Install the fusion client on all client nodes in Zone 2 (all node managers, and and client CM gateway roll Ambari HDFS client).

    Zone 2 Fusion Server

    ssh -q -t -o StrictHostKeyChecking=no wdfusion8.example2.host.com /bin/rm -f /tmp/fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/hdp/2.2.0/fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion8.example2.host.com:/tmp
ssh -q -t -o StrictHostKeyChecking=no wdfusion8.example2.host.com yum install -y /tmp/fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm

    Zone 2 NameNode(s)

    ssh -q -t -o StrictHostKeyChecking=no wdfusion7.example2.host.com /bin/rm -f /tmp/fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/hdp/2.2.0/fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion7.example2.host.com:/tmp
fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm 100% 1738KB 1.7MB/s 00:00
ssh -q -t -o StrictHostKeyChecking=no wdfusion7.example2.host.com yum install -y /tmp/fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm

    For each Zone 2 DataNode

    ssh -q -t -o StrictHostKeyChecking=no wdfusion8.example2.host.com /bin/rm -f /tmp/fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/hdp/2.2.0/fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion8.example2.host.com:/tmp
ssh -q -t -o StrictHostKeyChecking=no wdfusion8.example2.host.com yum install -y /tmp/fusion-hdp-2.2.0-client-2.0.1_SNAPSHOT-908.noarch.rpm

    Install Fusion IHC Servers

    Note: IHC servers can either be local to datanodes (secure setup) or in the same zone as there peer WD Fusion servers (default)

    IHC server Default

    On the HDP fusion server

    ssh -q -t -o StrictHostKeyChecking=no wdfusion8.ec2.wandisco.com /bin/rm -f /tmp/fusion-cdh-5.3.0-ihc-server-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/cdh/5.3.0/fusion-cdh-5.3.0-ihc-server-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion8.ec2.wandisco.com:/tmp
ssh -q -t -o StrictHostKeyChecking=no wdfusion8.ec2.wandisco.com yum install -y /tmp/fusion-cdh-5.3.0-ihc-server-2.0.1_SNAPSHOT-908.noarch.rpm

    On the CDH Fusion server

    ssh -q -t -o StrictHostKeyChecking=no wdfusion2.ec2.wandisco.com /bin/rm -f /tmp/fusion-hdp-2.2.0-ihc-server-2.0.1_SNAPSHOT-908.noarch.rpm
scp ./rpms/hdp/2.2.0/fusion-hdp-2.2.0-ihc-server-2.0.1_SNAPSHOT-908.noarch.rpm wdfusion2.ec2.wandisco.com:/tmp
ssh -q -t -o StrictHostKeyChecking=no wdfusion2.ec2.wandisco.com yum install -y /tmp/fusion-hdp-2.2.0-ihc-server-2.0.1_SNAPSHOT-908.noarch.rpm

    Configure IHC Servers

    list available IHC servers: 
    ssh wdfusion2.ec2.wandisco.com /bin/ls /opt/fusion/ihc-server
    for the HDP IHC server (on the CDH side) 
    ssh -q -t -o StrictHostKeyChecking=no wdfusion8.ec2.wandisco.com /opt/fusion/ihc-server/cdh-5.3.0/conf/configure-fusion-ihc-server
    for the CDH IHC server (on the HDP side) 
    ssh -q -t -o StrictHostKeyChecking=no wdfusion2.ec2.wandisco.com /opt/fusion/ihc-server/hdp-2.2.0/conf/configure-fusion-ihc-server

    Start The IHC servers

    start the IHC server for hip 
    ssh -q -t -o StrictHostKeyChecking=no wdfusion2.ec2.wandisco.com /etc/init.d/fusion-ihc-server-hdp_2_2_0 start
    start the IHC server on cdh 
    
ssh -q -t -o StrictHostKeyChecking=no wdfusion8.ec2.wandisco.com /etc/init.d/fusion-ihc-server-cdh_5_3_0 start

    Configure Fusion Servers

    1st server (inductor)

    ssh -q -t -o StrictHostKeyChecking=no wdfusion2.ec2.wandisco.com /bin/rm -rf /opt/fusion-server/application.properties /opt/fusion-server/dcone/db
ssh -q -t -o StrictHostKeyChecking=no wdfusion2.ec2.wandisco.com /bin/mkdir -p /opt/fusion-server/dcone/db
ssh -q -t -o StrictHostKeyChecking=no wdfusion2.ec2.wandisco.com /usr/bin/configure-fusion-server -y application_hostname=wdfusion2.ec2.wandisco.com application_port=6789 communication_hostname=wdfusion2.ec2.wandisco.com request_port=8023 node_identity=wdfs1 zone_identity=dc1 location_identity=location1 filesystem_uri=hdfs://wdfusieon2.ec2.wandisco.com:8020/ node_induction=no

sleep 60

    second server (inductee)

    
ssh -q -t -o StrictHostKeyChecking=no wdfusion8.ec2.wandisco.com /bin/rm -rf /opt/fusion-server/application.properties /opt/fusion-server/dcone/db
ssh -q -t -o StrictHostKeyChecking=no wdfusion8.ec2.wandisco.com /bin/mkdir -p /opt/fusion-server/dcone/db
ssh -q -t -o StrictHostKeyChecking=no wdfusion8.ec2.wandisco.com /usr/bin/configure-fusion-server -y application_hostname=wdfusion8.ec2.wandisco.com application_port=6789 communication_hostname=wdfusion8.ec2.wandisco.com request_port=8023 node_identity=wdfs2 zone_identity=dc2 location_identity=location2 filesystem_uri=hdfs://wdfusion7.ec2.wandisco.com:8020/ node_induction=yes inductor_host=wdfusion2.ec2.wandisco.com inductor_node_id=wdfs1 inductor_app_port=6789 inductor_location_id=location1 hdfs_replicate_dir=/repl1 application_hostname_list=${application_hostname_list} node_identity_list=${node_identity_list} location_identity_list=${location_identity_list

    Make core-site.xml changes

    • Using your management tool (Ambari or Cloudera Manager etc), add the following entries to the cluster-wide core-site.xml. The values are specific to each Data Center, i.e. you need one set of values for DC1 and another for DC2. 
      <property>
       <name>fs.fusion.impl</name>
       <value>com.wandisco.fs.client.FusionFs</value>
</property>
<property>
        <name>fs.fusion.server</name>
        <value>YOUR.WDFUSION.COM:8023</value>
</property>
Replace YOUR.WDFUSION.COM with your Fusion server that you configured in your mydefines.sh. Please use port :8023.
<property>
        <name>fusion.underlyingFs</name>
        <value>hdfs://vmhost08-vm0.host.domain.com:8020</value>
</property>
		
Replace the URL with your own cluster's underlying files system, make sure that you don't leave a trailing splash.		

<property>
        <name>fs.AbstractFileSystem.fusion.impl</name>
        <value>com.wandisco.fs.client.FusionAbstractFs</value>
</property>

      Notes
      Replace the hdfs://vmhost08-vm0.host.domain.com:8020 URL with your own cluster's underlying filesystem.
      Important! Take care not to add a trailing slash "/" at the end, it won't work. 

      <property>
        <name>fs.AbstractFileSystem.fusion.impl</name>
        <value>com.wandisco.fs.client.FusionAbstractFs</value>
</property>