7. Reference Guide

The reference guide walks through WD Fusion's various screens, providing a basic explanation of what everything does. For specific instruction on how to perform a particular task, you should view the Admin Guide

7.1 Technical Overview

What is WANdisco Fusion

WANdisco Fusion (WD Fusion) shares data between two or more clusters. Shared data is replicated between clusters using DConE, WANdisco's proprietory coordination engine. This isn't a spin on mirroring data, every cluster can write into the shared data directories and the resulting changes are coordinated in real-time between clusters.

100% Reliablity

Paxos-based algorithms enable DConE to continue to replicate even after brief networks outages, data changes will automatically catch up once connectivity between clusters is restored.

Below the coordination stream, actual data transfer is done as an asynchronous background process and doesn't consume MapReduce resources.

Replication where and when you need

WD Fusion supports Selective replication, where you control which data is replicated to particular clusters, based on your security or data management policies. Data can be replicated globally if data is available to every cluster or just one cluster.

The Benefits of WANdisco Fusion

Ingest data to any cluster, sharing it quickly and reliably with other clusters. Removing fragile data transfer bottlenecks, and letting you process data at multiple places improving performance and getting you more utilization from backup clusters.
Support a bimodal or multimodal architecture to enable innovation without jeopardizing SLAs. Perform different stages of the processing pipeline on the best cluster. Need a dedicated high-memory cluster for in-memory analytics? Or want to take advantage of an elastic scale-out on a cheaper cloud environment? Got a legacy application that's locked to a specific version of Hadoop? WANdisco Fusion has the connections to make it happen. And unlike batch data transfer tools, WANdisco Fusion provides fully consistent data that can be read and written from any site.
Put away the emergency pager. If you lose data on one cluster, or even an entire cluster, WANdisco Fusion has made sure that you have consistent copies of the data at other locations.
Set up security tiers to isolate sensitive data on secure clusters, or keep data local to its country of origin.
Perform risk-free migrations. Stand up a new cluster and seamlessly share data using WANdisco Fusion. Then migrate applications and users at your leisure, and retire the old cluster whenever you're ready.

7.2 A Primer on Paxos

Replication networks are composed of a number of nodes, each node takes on one of a number of roles:

Acceptors (A)

The Acceptors act as the gatekeepers for state change and are collected into groups called Quorums. For any proposal to be accepted, it must be sent to a Quorum of Acceptors. Any proposal received from an Acceptor node will be ignored unless it is received from each Acceptor in the Quorum.

Proposers (P)

Proposer nodes are responsible for proposing changes, via client requests, and aims to receive agreement from a majority of Acceptors.

Learners (L)

Learners handle the actual work of replication. Once a Client request has been agreed on by a Quorum the Learner may take the action, such as executing a request and sending a response to the client. Adding more learner nodes will improve availability for the processing.

Distinguished Node

It's common for a Quorum to be a majority of participating Acceptors. However, if there's an even number of nodes within a Quorum this introduces a problem: the possibility that a vote may tie. To handle this scenario a special type of Acceptor is available, called a Distinguished Node. This machine gets a slightly larger vote so that it can break 50/50 ties.

7.3 Paxos Node Roles in DConE

When setting up your WD Fusion servers they'll all be Acceptors, Proposers and Learners. In a future version of the product you'll then be able to modify each WD Fusion server's role to balance between resilience and performance, or to remove any risk of a tied vote.

7.3.1 Creating resilient Memberships

WD Fusion is able to maintain HDFS replication even after the loss of WD Fusion nodes from a cluster. However, there are some configuration rules that are worth considering:

Rule 1: Understand Learners and Acceptors

The unique Active-Active replication technology used by WD Fusion is an evolution of the Paxos algorithm, as such we use some Paxos concepts which are useful to understand:

Learners:
Learners are the WD Fusion nodes that are involved in the actual replication of Namespace data. When changes are made to HDFS metadata these nodes raise a proposal for the changes to be made on all the other copies of the filesystem space on the other data centers running WD Fusion within the membership.

Learner nodes are required for the actual storage and replication of hdfs data. You need a learner node where ever you need to store a copy of the shared hdfs data.
Acceptors:
All changes being made in the replicated space at each data center must be made in exactly the same order. This is a crucial requirement for maintaining synchronization. Acceptors are nodes that take part in the vote for the order in which proposals are played out.

Acceptor Nodes are required for keeping replication going. You need enough Acceptors to ensure that agreement over proposal ordering can always be met, even after accounting for possible node loss. For configurations where there are a an even number of Acceptors it is possible that voting could become tied. For this reason it is possible to make an Acceptor node into a tie-breaker which has slightly more voting power so that it can outvote another single Acceptor node.

Rule 2: Replication groups should have a minimum membership of three learner nodes

Two-node clusters (running two WD Fusion servers) are not fault tolerant, you should strive to replicate according to the following guideline:

The number of learner nodes required to survive population loss of N nodes = 2N+1
where N is your number of nodes.

So in order to survive the loss of a single WD Fusion server equipped datacenter you need to have a minium of 2x1+1= 3 nodes
In order to keep on replicating after losing a second node you need 5 nodes.

Rule 3: Learner Population - resilience vs rightness

During the installation of each of your nodes you can configure the Content Node Count number, this is the number of other learner nodes in the replication group that need to receive the content for a proposal before the proposal can be submitted for agreement.

Setting this number to 1 ensures that replication won't halt if some nodes are behind and have not received replicated content yet. This strategy reduces the chance that a temporary outage or heavily loaded node will stop replication, however, it also increases the risk that namenode data will go out of sync (requiring admin-intervention) in the event of an outage.

Rule 4: 2 nodes per site provides resilience and performance benefits

Running with two nodes per site provides two important advantages.

Firstly it provides every site with a local hot-backup of the namenode data.
Enables a site to load-balance namenode access between the nodes which can improve performance during times of heavy usage.
Providing the nodes are Acceptors, it increases the population of nodes that can form agreement and improves resilience for replication.

Replication Frequently Asked Questions

What stops a file replication between zones from failing if an operation such as a file name change is done on a file that is still transferring to another zone?

What stops a file replication between zones from failing if an operation such as a file name change is done on a file that is still transferring to another zone?

Operations, such as a rename only affects metadata, so long as the file's underlying data isn't changed, the operation to transfer the file will complete. Only then will the rename operation play out. When you start reading a file for the first time you acquire all the block locations necessary to fulfill the read, at this point metadata changes won't halt the transfer of the file to another zone.

7.4 Agreement recovery in WD Fusion

This section explains why when monitoring replication recovery, it may be possible to see a brief delay and seemingly out-of-order delivery of proposals at the catching-up node.

In the event that the WAN link between clusters is temporarily dropped, it may be noticed that when the link returns, there's a brief delay before the reconnected zones are back in sync and it may appear that recovery is happening with agreements being made out of order, in terms of the global sequence numbers (GSNs) associated with each agreement.

This behavior can be explained as follows:

"non-writer" nodes review the GSNs to determine which agreements the current writer has processed and which agreements they can remove from their own store, where they are kept in case the writer node fails and they have to take over.
when a new writer is elected, the presence/absence of a particular GSN tells the new writer which agreements can be skipped. There may be gaps in this sequence as not all propsals are filesystem operations. For example, writer and leader election proposials are not filesystem operations, therefore their GSNs are not written to the underlying filesystem.

Why are proposals seemingly being delivered out-of-order?

This is related and why you will see gsn's written "out-of-order" in the filesystem. Internally within Fusion "non-interfering" agreements are processed in parallel so we can increase throughout and the global sequence is not blocked on operations that may take a long time, such as a large file copy.

Example

Consider the following global sequence, where /repl1 is the replicated directory:

Copy 10TB file to /repl1/dir1/file1
Copy 10TB file to /repl1/dir2/file1
Chown /repl/dir1

Agreements 1. and 2. may be executed in parallel since they do not interfere with one-another. However, agreement 3. must wait for agreement 1 to complete before it can be applied to the filesystem. If agreement 2 completes before 1 then its gsn will be recorded before the preceding agreement and look on the surface like out-of-order delivery of GSNs.

Under the hood

DConE's Output Proposal Sequence (OPS) delivers agreed values in strict sequence, one-at-a-time, to an application. Applying these values to the application state in the sequence delivered by the OPS ensures the state is consistent with other replicas at that point in the sequence. However, an optimization can be made: if two or more values do not interfere with one another (see below for definition of 'interfere') they may be applied in parallel without adverse effects. This parallelization has several benefits, for example:

It may increase the rate of agreed values applied to the application state if there are many non-interfering agreements;
It avoids an agreement that takes a long time to complete (such as a large file transfer) from blocking later agreements that aren't dependent on that agreement having completed.

8. WD Fusion Configuration

This section lists the available configuration for WD Fusion's component applications. You should take care when making any configuration changes on your clusters.

8.1 WD Fusion Server

WD Fusion server configuration is stored in two files:

/etc/wandisco/fusion/server/application.properties

Property	Description	Permitted Values	Default	Checked at...
application.port	The port DConE uses for communication.	1-65535	6444	Startup
dcone.system.db.panic.if.dirty	If set to true and the DConE system database was not shut down 'cleanly' (i.e., the prevaylers weren't closed) then on restart the server will not start.	true or false	true	Startup
application.integration.db.panic.if.dirty	If set to true and the application integration database was not shut down cleanly (with the prevaylers closed) then on restart the server will not start.	true or false	true	Startup
database.location	The directory DConE will use for persistence.	Any existing path	None - must be present	Startup
executor.threads	The number of threads executing agreements in parallel.	1-Integer.MAX_VALUE	20	Startup
fusion.decoupler	The decoupler the Fusion server will use. See Decoupler Guide	dcone, disruptor, simple	dcone	Startup
disruptor.wait.strategy	The wait strategy to use when the disruptor is selected for fusion.decoupler.	blocking, busy.spin, lite.blocking, sleeping, yielding	yielding	Startup
jetty.http.port	The port the Fusion HTTP server will use.	1-65535	8082	Startup
request.port	The port Fusion clients will use.	1-65535	None - must be present	Startup
transport	The transport the Fusion server should use.	OIO, NIO, EPOLL	NIO	Startup
transfer.chunk.size	The size of the "chunks" used in a file transfer. Used as input to Netty's ChunkedStream.	1 - Integer.MAX_VALUE	4 KiB	When each pull is initiated
dcone.use.boxcar	Indicates use of boxcars or not.	true or false	false	startup
license.file	The path to the license file	A valid file system path to a license key file.	/etc/wandisco/server/license.key	On each periodic license check
max.retry.attempts	The maximum number of times to retry an agreed request.	1 - integer.MAX_VALUE	10	When executing an agreed request.
retry.sleep.time	The sleep time (ms) in between retries of an agreed request.	1 - Long.MAX_VALUE (notice the capital L, make sure you include it).	1000L	When executing an agreed request

/etc/hadoop/conf/core-site.xml

fs. prefix removal
Please take note that in WD Fusion 2.8 many of the properties in the following table have had the fs. prefix removed. The fs. is now used exclusively for filesystem specific properties.

Property	Description	Permitted Values (default value in bold)
fusion.http.authentication.enabled	Enables authentication on the REST API	true or false
fusion.http.authentication.type	Type of authentication used.	"simple" or "kerberos"
fusion.http.authentication.simple.anonymous.allowed	If type is "simple", whether anonymous API calls are allowed. If set to false, users must append a query parameter at the end of their URL "user.name=$USER_NAME"	true or false
fusion.http.authentication.kerberos.principal	If type is "kerberos", the principal the fusion server will use to login with. The name of the principal must be "HTTP".	'' (Putting simply an asterisk will cause the filter to pick up any principal found in the keytab that is of the form `"HTTP/"` and log in with all of them) `"HTTP/${HOSTNAME_OF_FUSION_SERVER}@${KERBEROS_REALM}" "HTTP/_HOST@${KERBEROS_REALM}"` (`"_HOST"` will auto-resolve to the hostname of the fusion server)
fusion.http.authentication.kerberos.keytab	If type is "kerberos", the path to a keytab that contains the principal specified.	Any String
fusion.http.authentication.signature.secret.file	Path to a readable secret file. File is used to authenticate cookies.	Any String
fusion.enable.early.pulls	A property targeted at FileSystems that do not support appends (e.g. S3, Azure). When set to the default "false" the Fusion server will ignore incoming HFlushRequests. The "fs." prefix has been removed as the property may not be specific to FileSystems in future.	true or false
fusion.http.authorization.enabled	Property that sets the state of authorization.	true or false.
fusion.http.authorization.authorized.read.writers	The read-writers config dictates which user is allowed to make write REST calls (e.g. DELETE, PATCH, POST, and PUT). Read-writers have both RW-permissions.	A comma-delimited list of authorized users.
fusion.http.authorization.authorized.readers	Users who have read-only permission. They are unable to do all of the calls noted in the read.writers entry, above.	A comma-delimited list of authorized users.
fusion.http.authorization.authorized.proxies	The core filter reads a new local property which specifies proxy principals - this is the remote user principal that the UI will authenticate as. The value for the property should be set to the user part of the UI kerberos credential, e.g. `hdfs`	A comma-delimited list of authorized users.
fusion.client.can.bypass	Enables or disables the ability for the client to bypass to underlying filesystem without waiting for a response from WD Fusion.	true or false
fusion.client.bypass.response.secs	Sets how long the client will wait for a response from Fusion for before bypassing to underlying.	integer (seconds)
fusion.client.bypass.retry.interval.secs	Sets how long to keep bypassing for once a client has been forced to bypass for the first time.	integer (seconds)
fusion.username.translations	This property enables administrators to handle user-mapping between replicated folders. This consists off a comma-separated list of regex rules. Each rule consists of a username (from an incoming request) seperated from a translated pattern by a "/". See further explanation Important: Take note that the username translation feature only translates the usernames on operations from remote zones.	null by default. pattern-string/translate-to-string

Username Translations

Example

<property>
     <name>fusion.username.translations</name>
    <value>hdp-(.*)/cdh-$1,([A-Z]*)-([0-9]*)-user/usa-$2-$1</value>
</property>

In the data center where the fusion.username.translations property is set, when a request comes in, it will check the username of the request against each listed pattern, and if the username matches that pattern, an attempt is made to translate using the listed value. If, during a check, none of the rules are found to match, we default to the username of the request, with no attempt to translate it.

Looking at the example translation rules:

hdp-(.*)/cdh-$1,([A-Z]*)-([0-9]*)-user/usa-$2-$1

Notice here that we have two rules:

hdp-(.*)/cdh-$1
([A-Z]*)-([0-9]*)-user/usa-$2-$1

To reiterate, we expect the following in the property:

Rules are comma separated.
Patterns and translations are separated by "/".
Patterns and translations don't contain "/".
White spaces should be accounted for in code, but are discouraged.

For the above config example, assume a createRequest comes in with the following usernames:

Username: ROOT-1991-user
1. We will check against the first pattern, hdp-(.*), and notice it doesn't match.
2. We will check against the second pattern, ([A-Z]*)-([0-9]*)-user, and notice it matches.
  1. Attempt to translate the username using usa-$2-$1.
  2. Username is translated to usa-1991-ROOT.
3. Create is done on the underlying filesystem using username, usa-1991-ROOT.
Username: hdp-KPac
1. We will check against the first pattern, hdp-(.*), and notice it matches.
2. Create is done on the underlying filesystem using username, cdh-KPac.
Username: hdfs
1. We will check against the first pattern, hdp-(.*), and notice it doesn't match.
2. We will check against the second pattern, ([A-Z]*)-([0-9]*)-user, and notice it doesn't match.
3. Username is left as hdfs. Create is done on the underlying filesystem using username, hdfs.

Because these are config properties, any data center can have any set of rules. They must be identical across fusion-servers that occupy the same zone but do not have to be identical across data centers.

See more about enabling Kerberos authentication on WD Fusion's REST API.

8.2 IHC Server

The Inter-Hadoop Communication Server is configured from a single file located at:

/etc/wandisco/fusion/ihc/server/{distro}/{version string}.ihc.

Property	Description	Permitted Values	Default	Checked at...
ihc.server	The hostname and port the IHC server will listen on.	String:[1 - 65535]	None - must be present	Startup
ihc.transport	The transport the IHC server should use.	OIO, NIO, EPOLL	NIO	Startup
ihc.ssl.enabled	Signifies that WD Fusion server - IHC communications should use SSL encryption.	true, false	false	Startup
http.server	The hostname and port the IHC HTTP server will listen on.	String:[1 - 65535]	None - must be present	Startup

8.3 WD Fusion Client

Client configuration is handled in

/etc/hadoop/conf/core-site.xml

Property	Description	Permitted Values	Default	Checked at...
fs.fusion.client.retry.max.attempts	Max number of times to attempt to connect to a Fusion server before failing over (in the case of multiple Fusion servers).	Any integer	3	Startup
fs.fusion.impl	The FileSystem implementation to be used.	See Usage Guide	None	Startup
fs.AbstractFileSystem.fusion.impl	The Abstract FileSystem implementation to be used.	See Usage Guide	None	Startup
fs.fusion.server	The hostname and request port of the Fusion server. Comma-separated list of hostname:port for multiple Fusion servers.	String:[1 - 65535] (Comma-separated list of Fusion servers)	None - must be present	Startup
fs.fusion.transport	The transport the FsClient should use.	OIO, NIO, EPOLL	NIO	Startup
fs.fusion.push.threshold	The number of bytes the client will write before sending a push request to the Fusion server indicating bytes are available for transfer.	Block size of underlying filesystem - `Long.MAX_VALUE`. (If the threshold is set to a figure less than the block size, the block size will be used. If the threshold is 0, pushes are disabled.)	The block size of the underlying filesystem	Startup
fs.fusion.ssl.enabled	If Client-WD Fusion server communications use SSL encryption.	true, false	false	Startup
fusion.underlyingFs	The address of the underlying filesystem	Often this is the same as the `fs.defaultFS` property of the underlying hadoop. However, in cases like EMRFS, the fs.defaultFS points to a local HDFS built on the instance storage which is temporary, with persistent data being stored in S3. Users are more likely to deploy S3 storage as the `fusion.underlyingFs`.	None - must be present	Startup
fs.hdfs.impl	DistributedFileSystem implementation to be used	See Usage Guide	None	Startup
fs.hdfs.impl.disable.cache	Disables the HDFS filesystem cache. Note that from 2.6.7, this parameter is no longer added to core-site.xml.	See Usage Guide	None	Startup

Usage Guide

There's a fixed relationship between the type of deployment and some of the Fusion Client parameters. The following table descibes this relationship:

Configuration	`fs.fusion.impl`	`fs.AbstractFileSystem.fusion.impl`	`fs.hdfs.impl`
Use of fusion:/// with HCFS	com.wandisco.fs.client.FusionHcfs	com.wandisco.fs.client.FusionAbstractFs	Blank
Use of fusion:/// with HDFS	com.wandisco.fs.client.FusionHdfs	com.wandisco.fs.client.FusionAbstractFs	Blank
Use of hdfs:/// with HDFS	Blank	Blank	com.wandisco.fs.client.FusionHdfs
Use of fusion:/// and hdfs:/// with HDFS	com.wandisco.fs.client.FusionHdfs	com.wandisco.fs.client.FusionAbstractFs	com.wandisco.fs.client.FusionHdfs

LocalFileSystems

We've introduced FusionLocalFs for LocalFileSystems using WD Fusion. This is necessary because there are a couple of places where the system expects to use a Local File System.

Configuration	`fs.fusion.impl`	`fs.AbstractFileSystem.fusion.impl`	`fs.hdfs.impl`
LocalFileSystems (See below)	com.wandisco.fs.client.FusionLocalFs	com.wandisco.fs.client.FusionLocalFs	com.wandisco.fs.client.FusionLocalFs

Therefore, for LocalFileSystems, users should set their fs.<parameter>.impl configuration to com.wandisco.fs.client.FusionLocalFs.

Usage

Set fs.file.impl to FusionLocalFs, (then any file:/// command will go through FusionLocalFs)
Set fs.fusion.impl to FusionLocalFs, (then any fusion:/// command will go through FusionLocalFs).

Further more, a user can now set any scheme to any Fusion*Fs and when running a command with that scheme, it will go through that Fusion*Fs. e.g.,

Set fs.orange.impl to FusionLocalFs, (then any oranges:/// command will go through FusionLocalFs).
Set fs.lemon.impl to FusionHdfs, (then any lemon:/// command will go through FusionHdfs).

Decoupler Guide

WD Fusion can decouple the handling of client requests from their processing using the Disruptor library. Consider "client request" to be shorthand for any coordinated filesystem operation submitted by client's FusionFs file system to the Fusion server, such as "make directory", "delete" or "close file"

Decoupler implementations

Disruptor (default): This is the default implementation. The disruptor provides a buffer so that the receiving thread can place the request in and return immediately. The potential advantage of the disruptor is that it uses relatively few resources (a fixed amount of memory and a single processing thread) whilst achieving low-latency and high throughput.
Disruptor strategy "runs hot".
In most deployments, running the decoupler in the Disruptor implementation makes sense because of its benefits in terms of reduced overhead for very large number of agreements compared to a more low-latency strategy. You should be aware that a side-effect of the disruptor strategy is that CPU cores will run at 100%.
Simple: In this implementation there is no decoupling at all - all processing is done on the receiving thread. This implementation should only be used for functional testing of the Fusion server.
DConE: When this option is set the incoming request is processed by DConE's internal decoupler. The size of the decoupler thread pool can be set through DConE's configuration, and the default is 50 processing threads. However, when there are a large number of agreements being made we may not want to use DConE's resources for the handling of client requests - this may starve the coordination engine of the threads it needs to perform agreement.

7. Reference Guide

7.1 Technical Overview

What is WANdisco Fusion

100% Reliablity

Replication where and when you need

The Benefits of WANdisco Fusion

7.2 A Primer on Paxos

Acceptors (A)

Proposers (P)

Learners (L)

Distinguished Node

7.3 Paxos Node Roles in DConE

7.3.1 Creating resilient Memberships

Rule 1: Understand Learners and Acceptors

Rule 2: Replication groups should have a minimum membership of three learner nodes

Rule 3: Learner Population - resilience vs rightness

Rule 4: 2 nodes per site provides resilience and performance benefits

Replication Frequently Asked Questions

What stops a file replication between zones from failing if an operation such as a file name change is done on a file that is still transferring to another zone?

7.4 Agreement recovery in WD Fusion

Why are proposals seemingly being delivered out-of-order?

Example

Under the hood

8. WD Fusion Configuration

8.1 WD Fusion Server

Username Translations

Example

8.2 IHC Server

8.3 WD Fusion Client

Usage Guide

LocalFileSystems

Usage

Decoupler Guide

Decoupler implementations

Replicated Directories

.fusion subdirectory

9. WD Fusion UI Reference Guide

9.1 Installation directories

WD Fusion Server

WD Fusion UI

9.2 WD Fusion Guide

Dashboard

System Usage Graphs

CPU Graph clarification

Replicated Folders

Filtering

Create Rule

Advanced Options

Consistency Check

Membership

Current

Unused

Create New

Fusion Nodes

Logs

Settings

About This Node

Graph Settings

Support

User Interface

UI Settings

Zone

Email Notifications

Hadoop

Client Downloads

Kerberos

Cluster Kerberos Configuration

Fusion Kerberos Configuration

Fix

Disk Monitoring

Create Monitor

DConE

Location Settings

License Settings

Fusion Server Settings

URI Selection

Set Push Threshold Manually

Client Bypass Settings

Consistency Check