The restart of a writer Fusion node on an AWS EMR zone during file transfer could result in that transfer failing.

Replication between HD Insight clusters requires an exclusion patter for replication rules to account for lack of append support in default HDI configuration.

Under some circumstances, file content replication to an S3 zone could result in target object sizes that were larger than those in the source zone.

Product documentation links to sample application.properties files are corrected.

The fs.fusion.s3.sse.enabled property has been introduced to allow support of AWS SSE-S3. Once SSE is enabled on the bucket, this configuration property can be set to true to enable SSE-S3.

Documentation for the use of SSL in an AWS deployment has been updated.
http://docs.wandisco.com/bigdata/wdfusion/2.10/#SSL-with-AWS

Correction to an auxiliary script that cleans stale repair tasks.

Documentation for replacing the product license has been updated.

Fusion indicates at runtime when a configuration needed for correct Presto operation is not present.

The restart of a writer Fusion node on an AWS EMR zone during file transfer could result in that transfer failing.

Replication between HD Insight clusters requires an exclusion patter for replication rules to account for lack of append support in default HDI configuration.

Under some circumstances, file content replication to an S3 zone could result in target object sizes that were larger than those in the source zone.

Product documentation links to sample application.properties files are corrected.

The fs.fusion.s3.sse.enabled property has been introduced to allow support of AWS SSE-S3. Once SSE is enabled on the bucket, this configuration property can be set to true to enable SSE-S3.

Documentation for the use of SSL in an AWS deployment has been updated.
http://docs.wandisco.com/bigdata/wdfusion/2.10/#SSL-with-AWS

Correction to an auxiliary script that cleans stale repair tasks.

Documentation for replacing the product license has been updated.

Fusion indicates at runtime when a configuration needed for correct Presto operation is not present.

Fixed an issue that could result in OutofMemoryError as a result of FUS-4643-2.

Fixed an issue where the none-writer zone was unable to detect that a repair was completed on the source zone, allowing task records to accumulate in non-writers.

We’ve clarified in the documentation that when replicating EMR to LocalFileSystem, you must enable "consistent view" to avoid replication errors that result in 0-length files.

Removed incorrect assertion in debug builds.

WD Fusion no longer applies default permissions on repaired files when the source zone does not have permissions for content. This could previously affect the outcome of a repair from an S3 zone to an HDFS zone.

Repair tasks that were initiated before, and that completed after the failure of an HDFS NameNode were previously not marked as Done, resulting in the accumulation of tasks in the Fusion server.

Under specific conditions, a non-writer Fusion node that becomes a writer can execute agreements that have already been processed by the previous writer node. For this to occur, a non-writer node that has been re-started must have switched to the writer role after observing continued agreement execution by a writer which has not failed. This can happen as a result of intermittent network failure or very large garbage collection pauses at that writer node.

Because the outcome of agreement re-execution is dependent on intervening activity in the underlying storage, the impact of this issue is indeterminate, however it can result in data loss.

Under some scenarios, querying the /fs/repair/task endpoint results in an HTTP 500 response code instead of the expected task information.

Cilent applications that reference the file system with a URI that contains both an authority and a path that has a whitespace character could fail to initialize their file system reference correctly. This could result in job failure or failure to access specific file system content.

When configured with the fusion:// URI scheme, clients that do not provide an authority in the URI may fail to obtain a file system reference correctly.

The WD Fusion 2.10.4 release introduced a serialization change the prevented wire-level compatibility with environments operating WD Fusion 2.10.3.x versions.

WD Fusion IHC servers were not defaulting socket configuration to accommodate high rates of connection re-establishment. This can be worked around by configuration of kernel properties to modify those defaults, and the 2.10.5 release does not require that configuration change to be made in the operating system.

A WD Fusion environment that has only a single replication rule may not succeed in removing that rule on request.

Installation of the MapR client library can result in an incorrect attempt to modify the cluster hadoop-env.sh script.

Actions taken on Cloudera parcel installation may not correctly set the libraries referenced by Cloudera Navigator.

Azure to Azure replication in WD Fusion 2.10.3.1 would not accommodate files larger than 4MB due to a lack of support for appends. See Known Issue.

Default exclusion rules did not include .hive-staging correctly.

Installation of WD Fusion for S3 can fail if the endpoint requires SSL certificates.

Swift installer requires explicit configuration for HTTPS endpoints and improved validation.

A specific set of actions in the installer could result in failure of validation for EMR environments only.

A failure of the repair API to provide all repair tasks from the non-initiating node when a mix of historical and ongoing repairs are present.

Resolution for a benign, but reported exposure to CVE-2014-0114 by updating the version of commons-beanutils to 1.9.3.

If the ui.hostname property is left at default 0.0.0.0 setting, it can affect the ability for the dashboard to display graph information for the node.

If the WD Fusion UI is configured to run on a host that is not that of the Fusion server, links generated to that node from other UI instances are incorrect, using the hostname from the Fusion server.

Silent installer for Fusion with Hive in a non-managed environment was not catered for.

In-place upgrade from 2.10.2 in a mixed HDP/S3 configuration can fail to detect Java on an unmanaged node.

In-place upgrade from 2.10.2 in a mixed HDP/S3 configuration can fail to detect the user correctly on an unmanaged node.

Change in text displayed during client installation for LocalFS variant.

Correct links to documentation for WD Hive installation.

Kerberos settings tab is not required for EMR or S3 variants.

Custom UI settings when a custom port is used may not update correctly.

Using custom setting for UI port and host may result in redirection to the UI server’s IP address rather than hostname.

The Fusion UI does not show table or database information if a database exists with whitespace in its location path.

If Kerberos and SSL for the cluster manager are enabled after WD Fusion installation, it can affect operation of truncate table.

The Hive Stack applied in HDP dpeloyments did not have correct permissions on the configuration file.

Under specific conditions, a non-writer Fusion node that becomes a writer can execute agreements that have already been processed by the previous writer node. For this to occur, a non-writer node that has been re-started must have switched to the writer role after observing continued agreement execution by a writer which has not failed. This can happen as a result of intermittent network failure or very large garbage collection pauses at that writer node.

Because the outcome of agreement re-execution is dependent on intervening activity in the underlying storage, the impact of this issue is indeterminate, however it can result in data loss.

Under some scenarios, querying the /fs/repair/task endpoint results in an HTTP 500 response code instead of the expected task information.

Cilent applications that reference the file system with a URI that contains both an authority and a path that has a whitespace character would fail to initialize their file system reference correctly. This could result in job failure or failure to access specific file system content.

Fixes issue when a hive table name contains a whitespace character

Fixes issue with combination of boxcars and fast bypass mechanism

Fixes issue where Fusion client raises a Null pointer exception

Fixes issue triggered by particular combination of system restart order

Corrects safety valve setting used to configure cluster for use of replicated metastore

Fixes NPE issue during handshake between replicated metastore and Fusion server

Fixes minor display issue with installer at Cloudera Impala configuration step

Fixes issue with combination of Sentry grants and service restart order

Fixes issue exhibited by read timeout exception from hiveserver2

Resolves problem with visibility of databases during consistency check

Corrected permissions on runtime directories

Hive plugin installation in a Kerberos-enabled CDH deployment automates keytab generation if required

Refines wording on upgrade process

Removes information on backup option from config files

fusion_env.sh updated to improve referencing of parcels

Resolution for issues with Java heap exhaustion

Resolves Fusion server panic on repair task cancellation

Fixes to non-interactive mode for talkback.sh script

Improved logic for pre-rename pull activity

Post-installation script improvements

NPE in Fusion server resolved for task retrieval API

Fix for authentication when SaSL-based auth and impersonation of Kerberos users is enabled

Simple extension to allow talkbacks to be processed without assuming use of /tmp

Correction to replication issue specific to HDP+Azure

Cleanup of Fusion metadata folders on Swift

No longer default to default region

Swift object store listing no longer ignores pseudo-subdirectories

Change in the configured location for log data is accommodated by talkback

Umbrella ticket for a set of individual issues related to management of CDH parcels

Simple correction to JAR naming

Fix for failure to delete replication rules

HDP 2.6.2 support

RHEL 7.4

Resolves issue specific to NetApp deployments

Documentation fix for approach to disabling ciphers deemed insecure in some environments

Fixes to post-install script that is run on upgrade or uninstallation

Allows the user to run talkback non-interactively by providing it with a configuration file

Optional extra check during talkback

Talkbacks load and respect the value of this parameter

Talkback accommodates a command mechanism dependent on the zone type it is run under

Corrected transfer progress listener

New defaults for exclusion patterns applied to replication rules

Improved IHC log output on SSL failure

Also corrects descriptions in operation

Stack packaging changes to account for non-root hardened installations

Improves log feedback

Confirmed interoperability with a variety of S3 compatible systems

Documentation improvement

Documentation improvement

Retain content within table boundaries

The possibility to configure Kerberos is restored on the security step during UI installation

Use of SPARK_CLASSPATH is deprecated and should not be used

Document need for JAVA_HOME to be set for user that runs the Fusion server process

Correction for MapR 5.0

Correct installation issue related to core-site modification for unmanaged installs

Remove addition of fs.defaultFS in the core-site safety valve

Correct instruction to restart Cloudera Manager service at specific step in Hive installation

The Fusion UI will now show tables or databases when there is any Database with a space in the location path.

Correct Impala client parcel download links

Corrected issue when installing Fusion with Hive plugin on HDP 2.6 single node cluster, the installer failed to detect the hive service has completed the restart.

The installer should query Ambari as to whether it is managing Kerberos, and only request kadmin credentials if this is the case

Clusters that do not have KMS enabled should not provide the KMS toggle in the Replicated path addition or edit page.

S3 listings no longer fail under some virtual directory configurations

Improved referencing existing configuration from core-site.xml

UI portion of FUS-4182

Fixed minor navigation issue

Improved startup during silent installation

Swift installer allows specification of HTTP or HTTPS

Minor revision

Corrected error relating to update of HTTP port setting with blanks

Zone icon in the secondary header should not be present

Disable validation of hive settings if no kadmin credentials are provided

Cluster graph bandwidth limit arrows now showing on enterprise license when required

Closed with spinner removal

Correction to when after going back from step 2 → step 1, especially when kadmin validation has not been performed, then the Next button is inactive.

Upgraded Jackson version

Properly configure the default_realm in krb5.conf

Fusion no longer asks for confirmation to start a node

The page instructing users to grant permissions has been removed, no longer required.

Related to FUS-3654 from 2.10.3

Path is more prominent in the breadcrumb

Corrected permissions issue when applied in non-default location

Corrected: The "Select All" checkbox is disabled the first time you navigate to the Replicated Rules tab after adding a rule on the remote zone. If you navigate away from the page and back, the checkbox is enabled.

False by default when no rules

No longer log warnings from comment lines

Hard link to el_x_ parcels

Support new UNMANAGED_ASF type

Inject the plugins in the position marked by fusion-ui-client.

New case when S3 plugin checks to see if it is running on EC2 first

Consistent representation

Better solution for displaying transfers identified as "stalled".

UI no longer fails to validate the object storage if the endpoint requires SSL certs

Warning message if Kerberos is configured, but the Cluster is not kerberized

Improved flow for localfs installation

Improved description of setting in UI

Improved order

Fixed intermittent display issue

The Repair button on bulk repair or repair tag was not enabled after all the options are filled

WANdisco Fusion is interoperable with a wide variety of systems, including Hadoop distributions, object storage platforms, and cloud environments.

WANdisco Fusion is architected for maximum compatibility and interoperability with applications that use standard Hadoop File System APIs. All applications that use the standard Hadoop Distributed File System API or any Hadoop-Compatible File System API should be interoperable with WANdisco Fusion, and will be treated as supported applications. Additionally, Fusion supports the replication of content with Amazon S3 and S3-compatible objects stores, locally-mounted file systems, and NetApp NFS devices, but does not require or provide application compatibility libraries for these storage services.

The talkback script used to capture environment and log information can be run in a non-interactive mode where user input is not required during execution. (FUS-3454, FUS-3649)

WD Fusion 2.10.3 overrides these modified Hadoop settings

WD Fusion will honor application intentions when specifying a file system location other than that of the Fusion-enabled local file system.

Ambari-based installations of the Hive Replicated Metastore accommodate nodes with multiple network interfaces or multiple hostnames. (HIVE-471)

Changes introduced in CDH 5.11 are incorporated into WD Fusion. (HIVE-482)

Multiple WD Fusion Replicated Hive Metastore instances can be deployed for redundancy and high availability. (HIVE-346, HIVE-227)

Responsiveness of the Hive CLI is improved. (FUS-3851, REPL-32, HIVE-363)

Hive commands that include "ADD IF NOT EXISTS PARTITION" will function correctly. (HIVE-439)

The WD Hiveserver2 Template can be removed succesfully. (HIVE-469)

Environments with very large numbers of Hive databases can be listed with good performance in the WD Fusion UI. (HIVE-497)

If Hive is configured to use a scratch directory through hive.start.cleanup.scratchdir, it will now be recreated with the correct permissions. (HIVE-283)

Hive installation succeeds when Ambari hostname labels differ from the actual hostnames. (HIVE-429)

Restarting the hosts on which the Hive Replicated Metastore and Hiveserver2 Template reside now results in the associated services starting. (HIVE-437, HIVE-446, HIVE-442)

WD Hive stacks and init scripts account for the removal of paths created under /var/run on RHEL7. (HIVE-450, HIVE-464)

Repair functionality is no longer limited by the maximum integer value. (HIVE-452)

Fix provided for a failure to progress during execution of an alter table add constraint operation. (REPL-39, HIVE-455)

Enabling Kerberos after installation functions correctly. (HIVE-473, REPL-48)

The WD Fusion UI now shows all databases in their correct location. (HIVE-503)

The display name of the Hiveserver2 Template is shortened. (HIVE-348, HIVE-311)

Ubuntu 16.04 is a supported operating system. (FUI-4611)

Conditions of significant load no longer risk application level timeouts due to contention over coordinated activities. (FUS-3576, FUS-3799, FUS-3810)

Better handling of failed TCP connections during final stages of file content transfer. (FUS-3837)

Internal buffers sizes used for notification event and request events can be configured. (FUS-3849)

WANdisco advises customers to avoid the use of OpenSSL as it can exhibit a memory leak. (FUS-3867)

Failed transfers no longer remain in incomplete status indefinitely. Stalled transfers are shown. (FUS-3833, FUS-3869)

Installation of new cluster nodes via Ambari now honors dependencies for the Fusion client libraries correctly. (FUS-3887)

The performance of replication to Amazon S3 endpoints is enhanced, enabled in part by the introduction of a new configuration parameter (fs.fusion.s3.transferThreads). (FUS-3978, FUS-3984)

Replication rule removal does not trigger a failure of the Fusion server. (FUS-3987)

Memory use during consistency check and regular operation is improved. (FUS-4013, DCO-709, FUS-4027, FUS-4050, FUS-3489, FUS-3733, FUS-4032)

The performance of the consistency check is improved. (FUS-2469)

The FUS-3433 known issue in prior releases is resolved, allowing metadata modifications to files that are recently moved from non-replicated to replicated locations to behave as expected. (FUS-3433)

Timeouts are applied on execution of REST APIs to prevent clients waiting indefinitely. (FUS-3477)

Fusion can accommodate S3 implementations that do not provide copyObject functionality. (FUS-3588)

The Fusion server REST API is available regardless of the state of quorum in the system. (FUS-3619)

Ingesting very large numbers of files to a replicated directory at once is handled with better performance than previous releases. (FUS-3620, FUS-2469, FUS-3663, FUS-3673, FUS-3631)

When performing repair to an Amazon EMR zone, Fusion will perform correctly regardless of the state of existing files in EMR. (FUS-3701)

Ill-defined bandwidth limit policies do not result in Fusion server failure. (FUS-3716, FUS-3756)

The rename with options method in the Hadoop FileSystem API is supported. (FUS-3739, REPL-24)

iNode modifications made on a NetApp instance during repair are ignored to allow completion. (FUS-3850)

A potential startup deadlock condition related to license checks has been corrected. (FUS-3863)

Additional configuration options are provided for replication with S3 endpoints:

(FUS-3902)

Internal configuration defaults have been modified to improve performance. (FUS-4099, FUS-4070)

WD Fusion 2.10.3 improves startup time for processing agreed proposals. (FUS-4071, FUS-4086)

Changes made to interaction with the S3 API to avoid triggering conditions that can result in read-after-write inconsistency, affecting correct outcomes for replication. (FUS-3908, FUS-3936)

Backoffs applied to conflicting proposals are limited. (DCO-698)

WD Fusion 2.10.3 resolves the known issue in 2.10 where a consistency check that is triggered from a non-writer node may never complete. (FUS-2675, FUS-3684, FUS-3775)

Introduced new executor thread pools to avoid processing starvation from long-running activities. (FUS-3799)

Fusion’s ability to scale effectively with extremely large numbers of outstanding agreements is improved. (FUS-3974, REPL-38)

There is no longer a requirement to run scheduled consistency checks for regular operation of WD Fusion. 2.10.3 turns off these checks by default. (FUS-4041, FUS-4003, FUS-3615)

The Fusion client library will determine whether the scheme of the URI used to reference the underlying file system indicates that coordination via Fusion is not required, and act accordingly. This improves interoperability with distcp in particular. (FUS-1970)

Information returned from the /fusion/fs/transfers REST API no longer reports negative values. The repair API is similarly improved. (FUS-3219, FUS-3291)

Multiple renames occurring in quick succession on a single file no longer result in extraneous files in non-originating zones. (FUS-3439)

Installation ensures that Oozie classpaths are configured to reference WD Fusion client libraries as required. (FUS-3659, FUS-3690)

Hadoop clusters that do not have ACL support no longer result in excessive logging of aclStatus messages. (FUS-3704, FUS-3657)

Improved resilience of the snapshot repair mechanism. (FUS-3881)

The behavior of the zone properties API when updating global exclusions has been corrected. (FUS-3741)

Configurations where a Hadoop cluster uses S3 as the underlying file system are improved. (FUS-3039)

WD Fusion interoperates with HDFS in a best-effort mode by default to avoid stalling from failed data pipeline errors. (FUS-3597)

File content transfer that fails for external reasons no longer marks those transfers as incomplete. (FUS-3605)

Non-recoverable failures during content transfer are no longer retried. (FUS-3697)

The default exclusions that apply to replication rules remain set regardless of changes to global zone properties. (FUS-3741)

Post-installation scripts allow for group names that include whitespace characters. (FUS-3774)

Upgrades in Ambari-based environments behave correctly. (FUS-3794)

Non-recoverable failures on file pulls (due to a source file being removed or modified) no longer result in retry behavior. (FUS-3805, FUS-3833)

Repairs operate correctly with FACL information. (FUS-3825)

Ambari-based installations performed to a non-standard directory work as expected. (FUS-4025)

Classpath issues that occurred when using Nifi are resolved. (FUS-3852)

Improvements to status representation of file transfers. (FUS-3869, FUS-3833)

WD Fusion can be reinstalled on a node that has had a previous installation in place that has not been uninstalled correctly.

Origin zone file renames do not affect successful content transfer.

Consistency check performance is improved.

Apache Commons Collection 3.2.2 is used in preference to 3.2.1. (FUS-1803)

Apache Spark2 follows a different model for third-party JAR integration than Spark. Fusion integrates correctly with Spark2. Resolution also applies to Accumulo. (FUS-4031)

Plugins can read configuration available on the CLASSPATH. (FUS-3815)

The service fusion-ihc-server-xxx_x_x restart command will indicate success or otherwise in response. (FUS-4039)

Customization of the java.util.logging.FileHandler.pattern can be performed. (FUS-4045)

Replication operation while a zone has been set to inbound is no longer affected by server restarts. (FUS-3721)

Impala 3.0 parcels are now provided or Ubuntu. (FUS-3992)

Links to HTTPS-enabled nodes are represented correctly. (FUI-4468, FUI-4517)

WD Fusion has added support for the following new platforms since Fusion 2.10:

Additionally, the platform support for IBM BigInsights 4.3 has been removed with the non-release of that version of BigInsights.

WANdisco Fusion is interoperable with a wide variety of systems, including Hadoop distributions, object storage platforms, and cloud environments.

WANdisco Fusion is architected for maximum compatibility and interoperability with applications that use standard Hadoop File System APIs. All applications that use the standard Hadoop Distributed File System API or any Hadoop-Compatible File System API should be interoperable with WANdisco Fusion, and will be treated as supported applications. Additionally, Fusion supports the replication of content with Amazon S3 and S3-compatible objects stores, locally-mounted file systems, and NetApp NFS devices, but does not require or provide application compatibility libraries for these storage services.

As a major release, Fusion 2.10 introduces incompatibilities with the network protocols and storage formats used by prior versions. Please contact WD Fusion support for information on the upgrade mechanism appropriate for your environment.

WD Fusion 2.10 adds support for replicating data efficiently from Network File Systems (NFS) for NetApp devices to any mix of on-premises and cloud environments. This feature allows data replication at any scale from NFS to other Fusion zones.

The WD Fusion user interface now presents a logical view of the Fusion operational components, Fusion zones and bandwidth limit policies in place of the physical map of locations. This makes it easier to observe the deployment of complex solutions and navigate directly to detailed views of individual item configuration.

An improvement has been made to the mechanism used by the HDFS and HCFS client library to detect when a working Fusion server is unavailable. The improvement allows clients to bypass the Fusion server when needed without waiting for a TCP connection loss or timeout.

Fusion 2.10 can be configured to replicate from storage system locations that do not provide write access for the identity used by the Fusion server.

Fusion configuration options now include custom S3 endpoints so that replication can occur to non-AWS S3 providers. Additionally, when Fusion is hosted in AWS EC2, replication can occur to an S3 endpoint that is in a region other than where the Fusion services reside.

The Fusion repair feature allows the transfer of initial content between Fusion zones that have not previously replicated, and can be used as a mechanism to perform once-off replication that remains consistent with other replication activity. Repair has been enhanced significantly in Fusion 2.10, including the following:

The WANdisco Fusion user interface can be accessed over HTTPS, and for that configuration to be performed independently of other SSL configuration.

You can choose to install WD Fusion 2.10 in a location other than the default /opt/wandisco. See Custom Location Installations.

Fusion 2.10 can operate in an environment where one Fusion zone does not allow inbound network connectivity. This is typical for a secured on-premises deployment, where it may be difficult to modify or establish corporate firewall rules to allow inbound TCP connections to the Fusion services.

ACL replication can be enabled to allow changes from local- and remote-originated zones to be replicated. ACL information will be represented in consistency check results as appropriate.

Among a range of minor improvements to logged information, Fusion 2.10 adds the ability to log the identity of the proxy user for which requests are made.

This feature introduces a mechanism to quickly prevent applications from using Fusion when interacting with the underlying file system, without the need to make configuration changes. The fusion.replicated.dir.exchange configuration property in core-site.xml specifies the location under which a directory named bypass can be created to trigger this. Subsequent client activity in that cluster will bypass coordination through Fusion.

The API to track the status of transfers under a replicated directory now allows that tracking to be limited to a subdirectory of a replicated location.

Fusion 2.10 can be installed as a non-root user with sufficient permissions (sudo tar, sudo ambari-server, sudo cp).

The Fusion 2.10 client library for HDFS and HCFS compatibility ensures that classpath conflicts do not occur with any client application, allowing Fusion to be used by applications that use alternative versions of the Guava and Netty libraries.

Periods of extended network outage between Fusion zones can be accommodated by limits that allow Fusion servers to identify a sidelined node, ensuring that operation of other nodes can continue in its absence. Prior to this release, bringing a sidelined node back into operation was a completely manual process. Fusion 2.10 adds a mechanism by which sidelined nodes can be recovered and participate in ongoing activity.

To support operation in environments where minimal security privileges must be allocated, the Fusion server can now operate as a principal without HDFS superuser privileges.

A configuration option (fusion.client.coordinate.read) is provided to allow coordination of open() requests, which by default is false.

This release provides an API by which a preferred writer node can be specified for a given replicated path. The writer node is the Fusion server instance responsible for executing modifications to the local zone’s metadata via the file system API.

License expiration allows continued operation for a short grace period (by default one month for production licenses), during which notifications are presented to the administrator about the license expiry. This is in addition to the existing warnings provided prior to license expiration.

Additionally, license expiry does not halt operation of the Fusion server, which remains available to service activities that occur in non-replicated locations.

WD Fusion has added support for the following new platforms since Fusion 2.9:

WANdisco certifies the interoperability of Fusion with a wide variety of systems, including Hadoop distributions, object storage platforms, cloud environments, and applications.

WANdisco Fusion is architected for maximum compatibility and interoperability with applications that use standard Hadoop File System APIs. All applications that use the standard Hadoop Distributed File System API or any Hadoop-Compatible File System API should be interoperable with WANdisco Fusion, and will be treated as supported applications. Additionally, Fusion supports the replication of content with Amazon S3 and S3-compatible objects stores, locally-mounted file systems, and NetApp NFS devices, but does not require or provide application compatibility libraries for these storage services.

To simplify the process of pre-deployment testing, WD Fusion is supplied with an evaluation license (also known as a "trial license"). This type of license imposes limits:

Customers entering production need a production license file for each node. These license files are tied to the node’s IP address. In the event that a node needs to be moved to a new server with a different IP address customers should contact WANdisco’s support team and request that a new license be generated. Production licenses can be set to expire or they can be perpetual.

For large deployments, Unlimited licenses are available, for which there are no usage limits.

If one or more of your nodes are down or expired, you can still perform a license update by updating the license file on all nodes, via the UI. In this situation, the license upgrade cannot be done in a coordinated fashion, from a single node, but it can be completed locally if done on all nodes.

The following manual procedure should only be used if the above method is not available, such as when a node cannot be started - maybe caused by ownership or permissions errors on an existing license file. If you can, use the procedure outlined above.

RHEL 6 x86_64
RHEL 7 x86_64
Oracle Linux 6 x86_64
Oracle Linux 7 x86_64
CentOS 6 x86_64
CentOS 7 x86_64
Ubuntu 12.04LTS
Ubuntu 14.04LTS
SLES 11 x86_64

Where’s Java?
Although WD Fusion only requires the Java Runtime Environment (JRE), Cloudera and Hortonworks may install the full Oracle JDK with the high strength encryption package included. This JCE package is a requirement for running Kerberized clusters.
For good measure, remove any JDK 6 that might be present in /usr/java. Make sure that /usr/java/default and /usr/java/latest point to an instance of java 7 version, your Hadoop manager should install this.

Ensure that you set the JAVA_HOME environment variable for the root user on all nodes. Remember that, on some systems, invoking sudo strips environmental variables, so you may need to add the JAVA_HOME to Sudo’s list of preserved variables.

-u The maximum number of processes available to a single user.
-n The maximum number of open file descriptors.

For optimal performance, we recommend both hard and soft limits values to be set to 64000 or more:

RHEL6 and later: A file /etc/security/limits.d/90-nproc.conf explicitly overrides the settings in security.conf, i.e.:

Ambari/Pivotal HD and Cloudera manager will set various ulimit entries, you must ensure hard and soft limits are set to 64000 or higher. Check with the ulimit or limit command. If the limit is exceeded the JVM will throw an error: java.lang.OutOfMemoryError: unable to create new native thread.

Comment out requiretty in /etc/sudoers
The installer’s use of sudo won’t work with some linux distributions (CentOS where /etc/sudoer sets enables requiretty, where sudo can only be invoked from a logged in terminal session, not through cron or a bash script. When enabled the installer will fail with an error:

keystore
A keystore (containing a private key / certificate chain) is used by an SSL server to encrypt the communication and create digital signatures.

truststore
A truststore is used by an SSL client for validating certificates sent by other servers. It simply contains certificates that are considered "trusted". For convenience you can use the same file as both the keystore and the truststore, you can also use the same file for multiple processes.

Workaround
Upgrade to Java 8u60 or greater, or ensure WD Fusion is able to make use of OpenSSL libraries instead of JDK. Requirements for this can be found at http://netty.io/wiki/requirements-for-4.x.html FUS-3041

Transport Layer Security (TLS) and its predecessor, Secure Socket Layer (SSL) are widely adopted protocols that are used transfer of data between the client and the server through authentication and encryption and integrity.

Recent research has indicated that some of the cipher systems that are commonly used in these protocols do not offer the level of security that was previously thought.

In order to stop WD Fusion from using the disavowed ciphers (DES, 3DES, and RC4), use the following procedure on each node where the Fusion service runs:

The time required to complete a deployment of WD Fusion will in part be based on its size, larger deployments with more nodes and more complex replication rules will take correspondingly more time to set up. Use the guide below to help you plan for deployments.

Of course, this is a guideline to help you plan your deployment. You should think ahead and determine if there are additional steps or requirements introduced by your organization’s specific needs.

See the deployment checklist for a list of the TCP ports that need to be open for WD Fusion.

See the Knowledge Base for instructions on setting up manual Kerberos settings. You only need these in special cases as the steps have been handled by the installer. See Manual Updates for WD Fusion UI Configuration.

See the above Knowledge Base article for instructions on setting up auth-to-local permissions, mapping a Kerberos principal onto a local system user. See the KB article - Setting up Auth-to-local.

Custom Fusion Request Port
You can provide a custom TCP port for the Fusion Request Port (also known as WD Fusion client port). The default value is 8023.

Strict Recovery
Two advanced options are provided to change the way that WD Fusion responds to a system shutdown where WD Fusion was not shutdown cleanly. Currently the default setting is to not enforce a panic event in the logs, if during startup we detect that WD Fusion wasn’t shutdown. This is suitable for using the product as part of an evaluation effort. However, when operating in a production environment, you may prefer to enforce the panic event which will stop any attempted restarts to prevent possible corruption to the database.

Push Threshold

Chunk Size
The size of the 'chunks' used in file transfer.

Once all settings have been entered, click Next step.

The default behavior for WD Fusion is to fix all replication to the Hadoop Distributed File System / hdfs:/// URI. Setting the hdfs-scheme provides the widest support for Hadoop client applications, since some applications can’t support the available "fusion:///" URI they can only use the HDFS protocol. Each option is explained below:

This option is available for deployments where the Hadoop applications support neither the WD Fusion URI nor the HCFS standards. WD Fusion operates entirely within HDFS.

This configuration will not allow paths with the fusion:/// uri to be used; only paths starting with hdfs:/// or no scheme that correspond to a mapped path will be replicated. The underlying file system will be an instance of the HDFS DistributedFileSystem, which will support applications that aren’t written to the HCFS specification.

This differs from the default in that while the WD Fusion URI is used to identify data to be replicated, the replication is performed using HDFS itself. This option should be used if you are deploying applications that can support the WD Fusion URI but not the Hadoop Compatible File System.

Benefits of HDFS.
The following advanced options provide a number of low level configuration settings that may be required for installation into certain environments. The incorrect application of some of these settings could cause serious problems, so for this reason we strongly recommend that you discuss their use with WANdisco’s support team before enabling them.

Once you have entered the information click Validate.

Once WD Fusion has been installed on all data centers you can proceed with setting up replication on your HDFS file system. You should plan your requirements ahead of the installation, matching up your replication with your cluster to maximize performance and resilience. The next section will take a brief look at a example configuration and run through the necessary steps for setting up data replication between two data centers.

The following steps are used to start replicating HDFS data. The detail of each step will depend on your cluster setup and your specific replication requirements, although the basic steps remain the same.

Installing clients via RPM/Deb packages no longer automatically stop and repackage Oozie. If Oozie was running prior to the client installation, you will need to manually stop Oozie, then Oozie setup command -

If possible, complete these actions through Ambari.

Solr will use a shared replicated directory.

The procedure for Cloudera CDH is much the same as the one for HDP, provided above. Note that path differences:

When running the installer, first set the following environmental variable:

This will change the installation directory from the default to the one that you provide, e.g.

The above example would install fusion-ui-server into /CustomInstallLocation/fusion-ui-server. Also, the WD Fusion server and IHC server will be installed under /CustomInstallLocation/fusion/server and /CustomInstallLocation/fusion/ihc/server/ respectively.

If you run with the FUSION_PREFIX, an additional line will appear on the summary screen of the installer:

By default local parcels are stored on the Cloudera Manager Server:/opt/cloudera/parcel-repo. To change this location, follow the instructions in Configuring Server Parcel Settings.

The location can be changed by setting the parcel_dir property in /etc/cloudera-scm-agent/config.ini file of the Cloudera Manager Agent and restart the Cloudera Manager Agent or by following the instructions in Configuring the Host Parcel Directory.

If you are replacing an existing package that was installed using a parcel, once the new package is activated you should remove the old package through Cloudera Manager. Use the Remove From Host button.

HttpFS is a server that provides a REST HTTP gateway supporting all HDFS File System operations (read and write). And it is interoperable with the webhdfs REST HTTP API.

While HttpFS runs fine with WD Fusion, there is an issue where it may be installed without the correct class paths being put in place, which can result in errors when running Mammoth test scripts.

Once the parcel has been installed and HDFS has been restarted, the HttpFS service must also be restarted. Without this follow-on restart you will get missing class errors. This impacts only the HttpFS service, rather than the whole HDFS subsystem.

Follow these steps on the client to configure impersonation without enabling cluster security.

For more detailed steps see Uninstalling Fusion - Cloudera.

For more detailed steps see Uninstalling Fusion - Ambari.

Connect WD Fusion to Amazon Web services, then set up replication between your on-premise cluster and AWS.
See Connect WD Fusion to Amazon Web Service.

See Bare Bones EC2 deployments for the required IAM roles needed for setting up for metering when installing a basic EC2 instance, instead of the default CFT-based installation, through the Amazon market place.

This is defined as the total amount of data held in AWS in directories that are replicated, excluding those files that are excluded from replication through regex-based filtering. WD Fusion therefore measures the amount of data under replication each hour.