Release Notes

WD Fusion 2.9.4 Build 324 - 21 March 2017

This point release provides support for CDH 5.9 & 5.10.

Available Packages

WD Fusion currently supports the following versions of Hadoop:

  • CDH 5.2.0 - 5.10.0
  • HDP 2.1.0 - 2.5.0
  • MAPR 4.0.1 - 5.0.0
  • Pivotal HD 3.0.0 - 3.4.0
  • IOP (BigInsights) 2.1.2 - 4.2

Available packages
The trial download includes the installation packages for CDH and HDP distributions only.

System Requirements

Before installing, ensure that your systems, software and hardware meet the requirements found in our online user guide at docs.wandisco.com/bigdata/wdfusion/

Certified Platforms / DBMS & DConE Support:

WANdisco certifies the interoperability of Fusion with a wide variety of systems, including Hadoop distributions, object storage platforms, cloud environments, and applications.

  • HDP 2.1.0 - 2.5.0
  • Ambari 1.6, 1.7, 2.0, 3.1
  • Pivotal HD 3.0 - 3.4
  • IBM BI 2.1.2 - 4.2
  • CDH 4.4, 5.2 - 5.10.0
  • EMC Isilon 7.2, 8.0
  • MapR M4.0, M5.0-M5.2
  • DConE 1.3
  • MySQL (Hive MetaStore)
  • Oracle BDA
  • Amazon EMR 4.6, 4.7.1, 5.0
  • Google Compute
  • Microsoft Azure

Client Applications Supported:

  • Apache Hive
  • Apache Spark SQL
  • Impala (some limitations - contact WANdisco)
  • Apache HBase
  • Apache SQOOP
  • Apache Flume
  • Apache Kafka
  • Apache Storm
  • Presto
  • Apache Drill

Installation

Find detailed installation instructions in the user guide at
- docs.wandisco.com/bigdata/wdfusion/install.html#procedure

Upgrades from an earlier version:

An upgrade from Fusion 2.9.1 can follow the procedure at
http://docs.wandisco.com/bigdata/wdfusion/upgrade.html

Previous Release Notes:

You can view the release notes for previous releases in the release archive:
- docs.wandisco.com/bigdata/wdfusion/archive.html

New Features

  • Support has been added for CDH 5.9 and CDH 5.10. FUI-4231, FUI-4232

Fixes

  • We've fixed an issue that prevented the completion of some WD Fusion installers when WebHDFS had been disabled. FUI-4231
  • Fixed an issue in the Ambari Stack that was causing Oracle Linux to not be correctly recognized as a supported Linux distribution. FUS-3551

Known Issues

  • If you see hard to diagnose write errors in Fusion's logs, please note that this can occur due to a limitation in Hadoop when small clusters (< 10 datanodes) are placed under heavy load. Read our knowledgebase article Small Cluster Limitations. FUS-2603

WD Fusion 2.9.3 Build 309 - 15 February 2017

New Features

Fusion 2.9.3 is a minor new release. Fatures added include:

  • Relocatable installer - FUS-3192
  • WD Fusion server can operate as HDFS non-superuser - FUS-2920
  • Support for EMC Isilon 8 - FUS-3232
  • Support for IBM BigInsights 2.1.2 - FUS-3302
  • SSL support for Fusion UI - FUS-4007

Fixes

  • Avoid incorrectly induced replication of recursive delete for non-preserving repair. FUS-3190
  • Fusion server restart may neglect agreement dependencies. FUS-3396
  • Non-replicated directory write no longer blocks on network partition. FUS-3121
  • Non-replicated directory write no longer blocks on quorum unavailable. FUS-3004
  • Improved IHC connection management. FUS-3189
  • Talkbacks no longer attempt SSH to KDC as root. FUS-3192
  • New edge node installation via Ambari. FUS-3228
  • Eliminate excessive org.jacaoco logging. FUS-3346
  • Debian package installation without RPM_INSTALL_PREFIX. FUS-3367
  • HttpFS server classpath corrected for CDH parcel installations. FUS-3201
  • Kerberos talkback warnings eliminated for non-secure clusters. FUS-3210
  • /fusion/fs/repairs REST endpoint fixed. FUS-3215
  • fusion-server.log available on AWS CFT deployment. FUS-3141
  • Upgrade from 2.9.0 no longer references "DUMMY" property. FUS-3163
  • Improved Java validation on UI restart. FUI-3349
  • Removed potential for renamed file under active transfer to duplicate single block in non-originating zone. FUS-3421

Known Issues

  • [Consistency Check Tool]
    Currently, a consistency check that is triggered from a non-writer node may never complete. FUS-2675
  • [Deploying with Apache Slider] Apache Slider does not appear to use the Default Hadoop classpath. As a result you should refer to our procedure for making it work - Running with Apache Slider. FUI-4107
  • [CDH Parcel upgrade]
    There's currently an issue where upgrading via CDH parcel fails after missing some alternatives. Use the following workaround:
    1. Get the parcel files and put them in place. You can either download through the UI installer or move the file in, e.g.
      mkdir /tmp/FUS293/
      cd /tmp/FUS293/
      wget "http://fusion.example.host.com:8083/ui/parcel_packages/FUSION-<version>-cdh5.<version>.parcel"
      wget "http://node01-example.host.com:8083/ui/parcel_packages/FUSION-<version>-cdh5.<version>.parcel.sha"
    2. On the Cloudera Manager server, copy the parcel and sha files to location that Cloudera Manager scans for new parcels, e.g.:
      cd /opt/cloudera/parcel-repo/
      cp /tmp/FUS2913/FUSION-2.9.3.<version>-cdh5.7.0-el6.parcel .
      chown cloudera-scm. FUSION-2.9.3.<version>-cdh5.7.0-el6.parcel
      cp /tmp/FUS2913/FUSION-2.9.3.<version>-cdh5.7.0-el6.parcel.sha .
      chown cloudera-scm. FUSION-2.9.3.<version>-cdh5.7.0-el6.parcel.sha
    3. Log in to Cloudera Manager; click the parcels menu link and then click "Check for New parcels".
      The "FUSION" parcel should now have a new "2.9.3.<version>-cdh5.7.0" version listed.
    4. Click on the Distribute link for the "2.9.3.<version>-cdh5.7.0" version of the FUSION parcel.
    5. Deactivate the "2.9.3.<version>-cdh5.7.0" version of the FUSION parcel, choosing the "Deactivate Only" option, since we need to make a change to core-site.xml before restarting.. (This is required because of a bug in the FUSION parcel, which prevents /etc/alternatives symlinks being created if there is an existing version of the FUSION parcel already activated).
    6. Activate the "2.9.3-<version>-cdh5.7.0", choosing the "Restart" option to restart all necessary services.
    7. cp /tmp/FUS2913/FUSION-2.9.3.<version>-cdh5.7.0-el6.parcel* /opt/cloudera/parcel-repo/
      chown cloudera-scm: /opt/cloudera/parcel-repo/FUSION-2.9.3.<version>-cdh5.7.0-el6.parcel*
    8. FUS-3418

    WD Fusion 2.9.1. Build 221 - 3 November 2016

    Important!
    Please note that unlike most point releases, 2.9.1 is not compatible with the previous release version. For this reason you can't use the upgrade procedure for moving an existing deployment to 2.9.1. You must perform a fresh installation in order to move to version 2.9.1.

    New Features

    • Support for Cloudera 5.8. FUI-3304
    • Support MapR 5.2.0. FUS-2870
    • Support for Amazon EMR 5.0.0, 4.7.0, and 4.6.0. FUS-2822
    • Keystone v3 support for Openstack Swift. IT-2089
    • This release introduces a technical preview of improvements to uni-directional network to allow WD Fusion nodes to operate more efficiently without the ability to accept incoming TCP connections. This functionality is suitable for non-production environments, and by default is disabled. FUI-2278
    • Inclusion of an additional plugin for replication with Amazon S3 for environments that do not require EMR compatibility. FUS-2026
    • Allow custom endpoints for S3 replication via fs.fusion.s3.endpoint. FUS-2803
    • Snapshot-based Repair mechanism for HCFS. FUS-2895
    • Progress notifications on repairs. FUS-2517
    • Installer now supports re-installations. FUI-528
    • License update after expiry. FUS-2897

    Fixes

    • Elimination of potential resource leakage from poorly-behaved file system clients. FUI-3432
    • Better detection of Java version across runtime components. FUS-2508
    • Solr index backup possible with Fusion. FUS-1169
    • Sample code added for bypassing fusion client to write to HDFS. See Admin Guide. FUS-1246
    • Cloudera Navigator Metadata Server now able to access Fusion. FUS-2134
    • Cloudera 5.7 file browser now able to access Fusion. FUS-2633
    • Potential Memory Leak with Apache Solr resolved. FUS-2728
    • Task file cleanup for /var/log/fusion/server/tasks. Files are cleaned up after 30 days by default. Period is configurable as task.removal.cleanup.interval (in ms). Files can also be cleaned externally without side-effect. FUS-2773
    • Log rotation on the fusion-ui.log not occurring. See Apache Appenders article for time-based configuration. FUI-3726
    • Permissions exceptions that are non-recoverable are not retried. FUS-2431, FUS-2070
    • Improvements to transaction dependency calculations prevents non-writers from sidelining under load. FUS-2421

    Known Issues

    • WD Fusion does not support the truncate command, the file system goes inconsistent. FUS-3022
    • Currently, the s3 plugin does not store permissions. FUS-2846
    • On RHEL6 / CDH 5.8 deployments, Cloudera won't restart so installation fails and Cloudera won't restart. The cloudera-scm-server.log says the error is:
      016-10-18 02:21:07,064 ERROR main:com.cloudera.server.cmf.Main: Server failed.
      java.lang.NoSuchMethodError: com.google.common.collect.Range.closedOpen(Ljava/lang/Comparable;Ljava/lang/Comparable;)Lcom/google/common/collect/Range;
      Currently, the out-of-date guava-11.0.2.jar is currently linked to the installer. [FUS-3045]

      Workaround

      In the unlikely event that you are impacted by this problem, you can work around it by removing the symlink to guava-11.0.2.jar.
    • There's currently the potential for install failure as a result of each Fusion service performing a different check for Java's home directory. In future releases we will apply a unified and consistent set of tests. As a workaround to this problem, customers can define a JAVA_HOME export to point to an installation of Java. HIVE-327
    See previous release for continuing known issues.

    WD Fusion 2.9.0.1 Build 18 August 2016

    Fixes

    • Fixed issue where running setfacl with -x flag could result in a null pointer exception. FUS-2783
    • Fixed issue where AclConverter was failing to setName for some supported Hadoop packages. FUS-2789


    WD Fusion 2.9 Build 88 - 2 August 2016

    New features

    • IHC servers use the value of the fusion.server property in the core-site.xml file to contact the WD Fusion server.
    • On retry, repairs start from the point at which failure occurred. FUS-2664
    • Significant performance improvements for non-replicated paths. FUS-2494
    • Repair functionality is more resilient to intermittent network failure. FUS-2874
    • Support added for IBM BigInsights 4.2 in Fusion Core. FUS-2639 FUI-2947

    Known Issues

    • [Fixed in 2.9.1] For installations to IBM Openstack/Swift storage, we currently only support Keystone 2.0. We will be adding support for Keystone 3.0 in a future release. FUI-3025
    • [Fixed in 2.9] Currently it's not possible to install WD Fusion with SSL enabled, as the installer is unable to see that the server has started. We'll fix this issue in the next release. The workaround is to enable the SSL after the installation has been completed. FUI-3095
    • [Fixed in 2.9] A rounding error can, in some specific cases, result in a Consistency Repair reporting either negative time taken or successful completion with 100% remaining. FUS-2492
    • [Fixed in 2.9] For IBM Big Replicate/IBM Hive deployments, there is currently a limitation that prevents the deployment of the Ambari stack, through WD Fusion UI. The "Deploy Stack" button will not work and you will see a "Failed to deploy Ambari stack: Stack service does not exist..." warning. The workaround for this is to complete the installation of the Stack through Ambari. FUI-3068

    Hive Metastore Plugin Known Issues

    • WD Hive Metastore and WD Hive Fusion Plugin version 1.1 does not currently support replication between different Hadoop distributions or distribution versions. FUI-3325
    • [Fixed in 2.9.1] There is a known issue that may appear if Hive contains two indexes that have the same name that then apply to a column of the same name (even though the indexes apply to different tables in different databases). In this scenario, repairs will fail. You should perform the following workaround. If you need to repair a database that contains Index tables then you will need to repair them in stages as follows: Workaround for FUI-3272
    • There is an issue with FSClient disconnections when trying to create a table with partitions using beeline. See Workaround for FUS-2592
    • There is an issue that prevents the Hive consistency check's repair tool from repairing individual table elements (i.e. Indexes or partitions). The repair of full tables works as expected. FUI-3322
    • [Fixed in 2.9] There is currently a bug where the Consistency Checker tree browser won't display children. FUI-3091
    • [Fixed in 2.9] [Cloudera-based deployements] When installing the Hive Metastore plugin, you must create a folder /etc/wandisco/hive on the Metastore host that you specify. This folder must have owner hive:hive and the hive user must be able to access and write to this location. FUI-3121