logo

LiveData For MultiCloud

Version 2.0.1 Build 379 - 12 March 2019
Last updated: 17 April 2019

1. Welcome

1.1. Product overview

LiveData for MultiCloud provides LiveData replication across different types of object storage services. It combines the functionality and, going forward, replaces the Plugins for Live S3 and Live Azure Blob Storage. It brings the added benefit of support for the coordination of activities between Fusion zones that reference underlying object storage services that do not use the same interface in each zone.

1.2. Documentation guide

This guide contains the following:

Welcome

This chapter introduces this user guide and provides help with how to use it.

Release Notes

Details the latest software release, covering new features, fixes and known issues to be aware of.

Concepts

Explains how LiveData for MultiCloud through WANdisco Fusion uses WANdisco’s LiveData platform.

Installation

Covers the steps required to install and set up LiveData for MultiCloud into a WANdisco Fusion deployment.

Operation

The steps required to run, reconfigure and troubleshoot LiveData for MultiCloud.

Reference

Additional LiveData for MultiCloud documentation, including documentation for the available REST API.

1.2.1. Symbols in the documentation

In the guide we highlight types of information using the following call outs:

The alert symbol highlights important information.
The STOP symbol cautions you against doing something.
Tips are principles or practices that you’ll benefit from knowing or using.
The i symbol shows where you can find more information, such as in our online Knowledgebase.

1.3. Contact support

See our online Knowledgebase which contains updates and more information.

If you need more help raise a case on our support website.

1.4. Give feedback

If you find an error or if you think some information needs improving, raise a case on our support website or email docs@wandisco.com.

2. Release Notes

2.1. LiveData for MultiCloud 2.0.1 Build 379

12 March 2019

WANdisco is pleased to present a minor update of LiveData for MultiCloud. This 2.0.1 release provides full functionality for Amazon S3 and Azure Blob Storage replication in a Live Data environment, with consistent replication of S3 objects and Azure Blob among multiple buckets and containers, and features to perform consistency checks and to bring buckets and containers to a consistent state for initial transfer of data.

The LiveData for MultiCloud 2.0.1 release is entirely focussed on bug fixes and performance improvements.

LiveData for MultiCloud 2.0.1 supports the latest version of WANdisco Fusion, 2.12.4.

2.1.1. System Requirements

Before installing or upgrading, ensure that your systems, software, and hardware meet the requirements. The requirements for WANdisco Fusion are found in the prerequisites checklist in the user guide.

LiveData for MultiCloud is tested on a limited number of operating systems then the main product. These are:

  • RHEL 6 x86_64

  • RHEL 7 x86_64

  • Oracle Linux 6 x86_64

  • Oracle Linux 7 x86_64

  • CentOS 6 x86_64

  • CentOS 7 x86_64

  • Ubuntu 16.04LTS

  • SLES 11 x86_64

  • SLES 12 x86_64

Unsupported:

  • Ubuntu 14.04LTS

  • Ubuntu 18.04LTS

2.1.2. Resolved Issues

  • WD-MXC-642 - S3 object storage class and ACLs are not replicated.

  • WD-MXC-650, WD-MXC-778 - Large numbers of parallel uploads could cause the proxy to run out of threads. This is now configurable.

  • WD-MXC-652 - Outbound connections only configurations were not respected.

  • WD-MXC-682 - Performance optimization for upload of small files.

  • WD-MXC-719 - Networking settings to enable/disable direct pull apply to the whole rule.

  • WD-MXC-731 - Resolving conflicting dependencies.

  • WD-MXC-738 - Replication page doesn’t display replication rule.

  • WD-MXC-764 - Don’t coordinate requests which resulted in non-success.

  • WD-MXC-794 - HTTPS requests are proxied to HTTP for non replicated buckets.

  • WD-MXC-800 - Not all big files were replicated when IHC pull is enabled.

2.1.3. Other Improvements

  • WD-MXC-683 - Object deletion from AWS Java SDK not replicated.

  • WD-MXC-718 - Proxy responds 403 SignatureDoesNotMatch to some S3 requests.

  • WD-MXC-732 - Use version id to improve behavior with S3 eventual consistency.

  • WD-MXC-744 - Disabled list object V2 support for Ceph Mimic.

  • WD-MXC-755 - Requests which originated from neither S3 nor Azure clients failed.

  • WD-MXC-757 - Replication doesn’t work on Ubuntu 16 MultiCloud installation.

  • WD-MXC-763 - Object tagging not replicated from Ceph to AWS S3.

  • WD-MXC-773 - Maximum stream buffer size configured for IHC pull.

  • WD-MXC-775 - Repair doesn’t work in vbucket with direct pull disabled.

  • WD-MXC-781 - Race condition which could have led to objects/blobs not being replicated.

  • WD-MXC-799 - S3 POST Object not replicated.

  • WD-MXC-634 - Repair does not work for blobs of type append or page.

  • WD-MXC-651 - Wrongly excluded /.* for both types of Oracle Object Storage.

  • WD-MXC-679 - Create StorageDriver needs to be synchronized.

  • WD-MXC-695 - Proxy responds with Internal Server Error to unsupported S3 operations requests.

  • WD-MXC-697 - Bucket API operation does not work between two replicated OOS Classic buckets.

  • WD-MXC-698 - IBM COS - AWS S3 repair fails when direct pull is disabled.

  • WD-MXC-702 - Using Minio + local storage - delete object fails via proxy.

  • WD-MXC-712 - Logging the storage connection string should not print the access key.

  • WD-MXC-717 - Azure download/upload is using S3 constant for part of a calculation.

  • WD-MXC-743 - Don’t log error message if the cc task is not found.

  • WD-MXC-751 - Proxy duplicates log entries for requests.

  • WD-MXC-758 - Allow inbound and outbound connections by default.

  • WD-MXC-765 - POST Object causes NPE.

  • WD-MXC-769 - Filter out unrelated request headers.

  • WD-MXC-770 - Validate features for Ceph Mimic provider.

  • WD-MXC-774 - Request coordination doesn’t happen during IHC pull.

  • WD-MXC-787 - Failed Consistency Check task not marked as done.

  • WD-MXC-806 - Allow fast SSL in Proxy using Conscrypt library.

  • WD-MXC-807 - Add configurable incoming connection limit.

  • WD-MXC-812 - IHC plugin should not create client instance per request.

  • WD-MXC-720 - The proxy log should use same rolling names as Fusion logs.

2.1.4. Known Issues

Some S3 concepts do not map directly to an environment that replicates buckets—e.g., bucket creation or deletion, and operations that act on objects by unique identifier provided by the S3 endpoint. There are also some aspects of the S3 API that are not yet supported—e.g., browser-based object upload/creation.

  • WD-SPX-48 - POST Object restore allows restoration of a temporary copy of an archived object. It is not supported by the Fusion Plugin for Live S3.

  • WD-SPX-104 - AWS KMS SSE is not yet supported.

  • WD-SPX-123 - Multi-chunk payloads are not yet supported.

  • WD-SPX-125 - Buckets configured with MFA serial security are not yet supported.

  • WD- MXC-187 - POST is an alternative to PUT for browser-based uploads. The Fusion Plugin for Live S3 does not support Post object replication with a virtual bucket.

  • WD-MXC-188 - Deletion of specific object versions is not supported because independent S3 buckets assign different version identifiers to replicated instances of an object.

  • WD-MXC-190 - Deletion of tagging for a specific object version is not supported because independent buckets assign different version identifiers to replicated instances of an object.

  • WD-MXC-207 - Complete Multipart Upload won’t replicate when Server-Side Encryption (SSE) is Enabled.

  • In Azure, Blob versioning will not be considered since there is no concept of Azure Blob versioning.

  • Non-replicated bucket must be created under the default-endpoint-url storage.

  • Server-Side Encryption will not be coordinated because of difference in implementation in AWS S3 and Azure Blob Storage. However, when the encryption is configured, server-side uploaded objects will be automatically encrypted at rest.

  • WD-SPX-498 - Credentials limitations For replicated buckets/containers with assigned virtual bucket: only the accessKey/secretKey pair which is defined in virtual bucket configuration will be used to access the proxy. Other accessKey/secretKey will be refused as invalid credentials (even though it may be valid with the underlying storage).

For non-replicated buckets/containers: Any accessKey/secretKey pair that appears in any virtual bucket configuration can be used to access the proxy. The same pair will be used when request is passed to the underlying storage.

  • WD-MXC-335 - There are a number of characters that are currently not supported in Azure blob storage objects. Please ensure that you do use any of the following characters in Azure Blob Storage names.

    Problem when put via command line cli - ( ) = $ ,      brackets, equals, dollar, comma
Problem when put via Java SDK -     ! * ( ) = $,       exclamation, asterisk, brackets, equals, dollar, comma

  • WD-MXC-629 - Manual configuration adjustment needed in HA setup. This issue will be addressed in the next release.

    1. On each node (where IHC is installed), edit /etc/wandisco/fusion/ihc/server/<distro>/core-site.xml so that the property fusion.server should be updated to contain a comma separated list of all Fusion Servers in the zone.

    2. Save the changes and restart the node.

  • WD-MXC-706 - Currently, MultiCloud does not take eventual consistency into account for S3 storage, resulting in an issue where a second 'put' of a file is pooled to the target zone when it is not completely put on the source zone. This results in the following:

    • If the files are unique (no overwrites) - there is no issue (as S3 has read-after-write consistency).

    • If there are overwrites, then we have this problem and it could be removed by enabling the versioning on source bucket.

  • WD-MXC-725 - The proxy requires credentials for non-replicated buckets/containers to be able to interact with them.

  • WD-MXC-791 - If Delete Multiple Objects is issued before some of the deleted objects are fully replicated, it might result in the object not being deleted in the non-originating zones.

  • WD-MXC-801 - Authentication via an IAM role is not yet supported.

  • WD-MXC-828 - In Ceph the Delete Bucket Metrics Operation, deletes the bucket if the bucket is empty.

2.2. LiveData for MultiCloud 2.0 Build 310

19 December 2018

WANdisco is pleased to present the first release of LiveData for MultiCloud. This 2.0 release provides full functionality for Amazon S3 and Azure Blob Storage replication in a Live Data environment, with consistent replication of S3 objects and Azure Blob among multiple buckets and containers, and features to perform consistency checks and to bring buckets and containers to a consistent state for initial transfer of data.

LiveData for MultiCloud 2.0 supports the latest version of WANdisco Fusion, 2.12.3.

2.2.1. Highlighted New Features

Installation

Supports installation of LiveData for MultiCloud using a standard RPM- or DEB-based installation process

User Interface

Simplified User Interface for common tasks such as replication rules management (create, list, update, delete), consistency check, consistency report and a "Make Consistent" tool.

High Availability

Supports Fusion high availability deployment in a MultiCloud environment. This allows for the Live S3 Plugin and its Proxy to operate in a High Availability configuration.

Improved Scalability

Horizontal scalability capabilities for MultiCloud environments where large number of simultaneous network connections try to access the service

Replication across firewall

Enables Fusion to replicate data from on-premises to Cloud (AWS or Azure) in a DMZ configuration, where on-premises Fusion Server can see and make connections to AWS / Azure Fusion server. However, the AWS / Azure Fusion server is completely firewalled from the on-premises network. No inbound connections are allowed on any port.

Enhanced Selective Replication

Replication rules will provide inclusion & exclusion patterns. User can be selective about content to be replicated at a finer level of granularity than a bucket.

Virtual Bucket

Allows applications to interact with the underlying object storage using the virtual bucket abstraction which maps into the actual S3 bucket or Azure Blob container.

Support Replication between

  • S3 ↔ S3

  • Azure Blob Storage ↔ Azure Blob Storage

  • S3 ↔ Azure Blob Storage

Non-Replicated Bucket / Container

Support for non-replicated bucket / container accessed via LiveData for MultiCloud

Dynamic configuration

API support for dynamic configuration for specific virtual bucket

Selective replication

API support for selective replication with inclusion and exclusion patterns

API Support

Add link to the sub-section of the supported AWS S3 and Azure Blob Storage APIs

Storage end-points

Supports the following S3 Compatible storage providers & Azure Blob Storage

  • AWS S3

  • AWS Snowball

  • Azure Blob Storage

  • IBM Cloud Object Storage (COS)

  • Dell EMC Elastic Cloud Storage

  • Ceph Object Storage (v0.94.x Hammer Release)

  • Dell Virtustream Storage Cloud

  • Oracle Cloud Object Storage

  • Oracle Cloud Object Storage Classic

  • Alibaba Cloud Object Storage Service (OSS)

  • Pure FlashBlade Object Storage

  • HGST (Western Digital) ActiveScale Object Storage

  • Minio Gateway for Azure Blob Storage

  • Scality Zenko CloudServer

2.2.2. System Requirements

Before installing or upgrading, ensure that your systems, software, and hardware meet the requirements. The requirements for WANdisco Fusion are found in the User Guide at http://docs.wandisco.com/bigdata/wdfusion/2.12/#_prerequisites_checklist.

LiveData for MultiCloud is tested on a limited number of operating systems then the main product. These are:

  • RHEL 6 x86_64

  • RHEL 7 x86_64

  • Oracle Linux 6 x86_64

  • Oracle Linux 7 x86_64

  • CentOS 6 x86_64

  • CentOS 7 x86_64

  • Ubuntu 16.04LTS

  • SLES 11 x86_64

  • SLES 12 x86_64

Unsupported:

  • Ubuntu 14.04LTS

  • Ubuntu 18.04LTS

2.2.3. Known Issues

Some S3 concepts do not map directly to an environment that replicates buckets—e.g., bucket creation or deletion, and operations that act on objects by unique identifier provided by the S3 endpoint. There are also some aspects of the S3 API that are not yet supported—e.g., browser-based object upload/creation.

  • WD-SPX-48 - POST Object restore allows restoration of a temporary copy of an archived object. It is not supported by the Fusion Plugin for Live S3.

  • WD-SPX-104 - AWS KMS SSE is not yet supported.

  • WD-SPX-123 - Multi-chunk payloads are not yet supported.

  • WD-SPX-125 - Authentication via an IAM role is not yet supported.

  • WD-SPX-125 - Buckets configured with MFA serial security are not yet supported.

  • WD- MXC-187 - POST is an alternative to PUT for browser-based uploads. The Fusion Plugin for Live S3 does not support Post object replication with virtual bucket.

  • WD-MXC-188 - Deletion of specific object versions is not supported because independent S3 buckets assign different version identifiers to replicated instances of an object.

  • WD-MXC-190 - Deletion of tagging for a specific object version is not supported because independent buckets assign different version identifiers to replicated instances of an object.

  • WD- MXC-207 - Complete Multipart Upload won’t replicate when Server-Side Encryption (SSE) is Enabled.

  • In Azure, Blob versioning will not be considered since there is no concept of Azure Blob versioning.

  • Non-replicated bucket must be created under the default-endpoint-url storage.

  • Server-Side Encryption will not be coordinated because of the difference in implementation in AWS S3 and Azure Blob Storage. However, when the encryption is configured, server-side uploaded objects will be automatically encrypted at rest.

  • WD-SPX-498 - Credentials limitations (Impacts both S3 and Azure implementations) -
    For replicated object containers (buckets, etc) with assigned virtual bucket: only the accessKey/secretKey pair which is defined in the virtual bucket configuration will be used to access the proxy. Other accessKey/secretKey will be refused as invalid credentials (even though it may be valid with the underlying storage).

    For non-replicated object containers: Any accessKey/secretKey pair that appears in any virtual bucket configuration can be used to access the proxy. The same pair will be used when a request is passed to the underlying storage.

  • WD-MXC-335 - There are a number of characters that are currently not supported in Azure blob storage objects. Please ensure that you do use any of the following characters in Azure Blob Storage names.

    Problem when put via command line cli - ( ) = $ ,      brackets, equals, dollar, comma
Problem when put via Java SDK -     ! * ( ) = $,       exclamation, asterisk, brackets, equals, dollar, comma

  • WD-MXC-642 - S3 object storage class and ACLs are currently not replicated. This issue will be addressed in the next release.

  • WD-MXZ-629 - Manual configuration adjustment needed in HA setup. This issue will be addressed in the next release.

    1. On each node (where IHC is installed), edit /etc/wandisco/fusion/ihc/server/<distro>/core-site.xml so that the property fusion.server should be updated to contain a comma-separated list of all Fusion Servers in the zone.

    2. Save the changes and restart the node.

  • WD-MXC-706 - Currently, MultiCloud does not take eventual consistency into account for S3 storage, resulting in an issue where a second 'put' of a file is pooled to the target zone when it is not completely put on the source zone. This results in the following:

    • If the files are unique (no overwrites) - there is no issue (as S3 has read-after-write consistency).

    • If there are overwrites, then we have this problem and it could be removed by enabling the versioning on source bucket.

3. Concepts

Familiarity with the following concepts will improve your use of LiveData for MultiCloud.

3.1. Product concepts

Virtual Bucket

Applications should be able to interact with the underlying object storage using the virtual bucket abstraction which will mapping into the actual bucket or container

Bucket

Associated storage name for AWS S3.

Objects

S3 object has data stored in the bucket.

Container

Associated storage for Microsoft Azure

Blobs

A Binary Large Object is a collection of binary data stored as a single entity in a container.

Table 1. Container names

S3

Azure Blob

WDOS (WANdisco Object Store)

Bucket

Container

Bucket

Object

Blob

Objects

3.1.1. MultiCloud Proxy

Applications use LiveData for MultiCloud via one or more MultiCloud-compatible endpoints that it provides in the MultiCloud proxy, which is the runtime component that proxies access to underlying MultiCloud-compatible storage services on behalf of LiveData for MultiCloud.

3.1.2. WANdisco Fusion Plugin

The Fusion Plugin for MultiCloud extends the WANdisco Fusion server to support the operation of the MultiCloud proxy. Coordinate activities among multiple MultiCloud proxy instances with the WANdisco Fusion Plugin that is installed for each WANdisco Fusion server.

3.1.3. Dependencies

A replication directory is associated with a replication rule that is used for all coordination against operations performed on that replication directory, and with one S3/Azure Blob bucket per zone.

3.2. Architecture

MultiCloud Architecture
Figure 1. MultiCloud Plugin Architecture

LiveData for MultiCloud plugin will provide three runtime components:

  • A MultiCloud Proxy - a process operating as a forward/reverse proxy that intercepts client requests and submits them for coordination

  • A MultiCloud Plugin for Fusion - plugin implementing FusionPlugin interface, runs inside Fusion server process, hooks into Fusion to coordinate intercepted requests

  • A MultiCloud Plugin for IHC - plugin implementing StorageFactory<CoordinatedFs> interface, allows the IHC to pull from object stores We will implement ‘Direct Pull’ as a default approach and IHC will be a configurable option and will be plugged-in as an when enabled in Fusion Core (FUS-5481)

3.2.1. Direct pull

MultiCloud Architecture
Figure 2. MultiCloud Direct pull

3.2.2. IHC pull

MultiCloud Architecture
Figure 3. MultiCloud IHC pull

3.3. Supported Functionality

3.3.1. S3 features

The Fusion Plugin for MultiCloud supports a broad range of S3-compatible features, including:

Virtual-hosted style URLs

where the virtual bucket name is part of the domain name in a virtual-hosted–style URL. For example: http://virtualbucket.s3proxyhost.yourdomain.com.

Path style URLs

where the bucket name is not included in the domain unless as a region-specific endpoint. For example: http://s3proxyhost.yourdomain.com/virtualbucket.

Client request signature validation

Requests are validated for correct credentials before being executed against an underlying S3-compatible storage. Versions 2 and 4 of AWS signatures are supported:

  1. AWS V4 Path Style

  2. AWS V4 Virtual Hosted Style

  3. AWS V4 Pre-signed URLs

  4. AWS V2 Path Style

  5. AWS V2 Virtual Hosted Style

  6. AWS V2 Pre-signed URLs

AWS signatures

The Fusion Plugin for Live S3 supports versions 2 and 4 of AWS signatures in Path Style. To enable virtual hosted style, follow the below steps:

  1. Add a tag <path-style-access>false</path-style-access> in

    /etc/wandisco/fusion/plugins/live-s3/proxy-plugin-site.xml

  2. Add a tag for IHC pull <direct-pull-enabled>true</direct-pull-enabled> in

    /etc/wandisco/fusion/plugins/live-s3/proxy-plugin-site.xml.

  3. Add the DNS entry for virtual-hosted–style URL in /etc/hosts. Open /etc/hosts file and add the virtual-hosted–style URL,

    Ex: 0.0.0.0 virtualbucket.s3proxyhost.yourdomain.com
Default config values in AWS CLI
Signature type

v4

Path addressing style

auto

Run the commands below to set the required signature type and addressing style:

  • Signature type v4: aws configure set default.s3.signature_version s3v4

  • Signature type v2: aws configure set default.s3.signature_version s3

  • Enable Path Style: aws configure set default.s3.addressing_style path

  • Enable Virtual Style: aws configure set default.s3.addressing_style virtual

Payload Options

S3 Payloads can be signed or unsigned in a single chunk. Multi-chunk payloads are not yet supported.

Server-side Encryption

AES-256 server-side encryption is supported. AWS KMS SSE is not yet supported.

IAM role authentication

IAM role-based authentication is not yet supported.

MFA Serial authentication

Buckets configured with MFA serial security are not yet supported.

3.3.2. Azure Blob Feature

  • AES-256 server-side encryption is not yet supported.

3.3.3. WANdisco LiveData Features

The Fusion Plugin for MultiCloud provide support for additional features beyond those of standard S3 and Azure Blob endpoints as a result of supporting LiveData functionality. These include:

Consistency Check

Determine and report on differences in content between replicated buckets/containers.

Make Consistent

The Make Consistent tool can fix issues of inconsistency between content among multiple buckets/containers.

Support for:

  • MultiCloud installation using RPM installer.

  • MultiCloud installation using Debian installer.

  • S3 APIs

  • Azure Blob APIs

  • Replication between

    • S3 ↔ S3

    • Azure Blob ↔ Azure Blob

    • S3 ↔ Azure Blob

  • API support for dynamic configuration for specific virtual bucket.

  • API support for selective replication with inclusion and exclusion pattern.

  • non-replicated bucket accessed via MultiCloud

  • S3 Compatible storage providers:

    • AWS S3 (AWS_S3)

    • Snowball (AWS_SNOWBALL)

    • Virtustream (VIRTUSTREAM)

    • Activescale (ACTIVESCALE)

    • IBM Cloud Object Storage (IBM_COS)

    • Alibaba Cloud Object Storage Service (ALIBABA_OSS)

    • Oracle Cloud Object Storage (ORACLE_OBJECT_STORAGE)

    • Oracle Cloud Object Storage Classic (ORACLE_OBJECT_STORAGE_CLASSIC)

    • Minio Gateway for Azure Blob Storage (MINIO_AZURE)

    • Ceph Object Storage (v0.94.x Hammer Release) (CEPH_HAMMER)

    • Pure FlashBlade Object Storage (PURESTORAGE)

    • Dell EMC Elastic Cloud Storage

    • Scality S3 Server Object Storage (SCALITY_S3)

3.3.4. Support for Cloud REST APIs

See the API chapter in the Reference section.

3.4. Deployment models

3.4.1. Use Cases for LiveData for MultiCloud

Use the Fusion Plugin for MultiCloud with S3/Azure for a variety of reasons, including:

Heterogenous storage

Your applications may benefit from accessing cloud data in different storage systems, perhaps both on-premises and in the cloud, or with multiple cloud providers to take advantage of cost arbitrage.

Multi-geo applications

Applications that operate in multiple, geographically-separate locations can work with a local S3/Azure endpoint in each location, and ensure that each location has access to the same data.

Improving performance

By having a local replica of a storage container/bucket, applications can operate more efficiently than if they need to work with data that are not physically close.

Improved availability

The impact of the failure of a single source of S3/Azure objects can be eliminated by having a strongly-consistent replica of those objects in another source.

Regulatory compliance

Your compliance needs may require that you store multiple copies of data in different locations, or with different service providers.

LiveData for MultiCloud automates the replication of data across S3 buckets/Azure containers and ensures that they store exactly the same information, even when applications change content in any of the replicated containers.

4. Installation

4.1. Pre-requisites

Along with the standard product requirements that you can find on the WANdisco Fusion Deployment Checklist, you also need to ensure that you have available:

  • Java 1.8

  • Fusion 2.12.3. (Packages for S3 or HDI)

  • Minimum 2 Fusion zones inducted

  • Credentials for accessing the to-be-replicated S3/Azure Blob endpoints.

    • For AWS S3, this will be in the form of an Access Key and Secret Access Key.

    • For Azure blob, this will be in the form of an Account name and Account Key.

  • Details of the endpoint URL by which applications access the S3/azure service normally, e.g.for S3 s3-us-west-1.amazonaws.com & for Azure blob core.windows.net Access to the hosts on which each WANdisco Fusion server is operating for the purpose of installation

  • The names of the Wandisco inc. Fusion zones across which replication will occur.

Also note that when using the S3 proxy service, authentication can be limited depending on your configuration:

  1. The Fusion Bucket and MultiCloud Bucket must be different.

  2. For buckets with an assigned virtual bucket, only the accessKey/secretKey pair which is defined in the vbucket configuration can be used to access the proxy. Other accessKey/secretKey will be refused as invalid credentials (even though they might actually be valid with the underlying storage).

  3. For non-replicated buckets, any accessKey/secretKey pair that appears in any vbucket configuration can be used to access the proxy. The same pair will be used when request is passed to the underlying storage.

MultiCloud Plugin must be installed in all zones - All nodes that will replicate MultiCloud objects must have the MultiCloud plugin installed.

4.1.1. System Requirements

Before installing or upgrading, ensure that your systems, software, and hardware meet the requirements. The requirements for WANdisco Fusion are found in the User Guide at http://docs.wandisco.com/bigdata/wdfusion/2.12/#_prerequisites_checklist.

4.2. Installation

Install the Fusion Plugin for MultiCloud using a standard RPM- or DEB-based installation process. Configure the plugin with simple command-line tools or manual changes to configuration files that are specific to the plugin.

4.2.1. Locate installation components

The following default locations are used during installation.

Table 2. MultiCloud Plugin

Directories

Location

MultiCloud installation location

/opt/wandisco/fusion/plugins/multicloud/

Configuration directory

/etc/wandisco/fusion/plugins/multicloud/
Table 3. MultiCloud Server

Directories

Location

Installation directory

/opt/wandisco/multicloud-proxy/

Configuration directory

/etc/wandisco/multicloud-proxy/

Log directory

/var/log/wandisco/multicloud-proxy/

There are RPM & DEB files that provide installable components for Centos and Ubuntu respectively:

For Centos

  • fusion-multicloud-plugin-asf-2.5.0-2.0.0.0_253.noarch.rpm

  • fusion-multicloud-ihc-plugin-asf-2.5.0-2.0.0.0_253.noarch.rpm

  • fusion-multicloud-proxy-asf-2.5.0-2.0.0.0_253.noarch.rpm

For Ubuntu

  • fusion-multicloud-plugin-asf-2.5.0_2.0.0.0—​253_all.deb

  • fusion-multicloud-ihc-plugin-asf-2.5.0_2.0.0.0-253_all.deb

  • fusion-multicloud-proxy-asf-2.5.0_2.0.0.0-253_all.deb

Obtain the files so that you can distribute them to the appropriate hosts in your deployment for Fusion. The plugin files needs to be installed on each Fusion server host in your deployment, while proxy files needs to be installed on each machine where you intend to operate a MultiCloud Proxy.

4.2.2. Install the plugin

Ensure you have read all known issues and pre-requisites and file name changes before beginning installation.

  1. Run the installer with appropriate filesystem permissions.

    Verifying archive integrity... All good.
Uncompressing WANdisco Live MultiCloud..................

    ::   ::  ::     #     #   ##    ####  ######   #   #####   #####   #####
   :::: :::: :::    #     #  #  #  ##  ## #     #  #  #     # #     # #     #
  ::::::::::: :::   #  #  # #    # #    # #     #  #  #       #       #     #
 ::::::::::::: :::  # # # # #    # #    # #     #  #   #####  #       #     #
  ::::::::::: :::   # # # # #    # #    # #     #  #        # #       #     #
   :::: :::: :::    ##   ##  #  ## #    # #     #  #  #     # #     # #     #
    ::   ::  ::     #     #   ## # #    # ######   #   #####   #####   #####

You are about to install WANdisco Live MultiCloud version 2.0.0.0

Do you want to continue with the installation? (Y/n)

    The installer will perform an integrity check and confirm the product version that will be installed. Enter "Y" to continue the installation.

  2. Follow the instructions to add all required Fusion servers

This plugin requires knowledge of all the fusion servers for this zone.
You can add Fusion servers and their request ports one by one with a ‘hostname:port’ format.
The hostname should be of a form that will allow the local proxy server to contact the specified hostname.
No checking is currently performed to validate any server addresses entered here.

Multiple entries can be added by separating entries with a ‘,’.
Adding the local fusion server to the fusion server list
The current fusion server list is:
 ‘<your.fusion.hostname>:<port>’
 Enter ‘P’ to print the current fusion server list.
 Entering a blank value ends the fusion server collection.
 Host and port to add (format: ‘hostname:port’)  []:

  1. Confirm the hostname and port for the virtual server. Enter the details or press enter to go with the default value, e.g. 0.0.0.0:8071.

    Enter the proxy server listen host and port [0.0.0.0:8071]:

  2. Enter hostname for the Proxy Server(s).

    Enter DNS compatible host names for the Proxy Server.
Multiple entries can be added by separating with a ‘,’.
The current proxy virtual host list is:
 ‘localhost’
 ‘127.0.0.1’

 Enter ‘P’ to print the current proxy virtual host list.
 Entering a blank value ends proxy virtual host collection.
 New Virtual Host []:

  3. Next, decide whether you want SSL encryption for the proxy

    Do you wish to enable proxy ssl? (Y/n): n

  4. Installation will then occur:

    Installing requested plugin components...
 wd-multicloud-plugin-2.0.0.0.tar.gz ... Done
 multicloud-plugin-ui-server-2.0.0.0-dist.tar.gz ... Done
 fusion-multicloud-plugin-hdi-3.6_2.0.0.0-260_all.deb ... Done
 fusion-multicloud-ihc-plugin-hdi-3.6_2.0.0.0-260_all.deb ... Done
 fusion-multicloud-proxy-hdi-3.6_2.0.0.0-260_all.deb ... Done
All requested components installed.

  5. A series of post-installation steps will run.

    Running additional post install actions...

Running fusion server plugin configuration

Running proxy server configuration
Moving existing ‘/etc/wandisco/multicloud-proxy/proxy-server_env.sh’ to ‘/etc/wandisco/multicloud-proxy/proxy-server_env.sh.22838.backup’

    A restart of the fusion-server is required, enter "y" to continue the installation.

    Restarting fusion-server is required as part of plugin activation
Do you wish to restart fusion-server now? (y/N) y
Restarting WANdisco Fusion Server: fusion-server
Stopped WANdisco Fusion Server process 5849 successfully.
Started WANdisco Fusion Server process successfully.

  6. Confirm that the installation has completed by checking the Plugin status on the Fusion UI settings tab.

    MultiCloud Architecture
    Figure 4. MultiCloud Plugin Status

4.2.3. Create a replication rule

LiveData for MultiCloud uses a replication rule to coordinate activities against each virtual bucket. Create a replication rule using the Fusion UI or API. See Create a rule.

4.2.4. Enable SSL encryption

SSL for Fusion

MultiCloud supports HTTP or HTTPS access. SSL should be enabled to offer clients the option of communicating with the plugin via HTTPS. Specifying yes to this setting will require further information on the keystore path, password and keytype.

Storage Type

  • The type of the storage for this zone. (S3, AZURE_BLOB).

  • S3: Type should be S3

  • AZURE_BLOB: Type should be Azure

Without SSL
configure-proxy-server : without ssl
./configure-proxy-server
Enter the proxy server listen host [0.0.0.0]:
Enter the proxy server listen port [8071]:
Do you want to enable ssl (yes/no)?
  [If yes, you need to provide the keystore path and encrypted password]: no
Enter Proxy server's DNS compatible host names (ie., virtual host) [localhost,127.0.0.1]: marip02-vm4.bdauto.wandisco.com
Enter the fusion server host and port [host:port]: marip02-vm4.bdauto.wandisco.com:8023
Is Fusion SSL enabled (yes/no)?
  [If yes, you need to provide the keystore path and encrypted password]: no
Please specify the appropriate storage type for the proxy server to accept api request:
1) S3
2) AZURE
#? 1

Storage type selected: S3
 ------------------------------------------------------------------------------------------------

 **** MultiCloud Proxy server details ****

Proxy server listen host: 0.0.0.0
Proxy server listen port: 8071
Proxy server SSL: false
MultiCloud proxy server's DNS compatible host names: localhost,127.0.0.1,marip02-vm4.bdauto.wandisco.com

 **** Fusion server details ****

Fusion server host and port: marip02-vm4.bdauto.wandisco.com:8023

Fusion server SSL: false


 **** Virtual bucket ****


    Storage Type: S3


 ------------------------------------------------------------------------------------------------

Which user should MultiCloud Proxy run as? [root]:
Which group should MultiCloud Proxy run as? [root]:
Enter the minimum memory(-Xms) for MultiCloud Proxy (in MB) [512]:
Enter the maximum memory(-Xmx) for MultiCloud Proxy (in MB) [1024]:

 -------------------------------------------------

 **** MultiCloud Proxy environment details ****

 Run as User: root
 Run as Group: root

 Minimum memory: 512m
 Maximum memory: 1024m

 -------------------------------------------------

Do you confirm the MultiCloud proxy server configuration details (yes/no): yes

MultiCloud proxy server configuration done successfully, start 'multicloud-proxy-server' to load the configuration

--------------------------------------------------------------------------------------------------------
 Note: You can edit the configuration values anytime in: /etc/wandisco/multicloud-proxy/core-site.xml, /etc/wandisco/multicloud-proxy/proxy-server-site.xml
       The multicloud-proxy-server must be restarted for the changes to take effect
  -------------------------------------------------------------------------------------------------------
With SSL
configure-proxy-server with ssl
[root@nnpxy02-vm0 ~]# /etc/wandisco/multicloud-proxy/configure-proxy-server

Enter the proxy server listen host [0.0.0.0]:
Enter the proxy server listen port [8071]:
Do you want to enable ssl (yes/no)?
  [If yes, you need to provide the keystore path and encrypted password]: yes
Enter the keystore file path: /etc/wandisco/multicloud-proxy/ssl/wandisco.ks
Please enter the password to be encrypted
> ********
log4j:WARN No appenders could be found for logger (com.wandisco.security.crypto.CryptoFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Enter Proxy server's DNS compatible host names (ie., virtual host) [localhost,127.0.0.1]: nnpxy02-vm0.bdfrem.wandisco.com
Enter the fusion server host and port [host:port]: nnpxy02-vm0.bdfrem.wandisco.com:8023
Is Fusion SSL enabled (yes/no)?
  [If yes, you need to provide the keystore path and encrypted password]: yes
Enter the fusion truststore file path: /opt/fusionssl/wandisco.ks
Enter the fusion truststore key type: JKS
Please enter the password to be encrypted
> ********
log4j:WARN No appenders could be found for logger (com.wandisco.security.crypto.CryptoFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Please specify the appropriate storage type for the proxy server to accept api request:
1) S3
2) AZURE
#? 1

Storage type selected: S3
 ------------------------------------------------------------------------------------------------

 **** MultiCloud Proxy server details ****

Proxy server listen host: 0.0.0.0
Proxy server listen port: 8071
Proxy server SSL: true
Keystore path: /etc/wandisco/multicloud-proxy/ssl/wandisco.ks
Keystore password: *****************************
MultiCloud proxy server's DNS compatible host names: localhost,127.0.0.1,nnpxy02-vm0.bdfrem.wandisco.com

 **** Fusion server details ****

Fusion server host and port: nnpxy02-vm0.bdfrem.wandisco.com:8023

Fusion server SSL: true
Keystore path: /opt/fusionssl/wandisco.ks
Keystore type: JKS
Keystore password: *****************************


 **** Virtual bucket ****


    Storage Type: S3


 ------------------------------------------------------------------------------------------------

Which user should MultiCloud Proxy run as? [root]:
Which group should MultiCloud Proxy run as? [root]:
Enter the minimum memory(-Xms) for MultiCloud Proxy (in MB) [512]:
Enter the maximum memory(-Xmx) for MultiCloud Proxy (in MB) [1024]:

 -------------------------------------------------

 **** MultiCloud Proxy environment details ****

 Run as User: root
 Run as Group: root

 Minimum memory: 512m
 Maximum memory: 1024m

 -------------------------------------------------

Do you confirm the multiCloud proxy server configuration details (yes/no): yes

MultiCloud proxy server configuration done successfully, start 'multicloud-proxy-server' to load the configuration

--------------------------------------------------------------------------------------------------------
 Note: You can edit the configuration values anytime in: /etc/wandisco/multicloud-proxy/core-site.xml, /etc/wandisco/multicloud-proxy/proxy-server-site.xml
       The multicloud-proxy-server must be restarted for the changes to take effect
  -------------------------------------------------------------------------------------------------------
[root@nnpxy02-vm0 ~]#
Enabling SSL

To enable SSL for the proxy server follow the steps below:

  1. Generate SSL certificates.

  2. Reconfigure the proxy server using /etc/wandisco/multicloud-proxy/configure-proxy-server and provide the new keystore value.

  3. Start the proxy server.

Approach to Connect Proxy Server with SSL in Azure cli

  1. Copy the ca.crt from proxy server /etc/wandisco/multicloud-proxy/ssl/ca.crt to the machine where azure cli is installed

  2. Export ca certificate in terminal, so that the azure cli requests will use our self-signed ca-certificate.

    export REQUESTS_CA_BUNDLE=ca.crt

  3. Run the azure cli command with BlobEndpoint (which will point proxy-server)

    az storage blob list --container-name <conatiner-name> --connection-string 'AccountName=chsiva;AccountKey=****==;BlobEndpoint=https://<Proxy-Server-Host-Name>:8071'
Approach to Connect Proxy Server with SSL in aws S3

  1. Connect MultiCloud Proxy using --no-verify-ssl

  2. The 'multicloud-proxy-server' can be connected via ssl without --no-verify-ssl by any of the following approaches.

    Approach 1:
    Passing the trusted CA root certificate using the --ca-bundle command line argument

    # aws s3 ls s3:// --endpoint-url https://multicloud-proxy-demo.wandisco.com:8071 --ca-bundle /etc/wandisco/multicloud-proxy/ssl/ca.crt

2018-05-21 13:38:04 vbucket

    Approach 2:
    Set the environment variable AWS_CA_BUNDLE with the absolute path to the trusted CA root certificate.

    # export AWS_CA_BUNDLE=/etc/wandisco/multicloud-proxy/ssl/ca.crt
# echo $AWS_CA_BUNDLE
/etc/wandisco/multicloud-proxy/ssl/ca.crt

# aws s3 ls s3:// --endpoint-url https://multicloud-proxy-demo.wandisco.com:8071
2018-05-21 13:41:04 vbucket

    Approach 3:
    In the .aws/config file, set the variable ca_bundle with the absolute path of the trusted CA root certificate.

    ca_bundle = /etc/wandisco/multicloud-proxy/ssl/ca.crt

# cat .aws/config
[default]
output = json
region = us-east-1
ca_bundle = /etc/wandisco/multicloud-proxy/ssl/ca.crt
s3 =
signature_version = s3
addressing_style = path

# aws s3 ls s3:// --endpoint-url https://multicloud-proxydemo.wandisco.com:8071
2018-05-21 13:41:04 vbucket
Changing the time zone

Logs use UTC timezone by default but this can be manually altered through log4j configuration if required. To alter the timezone the xxx.layout.ConversionPattern property needs to be overwritten.

log4j.appender.xxxxxlog.layout.ConversionPattern=%d{ISO8601}{UTC} %p %c - %t:[%m]%n

{UTC} can be replaced with, for example {GMT} or {ITC+1:30}. If offsetting from a timezone, + or - can be used, hours must be between 0 and 23, and minutes must be between 00 and 59. This property is located in /etc/wandisco/multicloud-proxy/log4j.properties

After updating the property, multicloud-proxy-server needs to be restarted for the changes to take effect.

S3 Payloads

Currently S3 Payloads can be signed or unsigned in a single chunk. Multi-chunk payloads are not yet supported and your deployment may need to be configured depending on the combination of your S3plugin and Multisite proxy.

<property>
<name>fs.fusion.s3.chunkedEncoding</name>
<value>false</value>
</property>

4.3. Upgrade

The following steps provide a basic sequence for upgrading an existing MultiCloud installation.

  1. Bring the Fusion server, UI server and IHC server to a stop.

    service fusion-server stop
service fusion-ui-server stop
service fusion-ihc-server-<version> stop

  2. Stop the MultiCloud proxy server.

    service multicloud-proxy-server stop

  3. Upgrade using the latest plugin RPM/DEM files.

    rpm -U <multicloud-plugin-rpm-package>

    or

    dpkg -i <multicloud-plugin-deb-package>

  4. Upgrade the proxy using the proxy RPM/DEM.

    rpm -U <multicloud-proxy-rpm-package>

    or

    dpkg -i <multicloud-proxy-deb-package>

  5. Start the Fusion, UI and IHC servers.

    service fusion-server start
service fusion-server-ui start
service fusion-ihc-server-<version> start

  6. Start the multicloud-proxy server.

    service multicloud-proxy-server start

4.3.1. Package removal

Important!
Do not remove plugin RPMs/Jars as part of an uninstallation procedure. There may be cases where Fusion’s database continues to reference the core plugin RPMs after their removal, causing problems with Fusion restarts.
Before removing core plugin RPMs/Jars you must first perform a flush of Fusion’s internal database, which would require a re-induction and configuration of your Fusion deployment. Contact Support for more information about this procedure.

Currently there is no programatic method for removing components although you can use the following commands to delete the plugin components, one at a time:

# yum erase -y -q {plugin-core-package}


WANdisco Fusion MultiCloud Plugin uninstalled successfully.

rm -rf /opt/wandisco/fusion-ui-server/plugins/multicloud-2.12/
rm -rf /opt/wandisco/fusion-ui-server/ui-client-platform/plugins/multicloud-plugin/
rm -rf /opt/wandisco/fusion-ui-server/ui-client-platform/downloads/core_plugins/multicloud/

5. Operation

5.1. Configuration

Once configured, restart MultiCloud server to use the configuration applied:

# service fusion-server restart Enter

Then start each instance of the S3 proxy:

# service multicloud-proxy-server start Enter

For MultiCloud you can validate operation against the virtual buckets defined for the environment using standard MultiCloud client applications, such as the AWS CLI tools, or tools like s3cmd and for Azure blob you can validate operation against the containers defined for the environment using standard azure client applications, such as the azure CLI tools, or tools like az cmd.

5.1.1. Configuring Applications to use LiveData for MultiCloud

Client applications can be configured to use the virtual buckets provided by the Fusion Plugin for MultiCloud in multiple ways:

As an HTTP(S) proxy

Applications that would normally communicate directly with the underlying S3/Azure Blob bucket can be directed to use the proxy through standard HTTP(S) proxy configuration. The benefit of this approach is that it requires no change to application code to direct them to the Fusion Plugin for MultiCloud.

Ensure that the plugin has been configured to accept requests for the original hostname used by client applications. Specify the DNS compatible host names to match when configuring the proxy. Applications can continue to use the original bucket name if it matches the underlying bucket referred to by the proxy.

As a new MultiCloud endpoint

Applications can direct S3 requests directly to the proxy, which provides virtual buckets. Configure your application to refer to the proxy as the S3 endpoint, and use the virtual bucket name.

5.2. Create a rule

Before you can replicate data, you need to set up a replication rule for the resource location in the underlying file system.

  1. Login to the Fusion UI and click on the Replication Rules tab. Click on the Create + button.

    MultiCloud Architecture
    Figure 5. MultiCloud Create a rule

  2. The Rule creation screen will appear. Enter the following details.

    MultiCloud Architecture
    Figure 6. MultiCloud Create a rule
    Virtual Storage name

    Choose a name, following the guidelines stated below, that will be the single identifier for the proxy’s virtual bucket. This name will be used by client applications when interacting with replicated objects, and will be available at the endpoints offered by each of the proxy instances.

    Virtual Storage naming guidelines

    • v-bucket name must be alphanumeric and lower-case.

      • If you use upper-case characters you can only replicate from Azure to AWS, not use the v-bucket name in the AWS or Azure CLI.

      • If you don’t follow these rules, you may not see errors until you perform an AWS to Azure replication or use the v-bucket name for the repositories.

Replication constraints
Include in replication

Select which resources you wish to replicate under the rule. Note that by default the pattern ".*" is used which will replicate everything in the bucket.

Exclude from replication

Select specific artifacts that you wish to block from replication.

Zones

You need to select a minimum of 2 zones, the local zone is automatically selected and is manditory.

  1. Continue adding your settings.

    MultiCloud Architecture
    Figure 7. MultiCloud Create a rule
Priority Zone

The Priority Zone is the zone which is most important, that which is most reliable or up to date. For example if your set up has a production zone and a disaster recovery zone then production would be more important and therefore your priority zone.

Connect to Underlying Storage
Storage name

The name of the underlying S3 bucket for this zone.

Access key

Credentials for the underlying bucket.

Secret access key

Further credentials for the underlying bucket.

Region

The region by which the bucket is located.

Endpoint URL

The endpoint by which the bucket can be accessed. This can be in any of the forms: <hostname>, <hostname>:<port>, http(s)://<hostname>, or http(s)://<hostname>:<port>.

Networking

Select whether or not you want to allow both inbound and outbound connections or limit to inbound connections only.

Create Rule

Click to create the rule.

5.2.1. Edit a rule

Once created, you can edit Replication Rules by clicking on the rule resource label.

MultiCloud Architecture
Include in replication

Click the edit icon to change.

Exclude from replication

Click the edit icon to change.

MultiCloud Architecture

Click the edit icon to make changes to Storage.

MultiCloud Architecture

Click update to save any changes that you make.

5.3. Administration

Once configured, applications interact with the virtual buckets that have been configured. Standard MultiCloud API operations are replicated with strong consistency among the underlying buckets, and content associated with objects created is replicated between these buckets.

Use the Fusion Plugin for MultiCloud to provide a Live Data environment, where applications can interact with any of the replicated buckets, and each bucket will provide access to the same content regardless of where change is initiated.

5.3.1. Regular Application Operation

Operate your applications without change, whether they are custom applications, standard command line or other tools for working with MultiCloud.

5.3.2. Consistency Check

Because applications that do not interact with the MultiCloud environment via the Fusion Plugin for MultiCloud can modify bucket content without coordination or replication, the product provides a consistency check feature. Use consistency check to report on any differences among the replicated buckets.

Consistency check is provided through a REST API call. A consistency check is a potentially long-running task, initiated with a specific REST operation. Provide the path associated with the virtual bucket when initiating the check to specify which virtual bucket to review.

Perform a consistency check
S3 data and eventual consistency

MultiCloud’s mechanism for dealing with data inconsistency is not instantaneous and operates under constraints imposed by the use of the S3 API. For this reason, all operations to make data consistent between zones run under the eventual consistency model. You may need to wait some time to see the results of a replica convergence.

Use this procedure to complete a check of consistency between one or more zones.

  1. Navigate to Replication tab.

  2. Select a rule on which to perform a consistency check.

  3. Click on the status panel and click on the check now button, then click on one of the available zones to be the source of truth.

    MultiCloud Architecture
    Figure 8. MultiCloud Consistency Check

    By clicking on a zone you see a graph of the differences between zones. If you select a zone that has data that is not available on the other zone(s), then the bar graphs will show green. Conversely, if the selected zone doesn’t contain some data that is on the other zone(s) then this will show as red on the bar.

    Objects

    Total number of objects.

    Size

    Total file size of replicated data under the rule.

    Directories

    Number of directories replicated under the rule.

    Download detailed report

    Click this link to download a JSON file containing all available data from the consistency check.

sample download report
{"inconsistencies":[{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":21,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":true}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":true}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":true}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":58,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":21,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":21,"directory":false}},"path":"test"},{"differences":{"epam01-zone1":{"zoneName":"epam01-zone1","size":0,"directory":true}},"path":"test"}]}
S3 ←→ S3

S.No

Properties

Support

1

bucket accelerate

No

2

bucket acl

No

3

bucket cors

No

4

bucket encryption

No

5

bucket lifecycle

No

6

bucket metrics

No

7

bucket requestPayment

No

8

bucket tagging

No

9

bucket versioning

No

10

bucket website

No

11

object content

No

12

object acl

Yes

13

object tagging

No
Azure Blob <→ Azure Blob

S.No

Properties

Support

1

container metadata

No

2

container permission

No

3

container policy

No

4

blob metadata

Yes

5

blob properties

Yes

6

blob type

Yes

7

blob content

Yes

8

blob content MD5

Yes
S3 <→ Azure

Bucket v Container diff:

Properties

Bucket

Container

Supported

Metadata

Tagging

Metadata

No

Object vs Block

Properties

Object

Block Blob

Supported

Metadata

Tagging

Metadata

No

Size

Content-Length

Content-Length

Yes
Wildcard Consistency checks

For the case of a wildcard, e.g. "*" only rules, the normal status tab is not shown, instead explains that consistency check and repair will need to be manually triggered using the script, noted below.

Using the Wildcard Rule scripts

In target folder there will be repairmanager.tar.gz distribution package, extract for the utility "repairmanager-1.0.jar".

How to prepare to run:
  We need to complete repair.properties file

    IS_INITIALIZED=false                 - IMPORTANT!!! The first run should be with value false

    FUSION_ENDPOINT=http://hostname:8082 - source fusion endpoint
    SRC_ZONE_NAME=srcZone                - source Zone name
    MCX_PLUGIN_CONFIG=config.xml         - local copy of proxy-plugin-site.xml

    MAIN_LOOP_DELAY=10                   - in seconds, how often to check tasks are done

    REPAIR_STATE_PATH                    - absolute or relative to user.dir path where repair state files to be stored
                                           default value is user.dir

    RUN_CC = true/false                  - whether run CC task after repair task
                                           default value is true

  Repair options : default values are true
    REPAIR_OPTION_ADD_MISSING
    REPAIR_OPTION_FIX_DIFFERENT
    REPAIR_OPTION_REMOVE_EXTRA



How to run
  java -jar repairmanager-1.0.jar
Notes on utility script usage

  1. If repair manager will be stopped by Ctl-C it can be run again and will start from the point it was stopped

  2. Repair manager creates files with replication rules names, where stored current state of repair process for each replication rule.

  3. When repair manager finishes, these files will contain records like

    • containerName :

    • Status :

    • repairTimeCreated :

    • repairTimeFinished : :

    • isCCrun :

    • isCCconsistent

    • nzan1:finished:1543585757614:1543585763437: :CC:true

Consistency Check API
# curl -X POST "http://<fusion-server>:<fusion-server-api-port>/plugin/multicloud/cc?vbucket=<virtual-bucket-name>&prefix=<key-prefix>"

  1. To check a particular directory under the virtual bucket associated with a replication path, add the ccpath to the end of the operation:

    # curl -i -X POST "http://localhost:8082/plugin/multicloud/cc?path=/repl-path&vbucket=vbucket&ccpath=dir1/" Enter
HTTP/1.1 202 Accepted
Content-Location: http://localhost:8082/fusion/task/84b417f3-ec60-11e7-aa4b-0242ac120002
Content-Length: 0
Server: Jetty(6.1.26)

Access the status of the consistency check with the cc task Id generated during consistency check operation at the location referenced:

# curl -i -X GET "http://localhost:8082/fusion/task/84b417f3-ec60-11e7-aa4b-0242ac120002" Enter
HTTP/1.1 200 OK
Content-Length: 938
Content-Type: application/xml
Server: Jetty(6.1.26)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><task><taskId>84b417f3-ec60-11e7-aa4b-0242ac120002</taskId><timeCreated>1514528494614</timeCreated><creatorNodeId>400b7699-56d2-44c5-b07c-f42a760b966f</creatorNodeId><timeUpdated>1514528530417</timeUpdated><isDone>true</isDone><aborted>false</aborted><properties><entry><key>CC_REPORT_PATH</key><value>/vbucket/.fusion/4fc19ba7-e744-11e7-85c7-0242ac120003/metadata/84b417f3-ec60-11e7-aa4b-0242ac120002/cc-report</value></entry><entry><key>TOTAL_INCONSISTENCIES_FOUND</key><value>0</value></entry><entry><key>TASK_TYPE</key><value>S3PROXY_CONSISTENCY_CHECK</value></entry><entry><key>LOCAL_COMPLETE</key><value>1514528530417</value></entry><entry><key>LOCAL_START</key><value>1514528494614</value></entry><entry><key>CONSISTENCY_CHECK_STATUS</key><value>CONSISTENT</value></entry></properties><previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/></task>

To access the consistency check report use the cc task Id generated during consistency check operation at the location referenced:

# curl -i -X GET "http://localhost:8082/plugin/multicloud/cc/report/84b417f3-ec60-11e7-aa4b-0242ac120002?path=/repl-path&withConsistencyReport=true" Enter
HTTP/1.1 200 OK
Content-Length: 301
Content-Type: application/xml
Server: Jetty(6.1.26)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><consistencyReport><path>/vb1</path><state>CONSISTENT</state><taskId>84b417f3-ec60-11e7-aa4b-0242ac120002</taskId><checksumMethod>MD5</checksumMethod><noInconsistencies>0</noInconsistencies><lastCheck>1514548334438</lastCheck></consistencyReport>

5.3.3. Make Consistent

Make Consistent through the UI

Resolve inconsistencies among buckets with the Make Consistent feature. Initiate a Make Consistent operation through the UI or via the REST API.

  1. After completing a Consistency Check, and finding inconsistency, click on the zone that represent the "source of truth", that is the zone with the most correct and up-to-date data.

    MultiCloud Architecture
    Figure 9. MultiCloud Make Consistent 01 - azure_zone1 clicked, in this example

  2. The graphs of the zones will update to highlight the differences between the zones, making it easier to determine the appropriate action.

    MultiCloud Architecture
    Figure 10. MultiCloud Make Consistent 02

  3. Select from the available options:

    Add missing objects

    Tick to ensure that files/directories that only exist in a subset of the zones are copied across to all. Default: ticked

    Remove extraneous objects

    Tick to ensure that files/directories that don’t appear in the "Zone of truth" are removed, to ensure that all zones contain the same files and directories. Default: unticked

    Update objects which differ

    When files/directories with the same name are present, ticking this option will ensure that they are all the latest version, taken from the copies in the zone of truth. Default: unticked

  4. Click Make consistent. This will begin the operation and a confirmation message, "Request for making zone consistent submitted".

    MultiCloud Architecture
    Figure 11. MultiCloud Make Consistent 03

  5. Run another Consistency Check to see if the zones have been made consistent.

Make Consistent through the REST API

In MultiCloud 2.0 it is possible to run the Make Consistent tool through the Fusion UI, although it remains possible to also run it using the REST API, as noted in the reference section.

5.3.4. Dynamic configuration of Virtual buckets

Virtual buckets (vbucket) can be added, listed, modified or deleted from the proxy-plugin-site.xml using the REST APIs below.

Add virtual buckets

This REST API adds virtual buckets to the proxy-plugin-site.xml. The details of the virtual bucket to be added have to be manually added to proxy-server-site.xml.

You must restart the S3Proxy server for changes to take effect.

An example xml file with the details of virtual bucket to be added in the proxy-plugin-site.xml:

<s3proxy>

   <vbucket name="vbucket1">

      <bucket name="bucket-a">
        <zonename>zone1</zonename>
        <accesskey></accesskey>
        <secretaccesskey>*</secretaccesskey>
        <region>us-east-1</region>
        <endpoint-url>s3.amazonaws.com</endpoint-url>
        <provider>AWS_S3</provider>
      </bucket>

      <bucket name="bucket-b">
        <zonename>zone2</zonename>
        <accesskey></accesskey>
        <secretaccesskey>*</secretaccesskey>
        <region>eu-west-1</region>
        <endpoint-url>s3-eu-west-1.amazonaws.com</endpoint-url>
        <provider>AWS_S3</provider>
      </bucket>

   </vbucket>

</s3proxy>
# curl -v -i -X PUT -H "Content-Type: application/xml" -d @config.xml "http://localhost:8082/plugin/s3proxy/config/vbucket?path=/repl1" Enter
> Content-Type: application/xml
> Content-Length: 776
>
* upload completely sent off: 776 out of 776 bytes
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)
List virtual buckets

This REST API gives a listing of virtual buckets configured in the proxy-plugin-site.xml.

# curl -v -X GET "http://localhost:8082/plugin/s3proxy/config/vbucket?vbucket=vbucket1" Enter
>
< HTTP/1.1 200 OK
< Content-Length: 862
< Content-Type: application/xml
< Server: Jetty(6.1.26)
<

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Vbucket>
<bucket>
<virtualBucket>vbucket1</virtualBucket>
<bucketName>bucket1</bucketName>
<zoneName>zone1</zoneName>
<accessKey></accessKey>
<secretAccessKey></secretAccessKey>
<provider>AWS_S3</provider>
<endPointRegion>us-east-1</endPointRegion>
<endPointUrl>s3.amazonaws.com</endPointUrl>
</bucket>
<bucket>
<virtualBucket>vbucket1</virtualBucket>
<bucketName>bucket2</bucketName>
<zoneName>zone2</zoneName>
<accessKey></accessKey>
<secretAccessKey></secretAccessKey>
<provider>AWS_S3</provider>
<endPointRegion>eu-west-1</endPointRegion>
<endPointUrl>s3-eu-west-1.amazonaws.com</endPointUrl>
</bucket>
</Vbucket>
Modify virtual buckets

This REST API modifies the virtual buckets configured in the proxy-plugin-site.xml. The virtual bucket details for modification has to be given as an xml file, from which the details will be taken and specified virtual bucket will be modified in the proxy-plugin-site.xml.

You must restart the S3Proxy server for changes to take effect.

An example xml file with details of virtual bucket to be modified in the proxy-plugin-site.xml:

<s3proxy>

   <vbucket name="vbucket">

      <bucket name="bucketNew1">
        <zonename>zone1</zonename>
        <accesskey></accesskey>
        <secretaccesskey>*</secretaccesskey>
        <region>us-east-1</region>
        <endpoint-url>s3.amazonaws.com</endpoint-url>
        <provider>AWS_S3</provider>
      </bucket>

      <bucket name="bucketNew2">
        <zonename>zone2</zonename>
        <accesskey></accesskey>
        <secretaccesskey>*</secretaccesskey>
        <region>eu-west-1</region>
        <endpoint-url>s3-eu-west-1.amazonaws.com</endpoint-url>
        <provider>AWS_S3</provider>
      </bucket>

   </vbucket>

</s3proxy>
# curl -v -X POST -H "Content-Type: application/xml" -d @config.xml "http://localhost:8082/plugin/s3proxy/config/vbucket?path=/repl1" Enter
> POST /plugin/s3proxy/config/vbucket/update?path=/repl1 HTTP/1.1
> Host: localhost:8082
> User-Agent: curl/7.47.0
> Accept: /
> Content-Type: application/xml
> Content-Length: 776
>
* upload completely sent off: 776 out of 776 bytes
< HTTP/1.1 202 Accepted
< Content-Length: 0
< Server: Jetty(6.1.26)
Delete virtual buckets

This REST API deletes the virtual buckets configured in the proxy-plugin-site.xml.

You must restart the S3Proxy server for changes to take effect.

You must first manually remove the details of the vbucket to be deleted from the proxy-server-site.xml.

# curl -v -i -X DELETE "http://localhost:8082/plugin/s3proxy/config/vbucket?path=/repl1&vbucket=vbucket1" Enter
* Connected to localhost (10.6.121.44) port 8082 (#0)
> DELETE /plugin/s3proxy/config/vbucket/delete?path=/repl1&vbucket=vbucket1 HTTP/1.1
> Host: localhost:8082
> User-Agent: curl/7.47.0
> Accept: /
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)

5.3.5. Support for non-replicated buckets

Non-replicated buckets can be accessed via Live S3 using the default endpoint url of underlying storage. The default endpoint url must be added during the proxy configuration. The data uploaded in the non replicated bucket will be redirected to the default endpoint url and will not be coordinated.

We can access the non-replicated buckets via s3proxy with below signature and addressing styles

  1. v2 Path style

  2. v4 Path style

Limitations

  • The non-replicated bucket has to be created under the default-endpoint-url storage.

  • For now only one default-endpoint-url can be configured, even when there are multiple virtual buckets

For Example:

<?xml version="1.0" encoding="UTF-8"?>

<s3proxy>

  <server-config>
    <protocol>http</protocol>
    <listen-host>127.0.0.1</listen-host>
    <listen-port>8080</listen-port>
    <keystore-path />
    <keystore-pass />
    <truststore-path />
    <truststore-pass />
    <virtual-host>localhost,127.0.0.1,s3proxydemo.wandisco.com</virtual-host>
  </server-config>

  <virtualbuckets>

    <virtualbucket name="vbucket1" repl-path="/repl1" />   --> say AWS bucket
    <virtualbucket name="vbucket2" repl-path="/repl2" />   ---> say IBM bucket
    <virtualbucket name="vbucket3" repl-path="/repl3" />   ---> say Scality bucket

    <default-endpoint-url>s3.amazonaws.com</default-endpoint-url>   --> but the default endpoint url can only point any one of the storage.

  </virtualbuckets>

</s3proxy>

5.3.6. Selective Replication

Selective Replication allows you to limit which data are replicated. This is achieved by using inclusion and exclusion rules for each virtual bucket configured. Based on the rule added in the inclusion/exclusion xml, specific live data can be included or excluded from replication. When both inclusion and exclusion rules are equal, then the priority will be given to inclusion rule.

You must restart the S3Proxy server for changes to selective replication to take effect.
Exclusion

The exclusion rule can be added, listed or deleted using REST APIs. The data will not be replicated to the other zones is it matches with the exclusion rule.

Template on exclusion xml:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<exclusions>
  <vbucket name="vbucket">
    <exclusion>/*/.txt</exclusion>
  </vbucket>
  <vbucket name="vbucket-1">
    </exclusion>
  </vbucket>
</exclusions>
Add Exclusion

The exclusion rule requested will be added to the exclusion-rules.xml for the specified virtual bucket.

For Example:

# curl -v -i -X POST 'http://localhost:8082/plugin/s3proxy/config/rules/exclusion?path=/repl_dir&vbucket=vbucket&exclusion=*.txt'
>
< HTTP/1.1 202 Accepted
HTTP/1.1 200 OK
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)
List Exclusions

The exclusion rules for the specified virtual bucket will be listed.

# curl -v -i -X GET 'http://localhost:8082/plugin/s3proxy/config/rules/exclusion?vbucket=vbucket'
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 249
Content-Length: 249
< Content-Type: application/xml
Content-Type: application/xml
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><exclusions><vbucket name="vbucket"><exclusion>//.fusion/</exclusion><exclusion>//.fusion/*</exclusion><exclusion>.txt</exclusion></vbucket></exclusions>
Delete Exclusion

The exclusion rules for the specified virtual bucket will be deleted from the exclusion-rules.xml.

# curl -v -i -X DELETE 'http://localhost:8082/plugin/s3proxy/config/rules/exclusion?path=/repl_dir&vbucket=vbucket&exclusion=/*/.pdf'
>
< HTTP/1.1 202 Accepted
HTTP/1.1 200 OK
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)
Inclusion

The inclusion rule can be added, listed or deleted using REST APIs. The data will be replicated to the other zone when it matches with the inclusion rule based on the exclusion rule. The default inclusion rule will be ".*"

When the exclusion and inclusion rules are same, then the priority will be given for inclusion rule

Template on inclusion xml:

inclusion-tmpl.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<inclusions>
  <vbucket name="vbucket">
    <inclusion>dir*</inclusion>
  </vbucket>
  <vbucket name="vbucket-1">
    <inclusion>.*</inclusion>
  </vbucket>
</inclusions>
Add Inclusions

The inclusion rule requested will be added to the inclusion-rules.xml for the specified virtual bucket.

# curl -v -i -X POST 'http://localhost:8082/plugin/s3proxy/config/rules/inclusion?path=/repl_dir&vbucket=vbucket&inclusion=dir/*'
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)

<
* Connection #0 to host localhost left intact
* Closing connection #0
List Inclusions

The inclusion rules for the specified virtual bucket will be listed.

# curl -v -i -X GET 'http://localhost:8082/plugin/s3proxy/config/rules/inclusion?vbucket=vbucket'
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 139
Content-Length: 139
< Content-Type: application/xml
Content-Type: application/xml
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><inclusions><vbucket name="vbucket"><inclusion>.</inclusion><inclusion>dir/</inclusion></vbucket></inclusions>
Delete Inclusions

The inclusion rules for the specified virtual bucket will be deleted from the inclusion-rules.xml.

# curl -v -i -X DELETE 'http://localhost:8082/plugin/s3proxy/config/rules/inclusion?path=/repl_dir&vbucket=vbucket&inclusion=dir/*'
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)

5.4. Troubleshooting

Observe information in the log files generated for the WANdisco Fusion server and LiveData for MultiCloud to troubleshoot issues at runtime. Exceptions or log entries with a SEVERE label may represent information that can assist in determining the cause of any problem.

As a distributed system, LiveData for MultiCloud will be impacted by the operation of the underlying S3 endpoints with which it communicates. You may also find it useful to review log or other information from these endpoints.

6. Reference

6.1. MultiCloud Configuration Properties

MultiCloud configuration is managed through the following files.

6.1.1. WANdisco MultiCloud Proxy properties

/etc/wandisco/multicloud-proxy/proxy-server-site.xml
Table 4. proxy-server-site.xml
Property Description Default Value

<multicloud>
<server-config>
<max-proxy-threads>200</max-proxy-threads>
</server-config>
</multicloud>

Proxy Threads which process incoming request

200

<multicloud>
<server-config>
<max-client-threads>256</max-client-threads>
</server-config>
</multicloud>

Client Threads which process the downstream requests

256

<multicloud> <server-config>
<max-client-request-queue>1024</max-client-request-queue>
</server-config>
</multicloud>

Request Queue where Requests wait to be sent to underlying storage (S3 / Azure Blob Storage)

1024

<multicloud>
<server-config>
<timeout>0</timeout>
</server-config>
</multicloud>

The total timeout for the request/response conversation

0

<multicloud>
<server-config>
<idle-timeout>60000</idle-timeout>
</server-config>
</multicloud>

The time in milliseconds that the connection can be idle before it is closed

60000

<multicloud>
<server-config>
<max-connections>2*</max-connections>
</server-config>
</multicloud>

The maximum number of connections

2*max-client-threads

<multicloud>
<server-config>
<incoming-connection-limit>0</incoming-connection-limit>
</server-config>
</multicloud>

The limit of incoming connections. This could be set to actively refuse extra connections

0

<multicloud>
<server-config>
<max-coordination-threads>256</max-coordination-threads>
</server-config>
</multicloud>

The amount of requests that wait for coordination at the same time

256

6.1.2. WANdisco MultiCloud Plugin properties

/etc/wandisco/fusion/plugins/multicloud/proxy-plugin-site.xml

6.1.3. Container set of features supported by S3 Providers

/etc/wandisco/fusion/plugins/multicloud/s3-bucket-features.xml

6.2. API Supports

6.2.1. S3 Applications

Support for the S3 API.

S.I. No.

Operations

Status

WDOS equivalent

Comments

1

GET Service

SUPPORTED

aws s3api list-buckets --endpoint-url http://<s3proxy-server-host>:<s3proxy-server-port>/

2

DELETE Multiple Objects

PARTIALLY SUPPORTED

WD_DELETE_MULTIPLE_OBJECTS Partially Supported(with version id will not be coordinated WD-SPX-11)

3

DELETE Object

PARTIALLY SUPPORTED

WD_DELETE_OBJECT +Partially Supported (with version id will not be coordinated WD-SPX-9)

4

DELETE Object tagging

PARTIALLY SUPPORTED

WD_DELETE_OBJECT_TAGGING

Partially Supported (with version id will not be coordinated WD-SPX-10)

5

GET Object

SUPPORTED

6

GET Object ACL

SUPPORTED

7

GET Object tagging

SUPPORTED

8

GET Object torrent

SUPPORTED

9

HEAD Object

SUPPORTED

10

OPTIONS object

SUPPORTED

11

POST Object

PARTIALLY SUPPORTED

WD_POST_OBJECT Partially Supported (supports only actual bucket name and not virtual bucket name) (WD-SPX-347)

12

POST Object restore

NOT SUPPORTED

Not supported (WD-SPX-48)

13

PUT Object

SUPPORTED

WD_PUT_OBJECT

14

PUT Object - Copy

SUPPORTED

15

PUT Object acl

SUPPORTED

16

PUT Object tagging

SUPPORTED

WD_PUT_OBJECT_TAGGING

17

SELECT Object Content (Preview)

SUPPORTED

18

Abort Multipart Upload

SUPPORTED

19

Complete Multipart Upload

SUPPORTED

WD_COMPLETE_MULTIPART

20

Initiate Multipart Upload

SUPPORTED

21

List Parts

SUPPORTED

22

Uploadar Pt

SUPPORTED

22

Upload Part - Copy

SUPPORTED

23

DELETE Bucket

SUPPORTED

WD_DELETE_BUCKET

24

DELETE Bucket analytics

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination (WD-SPX-224)

25

DELETE Bucket cors

SUPPORTED

WD_DELETE_BUCKET_CORS

26

DELETE Bucket encryption

SUPPORTED

WD_DELETE_BUCKET_ENCRYPTION

26

DELETE Bucket inventory

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination (WD-SPX-225)

The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the same region as the source bucket.

27

DELETE Bucket lifecycle

SUPPORTED

WD_DELETE_BUCKET_LIFECYCLE

28

DELETE Bucket metrics

SUPPORTED

WD_DELETE_BUCKET_METRICS

29

DELETE Bucket policy

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination (WD-SPX-229)

The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the same region as the source bucket.

30

DELETE Bucket replication

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination

The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the different region from the source bucket.

31

DELETE Bucket tagging

SUPPORTED

WD_DELETE_BUCKET_TAGGING

31

DELETE Bucket website

SUPPORTED

WD_DELETE_BUCKET_WEBSITE

32

GET Bucket (List Objects) Version 2

SUPPORTED

33

GET Bucket accelerate

SUPPORTED

34

GET Bucket acl

SUPPORTED

35

GET Bucket analytics

SUPPORTED

36

GET Buckets cors

SUPPORTED

37

GET Bucket encryption

SUPPORTED

38

GET Bucket inventory

SUPPORTED

39

GET Bucket lifecycle

SUPPORTED

40

GET Bucket location

SUPPORTED

41

GET Bucket logging

SUPPORTED

42

GET Bucket metrics

SUPPORTED

43

GET Bucket notification

SUPPORTED

44

GET Bucket Object versions

SUPPORTED

45

GET Bucket policy

SUPPORTED

46

GET Bucket replication

SUPPORTED

47

GET Bucket requestPayment

SUPPORTED

48

GET Bucket tagging

SUPPORTED

49

GET Bucket versioning

SUPPORTED

50

GET Bucket website

SUPPORTED

51

HEAD Bucket

SUPPORTED

52

List Bucket Analytics Configurations

SUPPORTED

53

List Bucket Inventory Configurations

SUPPORTED

54

List Bucket Metrics Configurations

SUPPORTED

55

List Multipart Uploads

SUPPORTED

56

PUT Bucket

SUPPORTED

WD_PUT_BUCKET

57

PUT Bucket accelerate

SUPPORTED

WD_PUT_BUCKET_ACCELERATE

58

PUT Bucket acl

SUPPORTED

WD_PUT_BUCKET_ACL

59

PUT Bucket analytics

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination

The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the same region as the source bucket.

60

PUT Bucket encryption

SUPPORTED

WD_PUT_BUCKET_CORS

61

PUT Bucket cors

SUPPORTED

WD_PUT_BUCKET_ENCRYPTION

62

PUT Bucket inventory

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination

The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the same region as the source bucket.

63

PUT Bucket lifecycle

SUPPORTED

WD_PUT_BUCKET_LIFECYCLE

64

PUT Bucket logging

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination

The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the same region as the source bucket.

65

PUT Bucket metrics UI

SUPPORTED

WD_PUT_BUCKET_METRICS

66

PUT Bucket notification

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination

PUT Bucket Notification has payload as xml configuration containing topic arn which has the region that has to be same as the bucket region.

67

PUT Bucket policy

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination

The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the same region as the source bucket.

68

PUT Bucket replication

PARTIALLY SUPPORTED / WON’T REPLICATE

Supported, but with no coordination

The specified API contains payload as xml configuration with the dependency on destination bucket which has to be in the different region from the source bucket.

69

PUT Bucket requestPayment

SUPPORTED

WD_PUT_BUCKET_REQUESTPAYMENT

70

PUT Bucket tagging

SUPPORTED

WD_PUT_BUCKET_TAGGING

71

PUT Bucket versioning

SUPPORTED

WD_PUT_BUCKET_VERSIONING

6.2.2. Azure APIs

S.I. No.

Operations

Actual Azure Storage -V1

Actual Azure Storage -V2

Actual Azure Storage - Blob Storage

WDOS equivalent

Comment

1

List Containers

SUPPORTED

SUPPORTED

SUPPORTED

2

Set Blob Service Properties

SUPPORTED

SUPPORTED

SUPPORTED

3

Get Blob Service Properties

SUPPORTED

SUPPORTED

SUPPORTED

4

Preflight Blob Request

SUPPORTED

SUPPORTED

SUPPORTED

5

Get Blob Service Stats

SUPPORTED

SUPPORTED

SUPPORTED

6

Create Container

SUPPORTED

SUPPORTED

SUPPORTED

WD_CREATE_BUCKET

7

Get Container Properties

SUPPORTED

SUPPORTED

SUPPORTED

8

Get Container Metadata

SUPPORTED

SUPPORTED

SUPPORTED

9

Set Container Metadata

SUPPORTED

SUPPORTED

SUPPORTED

WD_SET_BUCKET_METADATA

10

Get Container ACL

SUPPORTED

SUPPORTED

SUPPORTED

11

Set Container ACL

SUPPORTED

SUPPORTED

SUPPORTED

WD_SET_BUCKET_ACL

12

Delete Container

SUPPORTED

SUPPORTED

SUPPORTED

WD_DELETE_BUCKET

13

Lease Container

SUPPORTED

SUPPORTED

SUPPORTED

Supported, but with no coordination

No Coordination

14

List Blobs

SUPPORTED

SUPPORTED

SUPPORTED

15

Put Blob

SUPPORTED

SUPPORTED

SUPPORTED

WD_PUT_OBJECT

16

Get Blob

SUPPORTED

SUPPORTED

SUPPORTED

17

Get Blob Properties

SUPPORTED

SUPPORTED

SUPPORTED

18

Set Blob Properties

SUPPORTED

SUPPORTED

SUPPORTED

WD_SET_OBJECT_PROPERTIES

19

Get Blob Metadata

SUPPORTED

SUPPORTED

SUPPORTED

20

Set Blob Metadata

SUPPORTED

SUPPORTED

SUPPORTED

WD_SET_OBJECT_METADATA

21

Lease Blob

SUPPORTED

SUPPORTED

SUPPORTED

Supported, but with no coordination

No Coordination

22

Snapshot Blob

SUPPORTED

SUPPORTED

SUPPORTED

WD_SNAPSHOT_OBJECT

23

Copy blob

SUPPORTED

SUPPORTED

SUPPORTED

WD_COPY_OBJECT

24

Abort Copy Blob

SUPPORTED

SUPPORTED

SUPPORTED

Supported, but with no coordination

No Coordination

25

Delete Blob

SUPPORTED

SUPPORTED

SUPPORTED

WD_DELETE_OBJECT

26

Undelete Blob

SUPPORTED

SUPPORTED

SUPPORTED

WD_UNDELETE_OBJECT

27

Set Blob Tier

NOT SUPPORTED

SUPPORTED

SUPPORTED

WD_SET_OBJECT_TIER

28

Put Block

SUPPORTED

SUPPORTED

SUPPORTED

No Coordination

29

Put Block List

SUPPORTED

SUPPORTED

SUPPORTED

WD_PUT_BLOCK_LIST

30

Get Block List

SUPPORTED

SUPPORTED

SUPPORTED

31

Put Page

SUPPORTED

SUPPORTED

SUPPORTED

WD_PUT_PAGE

32

Get Page Ranges

SUPPORTED

SUPPORTED

SUPPORTED

33

Incremental Copy Blob

SUPPORTED

SUPPORTED

SUPPORTED

WD_INCREMENTAL_COPY_OBJECT

34

Append Block

SUPPORTED

SUPPORTED

SUPPORTED

WD_APPEND_BLOCK

6.2.3. S3 ←→ Azure

S.I. No.

S3

WDOS

Azure

1

Create Bucket

WD_CREATE_BUCKET

Create Container

2

Delete Bucket

WD_DELETE_BUCKET

Delete Container

3

PUT Bucket tagging

WD_PUT_BUCKET_TAGGING

Set Container Metadata

4

Put Object

WD_PUT_OBJECT

Put Blob

5

Delete Object

WD_DELETE_OBJECT

Delete Blob

6

Copy Object

WD_COPY_OBJECT

Copy Blob

7

Complete Multipart Upload

WD_COMPLETEMULTIPART_UPLOAD

Put Block List

8

PUT Object Tagging

WD_PUT_OBJECT_TAGGING

Set Blob Metadata

6.3. Consistency check API Calls

The following REST calls allow you to handle consistency checks programatically.

  • Specify the ccTaskId and the name of the zone that will be used as the source of truth for repair.

  • Repair ensures that each replicated bucket has the same content as the source of truth.

  • This may introduce object deletion and creation in all other replicated zones, so use this feature with care.

There are 3 types of operation available:

Recursive

True "Recursive true" will copy in the contents of any subfolders in the target zone.
False "Recursive false" will not repair the contents of any subfolders in the target zone.

Replace

True A repair with Replace true will overwrite all duplicate files and directories in target zone.
False A repair with Replace false will not overwrite the duplicate files and directories in target zone.

Preserve

True A repair with Preserve true will not remove any data that exists in target zone.
False A repair with Preserve false will remove all the data that exists in target zone.

To start a repair you need to add the cc taskId generated from the consistency check e.g.:

# curl -i -X PUT "http://localhost:8082/plugin/s3proxy/repair/84b417f3-ec60-11e7-aa4b-0242ac120002?srcZone=zone1&preserve=true&recursive=true&replace=false" Enter
HTTP/1.1 202 Accepted
Content-Location: http://localhost:8082/plugin/fusion/task/0088e14d-ec62-11e7-aa4b-0242ac120002
Content-Length: 0
Server: Jetty(6.1.26)

To access the status of the repair task at the location referenced e.g.:

# curl -i -X GET "http://localhost:8082/fusion/task/0088e14d-ec62-11e7-aa4b-0242ac120002" Enter
HTTP/1.1 200 OK
Content-Length: 679
Content-Type: application/xml
Server: Jetty(6.1.26)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><task><taskId>0088e14d-ec62-11e7-aa4b-0242ac120002</taskId><timeCreated>1514548109259</timeCreated><creatorNodeId>400b7699-56d2-44c5-b07c-f42a760b966f</creatorNodeId><timeUpdated>1514528530417</timeUpdated><isDone>true</isDone><aborted>false</aborted><properties><entry><key>TASK_TYPE</key><value>REPAIR_TASK</value></entry><entry><key>REPAIR_STATUS</key><value>COMPLETED</value></entry><entry><key>LOCAL_COMPLETE</key><value>1514548109903</value></entry><entry><key>LOCAL_START</key><value>1514548109259</value></entry></properties><previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/></task>

6.3.1. Start Consistency Check

curl -X POST "http://<fusion-server>:<fusion-server-api-port>/plugin/multicloud/cc?vbucket=<virtual-bucket-name>&prefix=<key-prefix>"

6.3.2. Consistency Check for pending List

curl -X GET "http://<fusion-server>:<fusion-server-port>/plugin/multicloud/cc/pending"

6.3.3. Consistency Check Task Status

curl -X GET "http://<fusion-server>:8082/fusion/task/<ccTaskId>"

6.3.4. Consistency Check Report

curl -X GET "http://<fusion-server>:<fusion-server-api-port>/plugin/multicloud/cc/report/<virtual-bucket-name>?withConsistencyReport=true"

6.4. Consistency check API for Wildcard

This section deals with the specific checks on wildcard buckets.

6.4.1. Consistency Check for Wildcard Bucket

curl -X POST "http://<fusion-server>:<fusion-server-port>/plugin/multicloud/cc?vbucket=<virtual-bucket-name>&bucket=<bucket-name>&prefix=<key-prefix>"

6.4.2. Consistency Check Report for Wildcard Bucket with Actual Bucket

curl -v -i -X GET "http://<fusion-server>:<fusion-server-port>/plugin/multicloud/cc/report/chbucket?bucket="<actual-bucket-name>"& withConsistencyReport=true"

6.4.3. Start a Consistency check for a specific virtual bucket associated with a prefix.

curl -v -i -X POST 'http://<fusion-server>:<fusion-server-api-port>/plugin/multicloud/cc?vbucket=<virtual-bucket-name>&prefix=<key-prefix>'
Example consistency checks
[root@host01-vm1 tmp]# curl -v -i -X POST "http://nnpxy01-vm5.bdfrem.wandisco.com:8082/plugin/multicloud/cc?vbucket=chbucket"
* About to connect() to host01-vm1.cluster.hostname.com port 8082 (#0)
*   Trying 10.x.y.z... connected
* Connected to host01-vm1.cluster.hostname.com (10.x.y.z) port 8082 (#0)
> POST /plugin/multicloud/cc?vbucket=chbucket HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: host01-vm1.cluster.hostname.com:8082
> Accept: */*
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Location: http://host01-vm1.cluster.hostname.com:8082/fusion/task/641086d9-c855-11e8-acca-e261d5d5398d
Content-Location: http://host01-vm1.cluster.hostname.com:8082/fusion/task/641086d9-c855-11e8-acca-e261d5d5398d
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)

<
* Connection #0 to host nnpxy01-vm5.bdfrem.wandisco.com left intact
* Closing connection #0
[root@host01-vm1 tmp]#

6.4.4. Consistency Check for pending List

curl -i -X GET "http://<fusion-server>:<fusion-server-port>/plugin/multicloud/cc/pending"

6.4.5. Repair for Wildcard Buckets

curl -v -i -X PUT "http://<fusion-server>:<fusion-server-api-port>/plugin/multicloud/repair/<virtual-bucket-name>?bucket=<actual-bucket-name>&srcZone=<source-zone-name>&preserve=true&recursive=true&replace=false"

6.4.6. Dynamic configuration of virtual buckets

Virtual buckets (vbucket) can be added, listed, modified or deleted from the proxy-plugin-site.xml using the REST APIs. The virtual bucket details for addition has to be given as an xml file, from which the details will be added to the proxy-plugin-site.xml.

Syntax of dynamic configuration of virtual buckets
List virtual buckets
curl -v -X GET 'http://<fusion-server>:8082/plugin/multicloud/config/vbucket?vbucket=chbucket'
Add virtual Buckets
curl -X PUT -H "Content-Type: application/xml" -d @config.xml "http://<fusion-server>:8082/plugin/multicloud/config/vbucket?membershipId=<membership-Id>"
Modify virtual buckets
curl -X POST -H "Content-Type: application/xml" -d @config.xml "http://<fusion-server>:8082/plugin/multicloud/config/vbucket"
Delete virtual buckets
curl -X DELETE "http://<fusion-server>:8082/plugin/multicloud/config/vbucket?vbucket=<virtual-bucket-name>&deleteRD=true"
Add virtual buckets

This REST API gives out the addition of virtual buckets to the proxy-plugin-site.xml. The virtual bucket details for addition has to be given as an xml file, from which the details will be added to the proxy-plugin-site.xml.

  1. Create replication directory in both DC’s 'fusion fs -mkdir /Newrepl'.

  2. Set a permission and ownership for the directory (/Newrepl) from both zones eg:

    fusion fs -chmod 777 /Newrepl
fusion fs -chown fusion:fusion /Newrepl
    Ensure that "fusion" ownership is replaced with the system user that you specified in your Fusion installation.

  3. Get the membership id

Get a membershipId
curl -X GET "https://<localhost>:8084/fusion/fs" --cacert /opt/fusionssl/ca.crt | xmllint --format -
Example
# curl -X GET "http://nnpxy01-vm5.bdfrem.wandisco.com:8082/fusion/fs"  | xmllint --format -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
101  1014  101  1014    0     0  95255      0 --:--:-- --:--:-- --:--:--  110k
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<replicatedDirectories>
  <total>1</total>
  <offset>0</offset>
  <size>1</size>
  <replicatedDirectory>
    <path>/repl</path>
    <membershipId>edd9786c-1a8d-4d7a-9466-4cd09f1644f1</membershipId>
    <familyRepresentativeId>
      <nodeId>d1cf01d6-72f8-40cf-846f-9e1a59d2d30f</nodeId>
      <dsmId>9d6615a9-c6c5-11e8-932e-da198c351b5a</dsmId>
    </familyRepresentativeId>
    <consistencyReport>
      <result>UNKNOWN</result>
      <lastCheck>-1</lastCheck>
      <nextCheck>-1</nextCheck>
      <lastCheckResult>UNKNOWN</lastCheckResult>
    </consistencyReport>
    <leader>
      <nodeId>cba7e4ff-5078-47a1-adfa-9a2b4721e7bc</nodeId>
      <dsmId>9d6615a9-c6c5-11e8-932e-da198c351b5a</dsmId>
    </leader>
    <isLeaderElected>true</isLeaderElected>
    <isLeader>false</isLeader>
    <writer>
      <nodeId>d1cf01d6-72f8-40cf-846f-9e1a59d2d30f</nodeId>
      <dsmId>9d6615a9-c6c5-11e8-932e-da198c351b5a</dsmId>
    </writer>
    <isWriterElected>true</isWriterElected>
    <isWriter>true</isWriter>
    <gsn>5154</gsn>
    <transfersInProgress>0</transfersInProgress>
  </replicatedDirectory>
</replicatedDirectories>
Add virtual bucket

Sample Configuration XML to Add virtual bucket: config.xml

<vbucket name="vbucket" repl-path="/Newrepl">

      <bucket name="chcontainer-nnpxy01-vm4">
        <storage-type>AZURE</storage-type>
        <zonename>zone1-nnpxy01-vm4</zonename>
        <accesskey>chvanmathi</accesskey>
        <secretaccesskey>wMUSV7rkwRblkfQfyNOhkdPrOyr8ctFZRXF/WvZ/ASrY8y6CPXavjKDO9bB1GyRYaQvRaOPOBwKLjo8Gy31fGw==</secretaccesskey>
        <endpoint-url>http://chvanmathi.blob.core.windows.net/</endpoint-url>
      </bucket>

      <bucket name="chcontainer-nnpxy01-vm5">
        <storage-type>AZURE</storage-type>
        <zonename>zone2-nnpxy01-vm5</zonename>
        <accesskey>chvanmathi</accesskey>
        <secretaccesskey>wMUSV7rkwRblkfQfyNOhkdPrOyr8ctFZRXF/WvZ/ASrY8y6CPXavjKDO9bB1GyRYaQvRaOPOBwKLjo8Gy31fGw==</secretaccesskey>
        <endpoint-url>http://chvanmathi.blob.core.windows.net/</endpoint-url>
      </bucket>


   <exclusions>
     <exclusion>.*/.fusion</exclusion>
     <exclusion>.*/.fusion/.*</exclusion>
   </exclusions>


   <inclusions>
     <inclusion>.*</inclusion>
   </inclusions>

   </vbucket>
Example
[root@nnpxy01-vm5 tmp]# curl -v -i -X PUT -H "Content-Type: application/xml" -d @config.xml "http://nnpxy01-vm5.bdfrem.wandisco.com:8082/plugin/multicloud/config/vbucket?membershipId=edd9786c-1a8d-4d7a-9466-4cd09f1644f1"
* About to connect() to nnpxy01-vm5.bdfrem.wandisco.com port 8082 (#0)
*   Trying 10.10.214.105... connected
* Connected to nnpxy01-vm5.bdfrem.wandisco.com (10.10.214.105) port 8082 (#0)
> PUT /plugin/multicloud/config/vbucket?membershipId=edd9786c-1a8d-4d7a-9466-4cd09f1644f1 HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: nnpxy01-vm5.bdfrem.wandisco.com:8082
> Accept: */*
> Content-Type: application/xml
> Content-Length: 1038
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
HTTP/1.1 100 Continue

< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Location: http://nnpxy01-vm5.bdfrem.wandisco.com:8082/fusion/task/9e72e70c-c84c-11e8-a87c-e261d5d5398d
Content-Location: http://nnpxy01-vm5.bdfrem.wandisco.com:8082/fusion/task/9e72e70c-c84c-11e8-a87c-e261d5d5398d
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)

<
* Connection #0 to host nnpxy01-vm5.bdfrem.wandisco.com left intact
* Closing connection #0
[root@nnpxy01-vm5 tmp]#
Add virtual bucket status
[root@nnpxy01-vm5 tmp]# curl -X GET http://nnpxy01-vm5.bdfrem.wandisco.com:8082/fusion/task/9e72e70c-c84c-11e8-a87c-e261d5d5398d | xmllint --format -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
111   894  111   894    0     0   201k      0 --:--:-- --:--:-- --:--:--  291k
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<dsmTask>
  <taskId>9e72e70c-c84c-11e8-a87c-e261d5d5398d</taskId>
  <timeCreated>1538709203622</timeCreated>
  <creatorNodeId>d1cf01d6-72f8-40cf-846f-9e1a59d2d30f</creatorNodeId>
  <timeUpdated>1538709204304</timeUpdated>
  <isDone>true</isDone>
  <aborted>false</aborted>
  <properties>
    <entry>
      <key>PROPOSAL_SENT</key>
      <value>true</value>
    </entry>
    <entry>
      <key>TASK_TYPE</key>
      <value>UPDATE_VBUCKET_CONFIG_BY_PATH</value>
    </entry>
  </properties>
  <previousTask>9e6eef6b-c84c-11e8-a87c-e261d5d5398d</previousTask>
  <message>VBucketUpdateProposal [virtualBucketDTO=com.wandisco.multicloud.dto.VirtualBucketDTO@2c4708da, operation=CREATE, path=/Newrepl, taskIdentity=9e72e70c-c84c-11e8-a87c-e261d5d5398d, proposerNodeId=d1cf01d6-72f8-40cf-846f-9e1a59d2d30f, origin=zone2-nnpxy01-vm5, vBucketName=vbucket]</message>
  <dsmId>9e6ec859-c84c-11e8-a87c-e261d5d5398d</dsmId>
</dsmTask>
[root@nnpxy01-vm5 tmp]#
List virtual buckets

This REST API gives out the listing of virtual bucket configured in the proxy-plugin-site.xml.

For example

[root@nnpxy01-vm5 tmp]# curl -v -X GET 'http://nnpxy01-vm5.bdfrem.wandisco.com:8082/plugin/multicloud/config/vbucket?vbucket=chbucket' | xmllint --format -
* About to connect() to nnpxy01-vm5.bdfrem.wandisco.com port 8082 (#0)
*   Trying 10.10.214.105... connected
* Connected to nnpxy01-vm5.bdfrem.wandisco.com (10.10.214.105) port 8082 (#0)
> GET /plugin/multicloud/config/vbucket?vbucket=chbucket HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: nnpxy01-vm5.bdfrem.wandisco.com:8082
> Accept: */*
>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 OK
< Content-Length: 1381
< Content-Type: application/xml
< Server: Jetty(6.1.26)
<
{ [data not shown]
106  1381  106  1381    0     0  52348      0 --:--:-- --:--:-- --:--:-- 55240* Connection #0 to host nnpxy01-vm5.bdfrem.wandisco.com left intact

* Closing connection #0
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<multicloud>
  <vbucket name="chbucket" repl-path="/repl">
    <bucket name="chcontainer-nnpxy01-vm4">
      <storage-type>AZURE</storage-type>
      <zonename>zone1-nnpxy01-vm4</zonename>
      <accesskey>chvanmathi</accesskey>
      <secretaccesskey>wMUSV7rkwRblkfQfyNOhkdPrOyr8ctFZRXF/WvZ/ASrY8y6CPXavjKDO9bB1GyRYaQvRaOPOBwKLjo8Gy31fGw==</secretaccesskey>
      <provider/>
      <endpoint-url>http://chvanmathi.blob.core.windows.net/</endpoint-url>
      <chunked-encoding>true</chunked-encoding>
      <path-style-access>true</path-style-access>
      <endPointProtocol>http</endPointProtocol>
      <endPointHost>chvanmathi.blob.core.windows.net</endPointHost>
    </bucket>
    <bucket name="chcontainer-nnpxy01-vm5">
      <storage-type>AZURE</storage-type>
      <zonename>zone2-nnpxy01-vm5</zonename>
      <accesskey>chvanmathi</accesskey>
      <secretaccesskey>wMUSV7rkwRblkfQfyNOhkdPrOyr8ctFZRXF/WvZ/ASrY8y6CPXavjKDO9bB1GyRYaQvRaOPOBwKLjo8Gy31fGw==</secretaccesskey>
      <provider/>
      <endpoint-url>http://chvanmathi.blob.core.windows.net/</endpoint-url>
      <chunked-encoding>true</chunked-encoding>
      <path-style-access>true</path-style-access>
      <endPointProtocol>http</endPointProtocol>
      <endPointHost>chvanmathi.blob.core.windows.net</endPointHost>
    </bucket>
    <exclusions>
      <exclusion>.*/.fusion</exclusion>
      <exclusion>.*/.fusion/.*</exclusion>
    </exclusions>
    <inclusions>
      <inclusion>.*</inclusion>
    </inclusions>
  </vbucket>
</multicloud>
Modify virtual buckets

This REST API modifies virtual bucket in the proxy-plugin-site.xml. The bucket details for modification has to be given as an xml file, from which the details will be taken and specified bucket name will be modified in the proxy-plugin-site.xml.

Sample Configuration XML to Modify Virtual Bucket: config.xml
<vbucket name="vbucket" repl-path="/Newrepl">

      <bucket name="chcontainer-New">
        <storage-type>AZURE</storage-type>
        <zonename>zone1-nnpxy01-vm4</zonename>
        <accesskey>chvanmathi</accesskey>
        <secretaccesskey>wMUSV7rkwRblkfQfyNOhkdPrOyr8ctFZRXF/WvZ/ASrY8y6CPXavjKDO9bB1GyRYaQvRaOPOBwKLjo8Gy31fGw==</secretaccesskey>
        <endpoint-url>http://chvanmathi.blob.core.windows.net/</endpoint-url>
      </bucket>

      <bucket name="chcontainer-New">
        <storage-type>AZURE</storage-type>
        <zonename>zone2-nnpxy01-vm5</zonename>
        <accesskey>chvanmathi</accesskey>
        <secretaccesskey>wMUSV7rkwRblkfQfyNOhkdPrOyr8ctFZRXF/WvZ/ASrY8y6CPXavjKDO9bB1GyRYaQvRaOPOBwKLjo8Gy31fGw==</secretaccesskey>
        <endpoint-url>http://chvanmathi.blob.core.windows.net/</endpoint-url>
      </bucket>


   <exclusions>
     <exclusion>ex.*</exclusion>
     <exclusion>.*/.fusion/.*</exclusion>
   </exclusions>


   <inclusions>
     <inclusion>in.*</inclusion>
   </inclusions>

   </vbucket>
Example
[root@nnpxy01-vm5 tmp]# curl -v -i -X POST -H "Content-Type: application/xml" -d @config.xml "http://nnpxy01-vm5.bdfrem.wandisco.com:8082/plugin/multicloud/config/vbucket"
* About to connect() to nnpxy01-vm5.bdfrem.wandisco.com port 8082 (#0)
*   Trying 10.10.214.105... connected
* Connected to nnpxy01-vm5.bdfrem.wandisco.com (10.10.214.105) port 8082 (#0)
> POST /plugin/multicloud/config/vbucket HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: nnpxy01-vm5.bdfrem.wandisco.com:8082
> Accept: */*
> Content-Type: application/xml
> Content-Length: 1018
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Location: http://nnpxy01-vm5.bdfrem.wandisco.com:8082/fusion/task/ca9bdf63-c84d-11e8-a87c-e261d5d5398d
Content-Location: http://nnpxy01-vm5.bdfrem.wandisco.com:8082/fusion/task/ca9bdf63-c84d-11e8-a87c-e261d5d5398d
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)

<
* Connection #0 to host nnpxy01-vm5.bdfrem.wandisco.com left intact
* Closing connection #0
[root@nnpxy01-vm5 tmp]#
Modify virtual buckets Status
[root@nnpxy01-vm5 tmp]# curl -X GET http://nnpxy01-vm5.bdfrem.wandisco.com:8082/fusion/task/ca9bdf63-c84d-11e8-a87c-e261d5d5398d | xmllint --format -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
101   913  101   913    0     0   134k      0 --:--:-- --:--:-- --:--:--  178k
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<dsmTask>
  <taskId>ca9bdf63-c84d-11e8-a87c-e261d5d5398d</taskId>
  <timeCreated>1538709707205</timeCreated>
  <creatorNodeId>d1cf01d6-72f8-40cf-846f-9e1a59d2d30f</creatorNodeId>
  <timeUpdated>1538709707365</timeUpdated>
  <isDone>true</isDone>
  <aborted>false</aborted>
  <properties>
    <entry>
      <key>PROPOSAL_SENT</key>
      <value>true</value>
    </entry>
    <entry>
      <key>TASK_TYPE</key>
      <value>UPDATE_VBUCKET_CONFIG_BY_PATH</value>
    </entry>
  </properties>
  <previousTask xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
  <message>VBucketUpdateProposal [virtualBucketDTO=com.wandisco.multicloud.dto.VirtualBucketDTO@5f192ae3, operation=UPDATE, path=/Newrepl, taskIdentity=ca9bdf63-c84d-11e8-a87c-e261d5d5398d, proposerNodeId=d1cf01d6-72f8-40cf-846f-9e1a59d2d30f, origin=zone2-nnpxy01-vm5, vBucketName=vbucket]</message>
  <dsmId>9e6ec859-c84c-11e8-a87c-e261d5d5398d</dsmId>
</dsmTask>
[root@nnpxy01-vm5 tmp]#
Delete virtual buckets

This REST API delete virtual buckets from the proxy-plugin-site.xml. The requested virtualBucket/bucket will be deleted from the proxy-plugin-site.xml

IMFORMATION: If the deleteRD is false, the virtual bucket will be remove from the proxy-plugin-site.xml but repl directory will not remove from the UI.

If the deleteRD is true, the virtual bucket will be remove from the proxy-plugin-site.xml and also repl directory from the UI.

[root@nnpxy01-vm5 tmp]# curl -v -i -X DELETE "http://nnpxy01-vm5.bdfrem.wandisco.com:8082/plugin/multicloud/config/vbucket?vbucket=vbucket&deleteRD=true"
* About to connect() to nnpxy01-vm5.bdfrem.wandisco.com port 8082 (#0)
*   Trying 10.10.214.105... connected
* Connected to nnpxy01-vm5.bdfrem.wandisco.com (10.10.214.105) port 8082 (#0)
> DELETE /plugin/multicloud/config/vbucket?vbucket=vbucket&deleteRD=true HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: nnpxy01-vm5.bdfrem.wandisco.com:8082
> Accept: */*
>
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< Content-Location: http://nnpxy01-vm5.bdfrem.wandisco.com:8082/fusion/task/829112e4-c84e-11e8-a87c-e261d5d5398d
Content-Location: http://nnpxy01-vm5.bdfrem.wandisco.com:8082/fusion/task/829112e4-c84e-11e8-a87c-e261d5d5398d
< Content-Length: 0
Content-Length: 0
< Server: Jetty(6.1.26)
Server: Jetty(6.1.26)

<
* Connection #0 to host nnpxy01-vm5.bdfrem.wandisco.com left intact
* Closing connection #0
[root@nnpxy01-vm5 tmp]#
Support for non-replicated buckets
S3 Proxy

The non-replicated buckets can be accessed via Live S3, only if the buckets is created under the storage account which is configured in the plugin-site.xml

We can access the non-replicated buckets via s3proxy with below signature and addressing styles

  1. v2 Path style

  2. v4 Path style

Azure Blob

The non-replicated containers can be accessed via Live Azure blob, only if the container is created under the storage account which is configured in the plugin-site.xml