Create exclusion templates

Exclusion templates are sets of rules that prevent certain file sizes or file names (defined using glob patterns) from being migrated to your target storage. You can create an exclusion template in the Azure Portal or with the CLI, which you can then apply to a migration as an exclusion. Exclusions are optional: you don't need to create any exclusions beyond the default exclusions to create a migration.

note

Deleting an exclusion template will not cause exclusions based on it to be removed from current migrations.

Create an exclusion template with the Azure Portal

1. In the Azure Portal, navigate to the LiveData Migrator resource page.
2. From the LiveData Migrator menu on the left, select Exclusion Templates.
3. Select Create.
4. Complete the details under the Basics section:
   • Exclusion Name The name for the exclusion template.
   • Exclusion Type Select one of the following:
     • File Size Exclusion = Value / Unit - The value and unit for the file size limit (for example: 100 GB).
     • Pattern = Pattern - The glob pattern for excluding file names and directory names (for example, /**/test*).
     • Date Exclusion = Datetime - Any files that have been modified before the specified date and time are excluded during migrations.
   • Description A brief description of what the exclusion does.
5. Select Review + create.
6. Select Create.

Create an exclusion template with the CLI

See the Azure CLI LiveData Extension - Exclusion Templates section for help with these parameters.

File size

Create a file size exclusion template that will prevent files being migrated if they exceed a certain size.

az livedata migrator exclusion-template create -g <resource_group> --migrator-name <migrator_name> \
--exclusion-template-name <exclusion_name> \
--exclude-files-over <file_size> \
--description <description>

Before date and time

Create a date-based exclusion template that checks the last modification date and time of a directory or file on the migration path. Migrations that use this template exclude directories and files that were last modified before the date and time specified.

Define the date and time in ISO format (for example, 2021-04-15T09:28:18.912Z). The time specified is the time for the LiveData Migrator host. If you don't specify a time, the exclusion uses 00:00 Zulu time.

az livedata migrator exclusion-template create -g <resource_group> --migrator-name <migrator_name> \
--exclusion-template-name <exclusion_name> \
--exclude-older-than <iso_date> \
--description <description>

File names and/or directories using glob

Create a glob exclusion template that will prevent files and/or directories being migrated if they match the glob pattern.

az livedata migrator exclusion-template create -g <resource_group> --migrator-name <migrator_name> \
--exclusion-template-name <exclusion_name> \
--exclude-pattern <glob> \
--description <description>

Manage exclusion templates

See our Azure CLI LiveData extension reference guide for commands to show and delete exclusion templates.

Default exclusions

LiveData Migrator for Azure automatically applies default exclusions. For example, any individual files larger than 4.55TiB (or 5.002 TB) are automatically excluded.

You can remove default exclusions from the migration, but not from the system or the exclusion templates list.

The default exclusions are:

Exclusion	Exclusion type	Description
(/|/.*/)\\.snapshot(/.*)?	Regex (Automata)	HDFS snapshots
/**/_temporary**	GLOB	Spark temporary directories
/**/.hive-staging**	GLOB	Hive Staging Content
.{1025,}	JAVA_PCRE	Filename length cannot exceed 1024
.\\._COPYING_$	JAVA_PCRE	HDFS Copying files
.*([^\\/]*\\/){61,}.*	Regex (Automata)	The number of path segments comprising a blob name cannot exceed 60.
/**/.spark-staging-**	GLOB	Spark Staging Content
[.|\\/]$	JAVA_PCRE	Names ending with '.' or '' character
(/|/.*/)\\.Trash(/.*)?	Regex (Automata)	HDFS Trash Locations
.*[\\\\].*	Regex (Automata)	Filepath or name cannot include a backslash.
5 TB	File size	File size cannot exceed 5TB

These exclusions cover many of the limitations set by ADLS Gen2 directory and file naming rules.

Next steps

Once you've created exclusion templates, add them to new or existing migrations as exclusions. See the required steps in Migrate your data.

Troubleshooting

If you are having trouble creating exclusion templates, you can find solutions in the troubleshooting guide.