Create exclusion templates

Exclusion templates are sets of rules that prevent certain file sizes or file names (defined using glob patterns) from being migrated to your target storage. You can create an exclusion template in the Azure Portal or with the CLI, which you can then apply to a migration as an exclusion. Exclusions are optional: you don't need to create any exclusions beyond the default exclusions to create a migration.

note

Deleting an exclusion template will not cause exclusions based on it to be removed from current migrations.

  1. In the Azure Portal, navigate to the LiveData Migrator resource page.
  2. From the LiveData Migrator menu on the left, select Exclusion Templates.
  3. Select Create.
  4. Complete the details under the Basics section:
    • Exclusion Name The name for the exclusion template.
    • Exclusion Type Pick one from the drop down menu:
      • File Size Exclusion = Value / Unit - The value and unit for the file size limit (for example: 100 GB).
      • Pattern = Pattern - The glob pattern to be used for excluding file names and directory names (for example: /**/test*).
      • Date Exclusion = Date - Migrations with this template will exclude any files that were last modified before the specified date and time in ISO format (for example: 2021-04-15T09:28:18.912Z). If you don't specify a time, the exclusion uses 00:00 Zulu time.
    • Description A brief description of what the exclusion is doing.
  5. Select Review + create.
  6. Select Create.

Default exclusions#

LiveData Migrator for Azure automatically applies default exclusions. For example, any individual files larger than 4.55TiB (or 5.002 TB) are automatically excluded.

You can remove default exclusions from the migration, but not from the system or the exclusion templates list.

The default exclusions are:

ExclusionExclusion typeDescription
(/|/.*/)\\.hive-staging.*Regex (Automata)Hive staging directories
(/|/.*/)\\.spark-staging-.*Regex (Automata)Spark staging directories
(/|/.*/)_temporary.*Regex (Automata)Spark temporary directories
[.|\\/]$Regex (JAVA_PCRE)File names cannot end with . or ' '
.*([^\\/]*\\/){255,}.*Regex (Automata)Blob names cannot exceed 254 path segments
.{1025,}Regex (JAVA_PCRE)File name length cannot exceed 1024
5 TBFile sizeFile size cannot exceed 5TB

These exclusions cover many of the limitations set by ADLS Gen2 directory and file naming rules.

Next steps#

Once you've created exclusion templates, add them to new or existing migrations as exclusions. See the required steps in Migrate your data.

Troubleshooting#

If you are having trouble creating exclusion templates, you can find solutions in the troubleshooting guide.