Skip to main content

Create exclusion templates

Exclusion templates are sets of rules that prevent certain file sizes or file names (defined using glob patterns) from being migrated to your target storage. You can create an exclusion template in the Azure Portal or with the CLI, which you can then apply to a migration as an exclusion. Exclusions are optional: you don't need to create any exclusions beyond the default exclusions to create a migration.


Deleting an exclusion template will not cause exclusions based on it to be removed from current migrations.

  1. In the Azure Portal, navigate to the LiveData Migrator resource page.
  2. From the LiveData Migrator menu on the left, select Exclusion Templates.
  3. Select Create.
  4. Complete the details under the Basics section:
    • Exclusion Name The name for the exclusion template.
    • Exclusion Type Select one of the following:
      • File Size Exclusion = Value / Unit - The value and unit for the file size limit (for example: 100 GB).
      • Pattern = Pattern - The glob pattern for excluding file names and directory names (for example, /**/test*).
      • Date Exclusion = Datetime - Any files that have been modified before the specified date and time are excluded during migrations.
    • Description A brief description of what the exclusion does.
  5. Select Review + create.
  6. Select Create.

Default exclusions

LiveData Migrator for Azure automatically applies default exclusions. For example, any individual files larger than 4.55TiB (or 5.002 TB) are automatically excluded.

You can remove default exclusions from the migration, but not from the system or the exclusion templates list.

The default exclusions are:

ExclusionExclusion typeDescription
(/|/.*/)\\.snapshot(/.*)?Regex (Automata)HDFS snapshots
/**/_temporary**GLOBSpark temporary directories
/**/.hive-staging**GLOBHive Staging Content
.{1025,}JAVA_PCREFilename length cannot exceed 1024
.\\._COPYING_$JAVA_PCREHDFS Copying files
.*([^\\/]*\\/){61,}.*Regex (Automata)The number of path segments comprising a blob name cannot exceed 60.
/**/.spark-staging-**GLOBSpark Staging Content
[.|\\/]$JAVA_PCRENames ending with '.' or '' character
(/|/.*/)\\.Trash(/.*)?Regex (Automata)HDFS Trash Locations
.*[\\\\].*Regex (Automata)Filepath or name cannot include a backslash.
5 TBFile sizeFile size cannot exceed 5TB

These exclusions cover many of the limitations set by ADLS Gen2 directory and file naming rules.

Next steps

Once you've created exclusion templates, add them to new or existing migrations as exclusions. See the required steps in Migrate your data.


If you are having trouble creating exclusion templates, you can find solutions in the troubleshooting guide.