Migrate API
Same name in other branches
- 8.9.x core/modules/migrate/migrate.api.php \migration
- 10 core/modules/migrate/migrate.api.php \migration
- 11.x core/modules/migrate/migrate.api.php \migration
Overview of the Migrate API, which migrates data into Drupal.
Overview of a migration
Migration is an Extract, Transform, Load (ETL) process. In the Drupal Migrate API, the extract phase is called 'source', the transform phase is called 'process', and the load phase is called 'destination'. It is important to understand that the term 'load' in ETL refers to loading data into the storage while in a typical Drupal context the term 'load' refers to loading data from storage.
In the source phase, a set of data, called the row, is retrieved from the data source. The data can be migrated from a database, loaded from a file (for example CSV, JSON or XML) or fetched from a web service (for example RSS or REST). The row is sent to the process phase where it is transformed as needed or marked to be skipped. After processing, the transformed row is passed to the destination phase where it is loaded (saved) into the target Drupal site.
Migrate API uses the Drupal plugin system for many different purposes. Most importantly, the overall ETL process is defined as a migration plugin and the three phases (source, process and destination) have their own plugin types.
Migrate API migration plugins
Migration plugin definitions are stored in a module's 'migrations' directory. The plugin class is \Drupal\migrate\Plugin\Migration, with interface \Drupal\migrate\Plugin\MigrationInterface. Migration plugins are managed by the \Drupal\migrate\Plugin\MigrationPluginManager class. Migration plugins are only available if the providers of their source plugins are installed.
Example migrations in Migrate API handbook.
Migrate API source plugins
Migrate API source plugins implement \Drupal\migrate\Plugin\MigrateSourceInterface and usually extend \Drupal\migrate\Plugin\migrate\source\SourcePluginBase. They are annotated with \Drupal\migrate\Annotation\MigrateSource annotation and must be in namespace subdirectory 'Plugin\migrate\source' under the namespace of the module that defines them. Migrate API source plugins are managed by the \Drupal\migrate\Plugin\MigrateSourcePluginManager class.
List of source plugins provided by the core Migrate module. Core and contributed source plugin usage examples in Migrate API handbook.
Migrate API process plugins
Migrate API process plugins implement \Drupal\migrate\Plugin\MigrateProcessInterface and usually extend \Drupal\migrate\ProcessPluginBase. They are annotated with \Drupal\migrate\Annotation\MigrateProcessPlugin annotation and must be in namespace subdirectory 'Plugin\migrate\process' under the namespace of the module that defines them. Migrate API process plugins are managed by the \Drupal\migrate\Plugin\MigratePluginManager class.
List of process plugins for common operations provided by the core Migrate module.
Migrate API destination plugins
Migrate API destination plugins implement \Drupal\migrate\Plugin\MigrateDestinationInterface and usually extend \Drupal\migrate\Plugin\migrate\destination\DestinationBase. They are annotated with \Drupal\migrate\Annotation\MigrateDestination annotation and must be in namespace subdirectory 'Plugin\migrate\destination' under the namespace of the module that defines them. Migrate API destination plugins are managed by the \Drupal\migrate\Plugin\MigrateDestinationPluginManager class.
Migrate API key concepts
Stubs
Taxonomy terms are an example of a data structure where an entity can have a reference to a parent. When a term is being migrated, it is possible that its parent term has not yet been migrated. Migrate API addresses this 'chicken and egg' dilemma by creating a stub term for the parent so that the child term can establish a reference to it. When the parent term is eventually migrated, Migrate API updates the previously created stub with the actual content.
Map tables
Once a migrated row is saved and the destination IDs are known, Migrate API saves the source IDs, destination IDs, and the row hash into a map table. The source IDs and the hash facilitate tracking changes for continuous migrations. Other migrations can use the map tables for lookup purposes when establishing relationships between records.
High-water mark
A High-water mark allows the Migrate API to track changes so that only data that has been created or updated in the source since the migration was previously executed is migrated. The only requirement to use the high-water feature is to declare the row property to use for the high-water mark. This can be any property that indicates the highest value migrated so far. For example, a timestamp property that indicates when a row of data was created or last updated would make an excellent high-water property. If the migration is executed again, only those rows that have a higher timestamp than in the previous migration would be included.
source:
plugin: d7_node
high_water_property:
name: changed
In this example, the row property 'changed' is the high_water_property. If the value of 'changed' is greater than the current high-water mark the row is processed and the value of the high-water mark is updated to the value of 'changed'.
Rollbacks
When developing a migration, it is quite typical that the first version does not provide correct results for all migrated data. Rollbacks allow you to undo a migration and then execute it again after adjusting it.
Documentation handbooks
Migrate API handbook. Upgrading to Drupal 8 handbook.
File
-
core/
modules/ migrate/ migrate.api.php, line 12
Functions
Title Sort descending | File name | Summary |
---|---|---|
hook_migrate_MIGRATION_ID_prepare_row | core/ |
Allows adding data to a row for a migration with the specified ID. |
hook_migrate_prepare_row | core/ |
Allows adding data to a row before processing it. |
hook_migration_plugins_alter | core/ |
Allows altering the list of discovered migration plugins. |
Classes
Title Sort descending | File name | Summary |
---|---|---|
DestinationBase | core/ |
Base class for migrate destination classes. |
FieldPluginBase | core/ |
The base class for all field plugins. |
MigrateDestination | core/ |
Defines a migration destination plugin annotation object. |
MigrateDestinationPluginManager | core/ |
Plugin manager for migrate destination plugins. |
MigrateFieldPluginManager | core/ |
Plugin manager for migrate field plugins. |
MigratePluginManager | core/ |
Manages migrate plugins. |
MigrateProcessPlugin | core/ |
Defines a migration process plugin annotation object. |
MigrateSource | core/ |
Defines a migration source plugin annotation object. |
MigrateSourcePluginManager | core/ |
Plugin manager for migrate source plugins. |
ProcessPluginBase | core/ |
The base class for all migrate process plugins. |
SourcePluginBase | core/ |
The base class for source plugins. |
Interfaces
Title Sort descending | File name | Summary |
---|---|---|
MigrateDestinationInterface | core/ |
Defines an interface for Migration Destination classes. |
MigrateProcessInterface | core/ |
An interface for migrate process plugins. |
MigrateSourceInterface | core/ |
Defines an interface for migrate sources. |
MigrateValidatableEntityInterface | core/ |
To implement by a destination plugin that should provide entity validation. |
Buggy or inaccurate documentation? Please file an issue. Need support? Need help programming? Connect with the Drupal community.