Steps for a Sitecore xDB Analytics Data Migration

As Oshyn, we organize an upgrade project into the following areas:

1. Code/Config Migration
2. Content Migration
3. Analytics Data Migration
4. Environment Setup
5. CI/CD pipelines
6. UI Tests to test site before migration and post migration

In this blog post, we’ll be discussing the third area: Analytics Data Migration. Since v7.5, the Sitecore platform collects analytics data in Mongo Database corresponding to visitors’ page visits and other user interactions with the site. With Sitecore 9, the architecture was updated to collect data in SQL server/MongoDB. Hence, during an upgrade from an earlier version of Sitecore to a 9.x version, this treasure trove of data that has been collected must be migrated. Migration of this data involves the following steps:

The Sitecore documentation for migration provides for detailed steps as to how to perform each of these phases. Here, I’ll provide you an overview of the different steps.

Configure Prerequisites:

There are multiple prerequisites:

1. Enable Verification: Although this might slow down the migration, it’s important for ensuring the validity of the migration.
2. Disable Recycling App Pool on Target Sitecore: To avoid the migration process being impacted by application pool recycles, this setting in IIS is disabled.
3. Add Indexes on MongoDB: To speed up the retrieval process of records from Mongo DB, add indexes in Mongo database for devices, and interactions.
4. Set ChangeTracking to 4 hours: Configure change tracking retention on the Shard0, Shard1 databases to be 4 hrs or even lower from default 5 days. This would prevent SQL server from keeping tracked changes for longer duration thereby reducing the performance of SQL Server.
5. Set MaxRequestLength to 2147483647: Set the MaxRequestLength on xConnect server to be as high as possible 2147483647 to avoid requests being rejected due to shorter length.

Install xDB Migration tool:

Install the Sitecore packages in this order to install the xDB migration tool.

1. Data Exchange Framework 2.1
2. MongoDB Provider for Data Exchange Framework 2.1
3. xConnect Provider for Data Exchange Framework 2.1
4. xDB Data Migration Tool for Data Exchange Framework 2.1
5. Sitecore Data Exchange Framework SDK 2.1

Configure xDB migration:

1. Add relevant connection strings for Mongo DB, xconnect.collection.
2. Deploy the collection model.
3. Create tenant and configure endpoint for Mongo and xconnect collection and xdb.referencedata.client endpoints.
4. Configure verification database to be able to monitor the migration.

Perform xDB Migration:

1. Enter 10 or 100 in the maximum count field in reading contacts from Mongo DB to test on small subset the migration.
2. Run the migration

Verify Migration:

1. Check on the verification database for the batch id and query the tables for the monitoring the status of the migration.
2. Check that the analytics reports are showing corresponding to the migration.

Now that this is verified, we perform the following fine tuning to run migration faster and completely on all the records:

1. Remove the limit set to 100 (or 10) above.
2. Change the Max Thread Count in the following item to 10 or above based on the resources of the machine running the job. Navigate to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read Contacts from MongoDB Pipeline > Iterate MongoDB Contacts and Run Pipelines.
3. Change the Minimum batch size to 200 or 300 or above based on resources in the following item: PipelinesMongoDB Contacts to xConnect Migration PipelinesRead Contacts from MongoDB Pipeline/Add xConnect Client to Context.