XM to XM Cloud Migration tool - 2.1 - Using GUI

XM to XM Cloud Migration tool has two ways to use. GUI and CLI. In this blog post, we will go through GUI. 

When we open the GUI app, it gives an option to choose Content and Media and/or Users. When users are selected to migrate, it will take the user email and invite them to the Sitecore Cloud Portal. 

In the next screen, we have the options to enter the Source CM Url, and an admin username and password. 

In the next screen, target XM Cloud details are provided. XM Cloud Environment ID, CM base URL, Client ID and Client Secret. In this screen, there is a clear instruction on where to get these details in XM Cloud application. Please note that credentials should be Organization credentials as it needs that level access to invite the users to the Cloud portal. 

If the source environment details are correct, then we will see the source content tree in the next screen. We can select the content, media, templates. Please avoid any items which are available out of the box in the target system. 

When we select Templates and Content, this tool automatically changes the order to migrate Templates first and then migrate the Content. 

In the next screen, if all the details are valid, we will get the success confirmation. If not, we will get failure to connect details.

Once we click on Migrate, tool will start doing the magic of extracting the content from the source environment as Protobuf method as it is a faster serialization mechanism, split it into small batches (RAIF) and migrate it to XM Cloud.

If everything goes well, we will get the success message and content will be migrated to the XM Cloud.

Appsettings.json

Whatever value which is entered via GUI are stored in Appsettings.json file. The same file can be used for migrating the content using CLI. We will discuss in another blog post on CLI. 

Happy Migrating!!!

XM to XM Cloud Migration tool - 2.1 - Overview of Application Configuration

This is a continuation of XM to XM Cloud Migration Tool blog posts. Previously I explained the installation process here. In this post, I will explain the application setting file. 

There is a file named appsettings.json created whenever we use this tool. When the tool is downloaded from Sitecore Developers portal, the archive extract does not have this configuration. It will be automatically created when we run the GUI or CLI version of this tool. 

When the json file is auto-created for the first time, it is loaded with default values. When we start using the GUI app, the values that are entered in the GUI app will be stored in this json file. 

Some of the mandatory fields are needed for the tool to work.

Source

• SitecoreCmBaseUri - The URL to the Sitecore XM CM on-premise instance.
• SitecoreCmUserName - The URL to the Sitecore XM CM on-premise instance.
• SitecoreCmPassword - The password to login to the Sitecore XM CM on-premise instance.

Target

• SitecoreXmCloudEnvironmentId - The environment ID of the XM Cloud instance.
• SitecoreXmCloudCmBaseUri - The XM Cloud CM instance base url.
• SitecoreXmCloudClientId - The XM Cloud Client ID.
• SitecoreXmCloudClientSecret - The XM Cloud Client Secret.

Apart from these, we have a section where we can configure the content to be migrated to the target system. It is similar to SCS config or Unicorn. 

Name is an unique name for that section and Path is the Sitecore content tree path to be migrated to the target system. ContenTransferScope supports only one option at this time - ItemAndDescendants. This allows to migrate the item and its descendants under that folder or item to the target system. For MergeStrategy, there is only one option supported at this time – OverrideExistingItem which is going to overwrite it in the target system. 

As this tool supports the migration of content in batches, there are two additional settings by which we can control the transfer speed. By default, these settings are pre-populated with an optimal number. In case if you want to increase or decrease, we can make this change in the json file. 

RAIF Configuration - The Sitecore XM to XM Cloud Migration Tool transfers content and media via RAIFs (Batches). The settings below can be adjusted to better suit transfer needs. The default values have been set to provide a balanced transfer. Most customers will not need to adjust these values.

ItemsPerRaif define the amount of the transferred items. Too low setting of ItemsPerRaif can result in a large number of RAIF files being transferred, where too high of ItemsPerRaif can affect the transfer speed. ItemsPerRaif default is 10000 items per RAIF file. 

MediaSizeLimit - The media size limit per RAIF. User can set the value of the MediaSizeLimit property as needed. Assign only long value. No decimal or string. The default value is 104857600 which is 100MB. Example - A RAIF can contain maximum 100MB of media, if a media file exceeds the 100MB, the one media item will be in a single RAIF.

Additionally, the json configuration has settings related to inviting users to XM Cloud organization. This is used when we migrate the source Sitecore users to XM Cloud. 

Happy Migrating!!!

Coveo for Sitecore - Endpoint deprecation issues and resolution - TLDR

Coveo planned to deprecate their Coveo for Sitecore endpoint and requested clients to use Organization specific endpoint for better control over traffic and rate limiting. I wrote a blog article earlier and provided the steps to migrate to the organization specific endpoint. 

Coveo for Sitecore - Endpoint Deprecation & Migrate to Organization Specific Endpoint: https://www.nehemiahj.com/2024/08/coveo-for-sitecore-endpoint-deprecation.html

Even after migrating to the right organization specific endpoint, we were still informed that we are using the deprecated endpoint and we may have issues to analytics and search related API calls. In Coveo Administration Console, Visit browser / event browser feature does not show the endpoint being used for the incoming traffic. 

So every time, we wanted to test whether it reaches the right endpoint, we worked with Coveo support and they provided the results. Results were contradictory. Sometimes it reached the right endpoint and sometimes it reached the wrong one 🤷‍♀️. Our setup is described below.

• We have Cloudflare as our entry point. 
• Azure Front Door is used for routing as we have multiple legacy sites and Sitecore sites. 
• Sitecore site is hosted in Azure VM as IaaS setup. 
• Coveo for Sitecore module is used to setup the Coveo related pages. 

As Coveo support requested, I did few changes to Cloudflare and Azure Front Door.

• Bypass Cache for Coveo requests in Cloudflare. Bypassed for any URL path /coveo/rest/*
• Disable cache and WAF in Azure Front Door - We didn't use WAF and Cache in Azure Front Door. So no changes were implemented. 

Scenario 1: Disabled the Sitecore proxy

Disabled the Sitecore proxy through the command center. How to do it: https://docs.coveo.com/en/3401/coveo-for-sitecore-v5/disable-the-reverse-proxy.

Result: All Coveo requests reached the correct organization endpoint. I could even see it in the network tab as Coveo requests are routed directly from client browser to Coveo organization endpoint domain. 

Scenario 2: Enabled the Sitecore proxy & requests from internet

This is a regular scenario where proxy is enabled in Sitecore and requests were made from internet. 

Result: All Coveo requests reached the deprecated endpoint. Coveo requested us to implement Bypass cache in Cloudflare and disable cache & WAF in Azure Front Door as mentioned above. But Coveo support mentioned that requests are sent to deprecated endpoint. 

Scenario 3: Enabled the Sitecore proxy & requests sent inside Sitecore Server

This was just a random request I made and Coveo support checked their side whether it is sent to deprecated endpoint or not. 

Result: All Coveo requests reached the correct organization endpoint. 

Coveo support brought in core engineering team to the discussion and they were skeptical that some rules in Cloudflare or Azure Front Door transforms the domain to the wrong domain. I was sure that there is no outbound rules which could have caused this issue. They asked us involve our network team to investigate as there is no problem with Coveo. 

In-Depth Analysis

Enabled Coveo Debug Logging:

We enabled Coveo Debug logging to understand the domain which is used to send the Coveo proxy request. To enable debug logging, you can modify the <log4net> <logger> section in this file \App_Config\Include\Coveo\Coveo.SearchProvider.Custom.config. 

After enabling debug logging, Coveo related logs were captured in Coveo.Search.txt file. It clearly showed that we are sending the traffic to Organization specific endpoint (which is the correct endpoint) in all the scenarios. We had to prove that to Coveo by other means. 

Captured Coveo Header Information with Domain:

Since I cannot install any sort of network tools in the server, I created a simple .NET app to read the incoming request and log the header details in a text file. I hosted it in a separate development Azure VM with public access. The bindings for this app are Coveo deprecated endpoint and Coveo organization specific endpoint domain. In the Sitecore Azure VM, I added the Coveo organization endpoint domain and deprecated endpoint domain in the server host file pointing to the development VM. 

So any requests to Coveo deprecated endpoint and organization endpoint will reach the app hosted in the development VM. I logged the entire request header information in both the scenarios. From my side, I confirmed that all the requests are having the right endpoint so it is not our problem but something to do with Coveo network side. We asked Coveo support to involve their network team and we shared the entire log information. 

End Result

Coveo Support found the issue at their side. When the Sitecore proxied request reached Coveo and if it has X-Forwarded-Host header, then Coveo considered as invalid request and routed all traffic to deprecated endpoint. The request did not have X-Forwarded-Host header if we send the request inside the server (without Cloudflare or Azure Front Door) so those requests were marked as valid and routed to organization endpoint. Unfortunately we do not have any control to customize the request to Coveo using Sitecore proxy as it is part of the Coveo for Sitecore module. 

On Nov13th 2024, Coveo support released the fix at their side and we gave a couple of Visit IDs to test and they confirmed that it reached the organization endpoint. Also, Coveo extended their deprecation deadline to one last time to June 3rd 2025. 

The X-Forwarded-Host (XFH) header is a de-facto standard header for identifying the original host requested by the client in the Host HTTP request header. Host names and ports of reverse proxies (load balancers, CDNs) may differ from the origin server handling the request, in that case the X-Forwarded-Host header is useful to determine which Host was originally used. - Mozilla Developer Docs.

Since we have Cloudflare and Azure Front Door, the X-Forwarded-Host header is added to the request and it reached Sitecore server. Coveo for Sitecore module uses the headers from the original request, transforms only a set of headers with different value and copies the rest of the headers from original request to the proxied Coveo request. 

Once Coveo made changes at their side, this started to work for every Coveo clients. 

Our traffic data is also moving towards organization specific endpoint.

Happy Customers 😊😊😊