XM to XM Cloud Migration tool - 2.1 - Overview of Application Configuration

This is a continuation of XM to XM Cloud Migration Tool blog posts. Previously I explained the installation process here. In this post, I will explain the application setting file. 

There is a file named appsettings.json created whenever we use this tool. When the tool is downloaded from Sitecore Developers portal, the archive extract does not have this configuration. It will be automatically created when we run the GUI or CLI version of this tool. 

When the json file is auto-created for the first time, it is loaded with default values. When we start using the GUI app, the values that are entered in the GUI app will be stored in this json file. 

Some of the mandatory fields are needed for the tool to work.

Source

• SitecoreCmBaseUri - The URL to the Sitecore XM CM on-premise instance.
• SitecoreCmUserName - The URL to the Sitecore XM CM on-premise instance.
• SitecoreCmPassword - The password to login to the Sitecore XM CM on-premise instance.

Target

• SitecoreXmCloudEnvironmentId - The environment ID of the XM Cloud instance.
• SitecoreXmCloudCmBaseUri - The XM Cloud CM instance base url.
• SitecoreXmCloudClientId - The XM Cloud Client ID.
• SitecoreXmCloudClientSecret - The XM Cloud Client Secret.

Apart from these, we have a section where we can configure the content to be migrated to the target system. It is similar to SCS config or Unicorn. 

Name is an unique name for that section and Path is the Sitecore content tree path to be migrated to the target system. ContenTransferScope supports only one option at this time - ItemAndDescendants. This allows to migrate the item and its descendants under that folder or item to the target system. For MergeStrategy, there is only one option supported at this time – OverrideExistingItem which is going to overwrite it in the target system. 

As this tool supports the migration of content in batches, there are two additional settings by which we can control the transfer speed. By default, these settings are pre-populated with an optimal number. In case if you want to increase or decrease, we can make this change in the json file. 

RAIF Configuration - The Sitecore XM to XM Cloud Migration Tool transfers content and media via RAIFs (Batches). The settings below can be adjusted to better suit transfer needs. The default values have been set to provide a balanced transfer. Most customers will not need to adjust these values.

ItemsPerRaif define the amount of the transferred items. Too low setting of ItemsPerRaif can result in a large number of RAIF files being transferred, where too high of ItemsPerRaif can affect the transfer speed. ItemsPerRaif default is 10000 items per RAIF file. 

MediaSizeLimit - The media size limit per RAIF. User can set the value of the MediaSizeLimit property as needed. Assign only long value. No decimal or string. The default value is 104857600 which is 100MB. Example - A RAIF can contain maximum 100MB of media, if a media file exceeds the 100MB, the one media item will be in a single RAIF.

Additionally, the json configuration has settings related to inviting users to XM Cloud organization. This is used when we migrate the source Sitecore users to XM Cloud. 

Happy Migrating!!!

Coveo for Sitecore - Endpoint deprecation issues and resolution - TLDR

Coveo planned to deprecate their Coveo for Sitecore endpoint and requested clients to use Organization specific endpoint for better control over traffic and rate limiting. I wrote a blog article earlier and provided the steps to migrate to the organization specific endpoint. 

Coveo for Sitecore - Endpoint Deprecation & Migrate to Organization Specific Endpoint: https://www.nehemiahj.com/2024/08/coveo-for-sitecore-endpoint-deprecation.html

Even after migrating to the right organization specific endpoint, we were still informed that we are using the deprecated endpoint and we may have issues to analytics and search related API calls. In Coveo Administration Console, Visit browser / event browser feature does not show the endpoint being used for the incoming traffic. 

So every time, we wanted to test whether it reaches the right endpoint, we worked with Coveo support and they provided the results. Results were contradictory. Sometimes it reached the right endpoint and sometimes it reached the wrong one 🤷‍♀️. Our setup is described below.

• We have Cloudflare as our entry point. 
• Azure Front Door is used for routing as we have multiple legacy sites and Sitecore sites. 
• Sitecore site is hosted in Azure VM as IaaS setup. 
• Coveo for Sitecore module is used to setup the Coveo related pages. 

As Coveo support requested, I did few changes to Cloudflare and Azure Front Door.

• Bypass Cache for Coveo requests in Cloudflare. Bypassed for any URL path /coveo/rest/*
• Disable cache and WAF in Azure Front Door - We didn't use WAF and Cache in Azure Front Door. So no changes were implemented. 

Scenario 1: Disabled the Sitecore proxy

Disabled the Sitecore proxy through the command center. How to do it: https://docs.coveo.com/en/3401/coveo-for-sitecore-v5/disable-the-reverse-proxy.

Result: All Coveo requests reached the correct organization endpoint. I could even see it in the network tab as Coveo requests are routed directly from client browser to Coveo organization endpoint domain. 

Scenario 2: Enabled the Sitecore proxy & requests from internet

This is a regular scenario where proxy is enabled in Sitecore and requests were made from internet. 

Result: All Coveo requests reached the deprecated endpoint. Coveo requested us to implement Bypass cache in Cloudflare and disable cache & WAF in Azure Front Door as mentioned above. But Coveo support mentioned that requests are sent to deprecated endpoint. 

Scenario 3: Enabled the Sitecore proxy & requests sent inside Sitecore Server

This was just a random request I made and Coveo support checked their side whether it is sent to deprecated endpoint or not. 

Result: All Coveo requests reached the correct organization endpoint. 

Coveo support brought in core engineering team to the discussion and they were skeptical that some rules in Cloudflare or Azure Front Door transforms the domain to the wrong domain. I was sure that there is no outbound rules which could have caused this issue. They asked us involve our network team to investigate as there is no problem with Coveo. 

In-Depth Analysis

Enabled Coveo Debug Logging:

We enabled Coveo Debug logging to understand the domain which is used to send the Coveo proxy request. To enable debug logging, you can modify the <log4net> <logger> section in this file \App_Config\Include\Coveo\Coveo.SearchProvider.Custom.config. 

After enabling debug logging, Coveo related logs were captured in Coveo.Search.txt file. It clearly showed that we are sending the traffic to Organization specific endpoint (which is the correct endpoint) in all the scenarios. We had to prove that to Coveo by other means. 

Captured Coveo Header Information with Domain:

Since I cannot install any sort of network tools in the server, I created a simple .NET app to read the incoming request and log the header details in a text file. I hosted it in a separate development Azure VM with public access. The bindings for this app are Coveo deprecated endpoint and Coveo organization specific endpoint domain. In the Sitecore Azure VM, I added the Coveo organization endpoint domain and deprecated endpoint domain in the server host file pointing to the development VM. 

So any requests to Coveo deprecated endpoint and organization endpoint will reach the app hosted in the development VM. I logged the entire request header information in both the scenarios. From my side, I confirmed that all the requests are having the right endpoint so it is not our problem but something to do with Coveo network side. We asked Coveo support to involve their network team and we shared the entire log information. 

End Result

Coveo Support found the issue at their side. When the Sitecore proxied request reached Coveo and if it has X-Forwarded-Host header, then Coveo considered as invalid request and routed all traffic to deprecated endpoint. The request did not have X-Forwarded-Host header if we send the request inside the server (without Cloudflare or Azure Front Door) so those requests were marked as valid and routed to organization endpoint. Unfortunately we do not have any control to customize the request to Coveo using Sitecore proxy as it is part of the Coveo for Sitecore module. 

On Nov13th 2024, Coveo support released the fix at their side and we gave a couple of Visit IDs to test and they confirmed that it reached the organization endpoint. Also, Coveo extended their deprecation deadline to one last time to June 3rd 2025. 

The X-Forwarded-Host (XFH) header is a de-facto standard header for identifying the original host requested by the client in the Host HTTP request header. Host names and ports of reverse proxies (load balancers, CDNs) may differ from the origin server handling the request, in that case the X-Forwarded-Host header is useful to determine which Host was originally used. - Mozilla Developer Docs.

Since we have Cloudflare and Azure Front Door, the X-Forwarded-Host header is added to the request and it reached Sitecore server. Coveo for Sitecore module uses the headers from the original request, transforms only a set of headers with different value and copies the rest of the headers from original request to the proxied Coveo request. 

Once Coveo made changes at their side, this started to work for every Coveo clients. 

Our traffic data is also moving towards organization specific endpoint.

Happy Customers 😊😊😊

XM Apps API - Use Case for large organizations

In one of my previous blog article, I discussed about the newly introduced XM Apps API to manage XM Cloud Apps. What could be the use case for these APIs? 

I worked for a multinational confectionery, food, and snack food company client that had over 160 brands. They were using Sitecore XP, .NET-based applications, and other technologies for their brand sites. One of the pain points for the brand managers was the inability to quickly set up, build, and launch promotional sites. About 10 years ago, we proposed a solution that allowed brand managers to request new Sitecore, .NET, or other tech stack sites through a centralized portal. With the approval of IT and other stakeholders, the setup for a new Sitecore site would be ready for the brand manager and their development agency in just 20 minutes. Previously, this process had an SLA of 3-4 days due to various manual tasks. We automated everything, reducing the setup time to 20 minutes, enabling the development agency to start developing using existing reusable components and launch the site within an hour.

With XM Cloud Site management features along with the XM Apps API, large organization can make use of centralized internal portal to tie up with XM Apps API to setup, launch and destroy sites in a matter of seconds to few minutes. With automation, there is a huge impact to the speed with low/no manual errors. 

Explanation

1. Brand manager wants to launch a promotional site for an upcoming event. 
2. Brand manager requests a new site using Centralized Site Request portal. As part of the request, budget, project code to charge to manage XM Cloud and other relevant platforms to host a site, development agency details with all the developer information including email, name and other details, project sponsor information are added. 
3. IT and other stakeholders of the organization review the request and approve it. 
4. The centralized site request tool is based on Task-Oriented Architecture. Adds the list of tasks in the queue. Agent (start with a simple PowerShell scripts) takes each pending tasks and executes it in the order as configured. 
5. With XM Apps API and the availability of other APIs, these tasks can be executed in the matter of seconds to minutes. 
6. Development agency will be informed automatically and development can be started immediately. 

Can It Be Done?

Certainly. Even with Sitecore XP hosted in IaaS with no support of APIs, we were able to bring down the Site setup SLA from 3 days to 20 minutes with this kind of automation 10 years ago. With the availability of XM Cloud APIs and other APIs, large organization can be benefitted to automate and manage huge number of requests with no manual errors. 

Happy Customers!