Sitecore xConnect : IncompatibleSyncTokensException

Background

I have been working on an ambitious project at work the past few months.  It involves an integration using Sitecore Data Exchange Framework (DEF).  When a user follows an email campaign link sent by my client's email marketing application, we want to track that user's visit in Sitecore xConnect.  An identifier is added to the query string of the website link.  On arriving on the site, the identifier is processed in a RequestBeginProcessor to identify the anonymous xConnect contact with a Contact Identifier.

The next step is to use Data Exchange Framework to synchronize data from the client's back-end CRM system and xConnect.  xConnect contacts with the custom contact identifier, which is the identifier from the back-end CRM system, are synchronized to xConnect over the back-end systems REST api.  Basic information (First Name, Last Name, email, as well as detailed history of that contact) are imported into the xConnect contact.

In Sitecore, we set up Profile Cards to identify content.  Once the website visitor has finished the web-visit, Sitecore tags the website visitor into one of several profiles.

A second DEF process synchronizes the Sitecore profile information back to the client's back end CRM system.  Their CRM system now contains website visit data, along with their visitor behavior.

The demo with the client went very well, until I tried to demo the new Identified contact in the Sitecore Experience Profile application.  The identified contacts would not appear.  I have been able to get this working on other environments.  It works on my development machine.  We rescheduled the demo and demonstrated on the client's UAT environment.  And I have just recently pushed the product into the production environment.  But for over a month the QA environment has remained broken.  With the product delivered into production, I finally had some time to revisit QA and figure out the problem.

The Problem

The Sitecore Experience Profile application reads contact data from the solr index Sitecore_Xdb.  The xConnect application has an application responsible for building and maintaining this index.  It is located at [xConnect installation directory]\App_Data\jobs\continuous\IndexWorker.  In the logs directory, the only hint I could find at the problem is this:

2019-10-07 00:12:59.570 -04:00 [Error] Failed indexing next set of changes. There will be an attempt to recover from the failure.
Sitecore.Xdb.Collection.Failures.IncompatibleSyncTokensException: Tokens are incompatible, they have different set of shards.
   at Sitecore.Xdb.Collection.Data.SqlServer.Managers.ChangeTracking.SyncToken.IsUpToDate(ISyncToken syncToken)


The best best Google hit I could find on this problem describes the problem and several steps for analysis very well:

https://sitecore.stackexchange.com/questions/12453/xconnect-indexworker-error-tokens-are-incompatible-they-have-different-set-of#answer-12454

The author's trick to use the solr admin dashboard Query utility and to query the Sitecore_Xdb index for id:xdb-index-token, then to take the results of that and pipe into the ConvertFrom-XdbIndexToken.ps1 function provided, eventually led me to finding Sitecore_Xdb and Sitecore_Xdb_Rebuild had different values stored.

Unfortunately, the author's suggestion to use XConnectSearchIndexer.exe -rr did not work for me.  It's become a command I have used many, many, MANY times over the past couple of months.  It seems like xConnect is doing a pre-check on Sitecore_Xdb and Sitecore_Xdb_Rebuild for the id:xdb-index-token, and if they do not match, throws an exception.  I wish it would not do this, and simply just delete the Sitecore_Xdb_Rebuild and build it from scratch from Sitecore_Xdb without this pre-check.

The Solution

This morning I finally had the bright idea to just delete both the Sitecore_Xdb and Sitecore_Xdb_Rebuild.  I followed the below instructions to recreate both the Sitecore_Xdb and Sitecore_Xdb_Rebuild indexes.

https://doc.sitecore.com/developers/91/platform-administration-and-architecture/en/walkthrough--using-solrcloud-for-xconnect-search.html

The index has been succesfully rebuilt.  This confirms my analysis that the Sitecore_Xdb_Rebuild was incompatible with the  Sitecore_Xdb index.