Populating a Store-specific external search

Initial Setup

This section explains which steps need to be completed to set up the integration between the commercetools Platform, a message queuing service, and an external search service. The first four steps are specific to the service providers and therefore not explained here in detail.

1. Set up Store-specific search indices.
2. Set up a message queue for notifications about changes on Products.
3. Set up a message queue for notifications about changes on Product Selections.
4. Set up a message queue for notifications about changes on Stores.

Once this is complete, you need to set up three subscriptions to get notified upon changes on product data related to a Store.

Set up Subscriptions

The following subscriptions need to be set up on the commercetools Platform:

Subscription for Changes on Products

Create a Subscription to listen to any updates on Product with the following changes field:

{
  ...
  "changes": [
    {
      "resourceTypeId": "product"
    }
  ]
}

Subscription for Changes on Product Selections

By using Message Subscriptions on Product Selection Messages, you get notified whenever a Product was added or removed to/from a Product Selection.

Subscription for Changes on Stores

By using Message Subscriptions on Store Messages, you get notified whenever a Product Selection was added or removed to/from a Store or whenever a Product Selection is activated or deactivated for a Store.

When you are certain that the Stores' configuration does not change, you can skip this step since there is no need to listen to those changes.

Initial product data feed

You now have the search indices and message queues in place and you also set up the Subscriptions that make sure you will not miss any update on the product data from now on. What is left to do, is to fill the search indices with product data since the indices are still empty.

For each search index you have to find all Products available for the respective Store:

1. first fetch all active Product Selections of the Store.
2. for each Product Selection, iterate on all Products' References.
3. for each Product, fetch the projected product in the Store to put this data into the search index.

Keeping the search indices up to date

Feeding the search indices with initial product data is now complete, but over time it is likely that product data or the Product Selection is changed on the commercetools Platform and you need to reflect those changes in the search indices to be up to date in the store front also.

Reacting to changes on Products

Upon every change on a Product, the commercetools Platform sends a Message to the Product queue you set up before. You react upon such message in the following way:

Checking which Stores are impacted

In case you maintain a preconfigured list of Stores, you can avoid this step by going straight to Updating the search index. Otherwise you need to fetch all the Stores in which the updated Product is available by:

1. finding out which Product Selections the Product belongs to.
2. iterating over all returned Product Selections and checking which Stores are impacted.

In the second step you use following where query to filter for Stores that contain activated Product Selections with the changed Product:

productSelections(active=true and productSelection(id in :productSelectionIds))

and you pass productSelectionIds as input variables.

Updating the search index

Since you now found out to which Store(s) the updated Product belongs, you can now update the respective search indices with the changed product information.

For each impacted Store,

• if the Product was deleted, it can be removed from the search index of the corresponding Store.
• if the Product was updated, fetch the projected Product in the Store.
  • If the Product is available in the Store, put the returned data into the search index of the corresponding Store.
  • If a 404 Not Found error is returned, the Product is not available in this Store and you delete it from the search index of the corresponding Store.

As an optimization, we recommend comparing the version of the updated Product with the version stored in the search index, and to only update in case the version number from the received message is higher than the number in the search index.

Reacting to changes on Product Selections

In the section before you handled changes on the individual Products, but you also want to update our Store's search index whenever a new Product was added or removed from the Store's Product Selections. This case is not covered by changes on Products and you need to handle it by processing messages from the Product Selection queue you set up earlier like this:

Checking which Stores are impacted

In case you maintain a preconfigured list of Stores, you can avoid this step by going straight to Updating the search index. Otherwise you need to fetch all the Stores impacted by the change.

From the ProductSelectionProductAddedMessage or the ProductSelectionProductRemovedMessage you get the ID of Product Selection.

You can check which Stores are impacted by using the following where query to filter for Stores that contain the updated Product Selection:

productSelections(active=true and productSelection(id = :productSelectionId))

and you pass productSelectionId as input variables.

Updating the search index

If a Product was added to a Product Selection,

1. fetch the Product Projection in the Store and
2. put the returned data into the search index of the corresponding Store.

If a Product was removed from a Product Selection, delete it from the search index of the corresponding Store.

Reacting to changes on Stores

You also want to update our Store's search index whenever a Product Selection was added or removed to/from a Store or whenever a Product Selection was activated or deactivated in a Store. You handle this by processing messages from the Store queue you set up earlier like this:

Checking which Stores are impacted

You get the ID of the impacted Store from the Store Messages.

Updating the search index

In case a Store was created, feed all product data into the new search index as described in initial product data feed.

In case a Store was deleted, you can remove the corresponding search index.

In case the Product Selection configuration of a Store was changed, you have two options:

• You ignore the details from the StoreProductSelectionsChangedMessage and completely reindex all data for this Store as described in reindex all data from time to time.
• Alternatively, you are more specific with our updates and reindex only with those Products that are affected by the configuration change:
  1. For each Product Selection that was added, removed, activated or deactivated, you can iterate on all Products' References.
  2. For each Product, you fetch the projected product in the Store.
  • If the Product is not found you remove it from the search index.
  • Otherwise you put this data into the Store's search index.

Best practices

Detect unnecessary updates

We recommend you to store the version of the Products in the search indices, and to use this information to detect if some product data is already up to date and does not need any further updates.

Reindex all data from time to time

A search index can become out of date in case an update was missed, or due to changes on a Store's configuration. To keep such inconsistencies to a minimum, we recommend reindexing all product data in a search index, not only on demand, but also from time to time.

The following steps suggest a way how to do that without impacting your live traffic for the search index of one Store:

1. Create a second search index for this Store.
2. Let the Product updates now apply to both search indices: the live index and the new index.
3. Feed all product data into the new search index as described in initial product data feed.
4. Once the new search index is ready, move your live traffic over to it.
5. Delete the old live index that is not used anymore.