Migrating Persistent Indexes of Elasticsearch
This section describes how to migrate Elasticsearch indexes during the upgrade of UX1 Marketplace from version 9.x to 10.0. Before you proceed, make sure that you have performed necessary checks and the migration is necessary.
Pre-Migration Requirement
-
Before upgrading to UX1 Marketplace 10.0, you must set the skipAutoReindex: true flag in your config.yaml file.
ux1-marketplace: elastic: ... skipAutoReindex: trueSetting this flag prevents the automatic deletion of the existing Elasticsearch pod, allowing it to run in parallel so you can manually migrate data without downtime.
Important: If this flag is not set before the upgrade, all data in the Elasticsearch pod will be lost.
Step 1: Verify cluster state after upgrade
After UX1 Marketplace is upgraded to version 10.0, from the environment with access to the Kubernetes cluster, where CloudBlue Commerce is deployed, run the following command:
kubectl get pods -n <namespace> | grep elastic
The output must be similar to the following:
ux1-marketplace-elastic-xxxxx 1/1 Running ux1-marketplace-elastic-v8-xxxxx 1/1 Running
Step 2: Perform re-indexing
Run the following command to re-index all indices from Elasticsearch 7 to Elasticsearch 8:
kubectl exec -n <namespace> deploy/ux1-marketplace-elastic-v8 -- sh -c '
ES7="http://ux1-marketplace-elastic:9200"
ES8="http://localhost:9200"
ANALYSIS="{
\"tokenizer\": {
\"ngram_tokenizer\": {
\"type\": \"ngram\",
\"min_gram\": \"3\",
\"max_gram\": \"3\",
\"token_chars\": [
\"letter\",
\"whitespace\",
\"punctuation\",
\"digit\"
]
}
},
\"analyzer\": {
\"trigrams\": {
\"type\": \"custom\",
\"tokenizer\": \"ngram_tokenizer\",
\"filter\": [\"lowercase\"]
},
\"lowerCaseWords\": {
\"type\": \"custom\",
\"tokenizer\": \"whitespace\",
\"filter\": [\"lowercase\"]
}
}
}"
for idx in syncsettings product landingpage plan mpsettings events; do
echo "=== $idx ==="
mappings=$(curl -s "$ES7/$idx/_mapping" \
| sed "s/{\"$idx\":{\"mappings\":\(.*\)}}/\1/")
if curl -s -I -XHEAD "$ES8/$idx" 2>/dev/null | grep -q "200 OK"; then
curl -s -XDELETE "$ES8/$idx"
echo "deleted"
fi
case $idx in
product|landingpage|plan)
body="{
\"settings\": {
\"index\": {
\"number_of_shards\": \"1\",
\"analysis\": $ANALYSIS
}
},
\"mappings\": $mappings
}"
;;
*)
body="{
\"settings\": {
\"index\": {
\"number_of_shards\": \"1\"
}
},
\"mappings\": $mappings
}"
;;
esac
echo "$body" | curl -s -XPUT "$ES8/$idx" \
-H "Content-Type: application/json" \
-d @-
curl -s -XPUT "$ES8/$idx/_settings" \
-H "Content-Type: application/json" \
-d "{\"index.max_result_window\":500000}"
echo
reindex="{
\"source\": {
\"remote\": {
\"host\": \"$ES7\"
},
\"index\": \"$idx\"
},
\"dest\": {
\"index\": \"$idx\"
}
}"
echo "$reindex" | curl -s -XPOST \
"$ES8/_reindex?wait_for_completion=true" \
-H "Content-Type: application/json" \
-d @-
echo
done
'
The estimated execution time is approximately 1 hour 40 minutes on an installation with 3,600,000 plans and 1,000,000 products.
Step 3: Verify data
Compare Elasticsearch version 7 and 8 document count. To do that, run the following command:
kubectl exec -n <namespace> -ti deploy/ux1-marketplace-elastic-v8 -- sh -c ' ES7="http://ux1-marketplace-elastic:9200" ES8="http://localhost:9200" printf "%-40s %-12s %-12s " "index" "es7_docs" "es8_docs" for idx in $(curl -s "$ES7/_cat/indices?h=index" | grep -vE "^\."); do c7=$(curl -s "$ES7/$idx/_count" | sed -n "s/.*\"count\":\([0-9]*\).*/\1/p") c8=$(curl -s "$ES8/$idx/_count" | sed -n "s/.*\"count\":\([0-9]*\).*/\1/p") printf "%-40s %-12s %-12s " "$idx" "$c7" "$c8" done'
Output of this command will be similar to the following:
index es7_docs es8_docs syncsettings 3 3 product 192 192 landingpage 0 0 plan 667 667 events 8 8 mpsettings 10 10
If the migration was successful, the number of documents will match for both Elasticsearch versions. A slight number deviation is acceptable, as new documents may be added or removed during the re-indexing process.
Step 4: Switch to Elasticsearch version 8
After data verification, switch to Elasticsearch version 8.
Important: This step will cause UX1 Marketplace downtime for the time the pod will restart. A usual restart time is expected for the pod, which is approximately 1 minute or less.
To switch Elasticsearch to version 8, run the following commands:
kubectl set env -n <namespace> deployment/ux1-marketplace-connector \ ELASTIC_SEARCH_URIS="http://ux1-marketplace-elastic-v8:9200" \ ELASTIC_COMPATIBILITY_MODE="true" kubectl rollout restart -n <namespace> deployment/ux1-marketplace-connector kubectl rollout status -n <namespace> deployment/ux1-marketplace-connector
Step 5: Verify search functionality
After the switch, you need to verify that search is operating properly. To do that, complete the following steps:
-
In UX1, go to Marketplace.
-
Check that the filters work properly: vendor, categories, billing period and other.
-
Ensure the products and plans are displayed correctly.
-
Check categories and service plans in Sales channels.
If all the data is displayed without any errors, the switch was successful.
Note: Elasticsearch 7 pod must remain running to enable rollback possibility.
Rollback procedure
If errors were found during verification step, you need to switch Elasticsearch back to version 7.
To switch Elasticsearch back to version 7, use the following command:
kubectl set env -n <namespace> deployment/ux1-marketplace-connector \ ELASTIC_SEARCH_URIS="http://ux1-marketplace-elastic:9200" \ ELASTIC_COMPATIBILITY_MODE="false" kubectl rollout restart -n <namespace> deployment/ux1-marketplace-connector kubectl rollout status -n <namespace> deployment/ux1-marketplace-connector
Post-migration notes
-
Make sure that the pod with Elasticsearch version 8 is operates without issues for several days.
-
Elasticsearch 7 support will be removed in the next UX1 Marketplace release. The duplicated Elasticsearch 7 index will be removed automatically with the next UX1 Marketplace version or could be removed manually after a reasonable time of smooth Elasticsearch 8 operation.
-
Data added after re-indexing will be synchronized automatically using the Sync and Plan Indexer tasks.
-
Remove the skipAutoReindex flag from your config.yaml prior to the next upgrade.