Deploying Rating Engine

Deployment Overview

Rating Engine is deployed to a Kubernetes cluster and can be deployed to the cloud as well as on premises.

The Rating Engine APS application is deployed as a Docker container and includes the following components:

• Rating Engine
• Subscription History Management
• Rating Adapter
• Product Gateway

Installation Prerequisites

Before installing Rating Engine, ensure that:

• CloudBlue Commerce 20.5 is installed.

• A service node with a Kubernetes cluster is configured and registered in CloudBlue Commerce
• Rating Engine has access to the BSS Database. For configuration instructions, please refer to Allowing a Microservice to Access the BSS Database PostgreSQL Server.

Infrastructure Requirements

The minimum Kubernetes resources required for the microservice deployment, including on-premise Kubernetes cluster deployment, are:

Requests:
CPU:	1000m
Memory:	1280Mi

For Rating Engine databases:

• Hardware requirements: please see BALINBE
• Software requirements: PostgreSQL 11.5

Firewall Requirements

The following ports must be open on the micro-service node and BSS database node:

From	To	Protocol	Port	Purpose
CloudBlue Commerce Management Node	Micro-service node	TCP	8081	For access to the Rating Engine service backend (HTTPS is used for communication)
Micro-services node	BSS database node	TCP	5432	For connection to the PostgreSQL database

Installing or Upgrading Rating Engine

During Rating Engine installation or upgrade, zero downtime of CloudBlue Commerce is guaranteed. However, during upgrade, the related event processing is suspended; a special messaging service collects unprocessed events and, after the upgrade is complete, sends them to be processed by the upgraded Rating Engine service.

Before installing Rating Engine:

1. In the Provider Control Panel (Operations), go to System > Settings > Public API > Allowed Networks.
2. Add the Backnet IP address of the micro-service node.

To install Rating Engine:

1. For the Rating Engine database host, deploy a PostgreSQL instance (please see Infrastructure Requirements).

   Note: You may use an existing PostgreSQL instance. However, it must not be the same as the OSS or BSS database instance.

2. On the PostgreSQL instance, create four databases for:

   1. the Rating Engine APS application

   2. the Rating Engine service module

   3. the Subscription History Management service module

   4. the Rating Engine Adapter service module

3. Go to the CloudBlue Commerce Management Node under the root user using SSH.
4. Create a configuration file with the name re_values.yaml and configuration parameters below.

5. To install (or upgrade) the Rating Engine APS application, prepare the re_values.yaml configuration file and the deploy.sh deployment script, and then run the script.

   Note: By default, the latest version is installed.

6. Make sure that the application is successfully installed:
   1. Go to CloudBlue Commerce Provider Control Panel (Operations) > Services > Applications.
   2. Make sure that the Rating Engine APS application is in the list. If it is not in the list, please see the troubleshooting section.

Rating Engine Deployment Configuration Parameters (re_values.yaml)

The parameters in the table are included in the re_values.yaml configuration file, which is used during the Rating Engine deployment.

Name	Description	Default Value	Example
oauthkey	A randomly generated key. Can be generated using the uuidgen command. Together with oauthsecret, oauthkey will be used by CloudBlue Commerce to generate a public key for Rating Engine to access the APS bus.		d8d4656a-7b87-11e7-bb31-be2e44b06b36
oauthsecret	A randomly generated secret. Can be generated using the uuidgen command. Together with oauthkey, oauthsecret will be used by CloudBlue Commerce to generate a public key for Rating Engine to access the APS bus.		9e7050c2-0773-436b-bb33-b482cd225b8f
javaheapsize	A parameter for JVM options (including Java heap size)	-Xms512m -Xmx1024m	-Xms512m -Xmx2g
dockerrepo	The hostname of the Docker registry where the Rating Engine APS application can be obtained	skeleton.jfrog.io	skeleton.jfrog.io
dshost	The Rating Engine database PostgreSQL instance hostname or IP address		rating.database.com
dslogin	The user name to access the PostgreSQL database	ratingengine	ratinguser
dspassword	Base64-encoded password of the user defined by dslogin to access PostgreSQL. To encode a password, use this command: echo -n somepassword | base64		VUNqc2RXM0cxdGNHMGE=
dsdbname	The name of the Rating Engine APS application database	ratingapp	ratingapp
adapterdsdbname	The name of the Rating Engine Adapter service module database	adapter	adapter
redsdbname	The name of the Rating Engine service module database	ratingengine	ratingengine
smdsdbname	The name of the Subscription History Management service module database	subscription	subscription
bssdshost	The BSS database hostname or IP address		bss.database.com
bssdsdbname	The BSS database name	pba	bssdatabase
bssdslogin	The user login name to access the BSS database	pba	bssdbuserlogin
bssdspassword	Base64-encoded password of the user defined by bssdslogin to access the BSS database. To encode a password, use this command: echo -n somepassword | base64		QkxMZDZvOWljc251MFI=
postgresbatchsize	This parameter defines how many subscriptions will be processed in one batch. Important: Rating Engine Migrator tasks in OSS may fail with timeout because of postgresbatchsize. To avoid it, decrease the value of this parameter.	5000
migratorplantaskgroupssize	This parameter defines how many migration threads (and tasks) will be run at the same time.	4
planspertask	This parameter defines how many service plans will be processed within one Rating Engine Migrator tasks.	10
migratorprecheckappnames	A comma-separated list of applications that must be prechecked before shifting subscriptions to Rating Engine. Values: [] (default): All installed applications will be prechecked. <comma_separated_list>: a list of applications to be prechecked.	[]
mindbpoolsize	The minimum number of connections to each DB in the connections pool		1
maxdbpoolsize	The maximum number of connections to each DB in the connections pool		64
durationprecision	The rounding precision for the duration of prorated billing periods (the number of decimal places after the decimal point) Warning: durationprecision must have the same value as the BSS parameter DURATION_PRECISION.	2	2
migrator.skipBNOrders	This parameter defines whether the Shift to Rating Engine tool (Rating Engine Migrator) will skip BN orders when shifting subscriptions to Rating Engine. BN orders (also known as Not Invoiced Purchase orders) are an internal order type for backward compatibility with legacy services. Values: true: the Shift to Rating Engine tool (Rating Engine Migrator) will be skipping BN orders when shifting subscriptions to Rating Engine. The downtime will be reduced. For most installations, true should be used. false (default): the Shift to Rating Engine tool (Rating Engine Migrator) will not be skipping BN orders when shifting subscriptions to Rating Engine. For special cases, false should be used.	false	true
migrator.notification	This parameter defines whether notifications from the Shift to Rating Engine tool (Rating Engine Migrator) are displayed in the UX1 control panel. Values: true (default): notifications from the Shift to Rating Engine tool are displayed in the UX1 for Providers UI. false: notifications from the Shift to Rating Engine tool are not displayed in the UX1 for Providers UI. You can use it to reduce distraction of business users by technical messages. However, technical staff will not see them either.	true	true
timezone	The time zone that will be used by Rating Engine. Warning: Set timezone to the same value as the Billing Application Node time zone. The value must the same as the time zone file name in the /usr/share/zoneinfo directory. For example, Europe/Moscow, Israel, EST.	UTC

Sample re_values.yaml

# Generate by uuidgen on deployment
oauthkey: 772e7800-48fc-4738-b29d-adc5443a4e29
# Generate by uuidgen on deployment
oauthsecret: c0a2b5b5-8de8-4015-ba5c-6462a08c1c0f
javaheapsize: -Xms512m -Xmx2048m
dockerrepo: odindevops-a8n-docker.jfrog.io
dshost: 10.0.0.1
dslogin: ratingengine
dspassword: XXXXXXXXXXXX
dsdbname: ratingapp
redsdbname: ratingengine
smdsdbname: subscription
adapterdsdbname: adapter
bssdshost: 10.0.0.2
bssdsdbname: pba
bssdslogin: pba
bssdspassword: XXXXXXXXXXXXX
durationprecision: 2
# set to actual timezone of the deployment
timezone: UTC

Passwords in the file must be base64-encoded. To correctly encode a password, use this command:

echo -n somepassword | base64

deploy.sh

This script is used to install or update the Rating Engine microservice in a Kubernetes cluster.

To deploy a particular Rating Engine version to a Kubernetes cluster, pass the version as the first argument to the following installation script:

#!/usr/bin/env bash

#REPO='skeleton.repo.int.zone'  # to local
REPO='odindevops-a8n-docker.jfrog.io'  # to prod

SERVICE_NAME='ratingengine'
POD_NAME='a8n/ratingengine-backend'
VALUES_FILE='re_values.yaml'
VERSION="$1"

helm repo update

if helm version --short | grep -P 'v2'; then
    helm_install="helm install ${POD_NAME} --name ${SERVICE_NAME}"
else
    helm_install="helm install ${SERVICE_NAME} ${POD_NAME}"
fi

if [[ $(helm list | grep "\b${SERVICE_NAME}\b") ]]; then
    if [ "${VERSION}" ]; then
        helm upgrade -f ${VALUES_FILE} --set dockerrepo=${REPO} --version ${VERSION} ${SERVICE_NAME} ${POD_NAME}
    else
        helm upgrade -f ${VALUES_FILE} --set dockerrepo=${REPO} ${SERVICE_NAME} ${POD_NAME}
    fi
else
    if [ "${VERSION}" ]; then
        $helm_install --set dockerrepo=${REPO} --values ${VALUES_FILE} --version ${VERSION}
    else
        $helm_install --set dockerrepo=${REPO} --values ${VALUES_FILE}
    fi
fi

Increasing Rating Engine Performance Through Automatic Scaling in Kubernetes

You can increase the performance of Rating Engine using the Kubernetes Horizontal POD Autoscaler. To do so, enable the autoscaler using the following command:

kubectl autoscale deployment ratingengine --cpu-percent=80 --min=1 --max=5

Where ratingengine is the value of the ${SERVICE_NAME} variable from deploy.sh.

The command will create the following autoscale rule for the service:

If the service node is loaded over 80% then run another POD and distribute requests between PODs; the minimum number of PODs running at the same time is one; the maximum number of PODs running at the same time is five.

To remove the autoscale rule, use the following command:

kubectl delete hpa ratingengine

Troubleshooting Rating Engine Deployment

This section describes the known issues of the Rating Engine deployment.

Note: See also Kubernetes and Helm Commands for Troubleshooting.

Problem 1: The deploy.sh script fails during the installation of the Rating Engine service.

Make sure that your Helm host has access to the appropriate Docker registry.

Problem 2: The Rating Engine service is installed as a POD and is running but theRating Engine APS application is not installed.

1. Go to the CloudBlue Commerce management node using SSH.

2. Make sure that the Rating Engine service name defined by the ${SERVICE_NAME} variable is resolved by the management node by using the following command:

   $ nslookup ratingengine

3. If the service name is not resolved:
   1. On the MN node, open the /etc/resolv.conf file. Make sure that the kube-dns service IP address is properly configured.
   2. Complete the installation process.
4. If the service name is resolved, there can be other error sources:
   • Errors caused by the Rating Engine service: inspect the RE logs; if necessary, contact the CloudBlue Commerce support.
   • Other deployment errors: contact the CloudBlue Commerce support.

   Resolve the errors and go to the next step.

5. Reinstall the Rating Engine POD:

   1. Uninstall the Rating Engine POD using the following command:

      helm del --purge ratingengine

   2. Install the Rating Engine POD using deploy.sh.

Problem 3: The Rating Engine APS application is installed, but BSS does not receive CoS charges for a service plan.

1. Go to the CloudBlue Commerce management node using SSH.
2. Make sure that the Rating Engine POD is running.
3. Make sure that the service plan is configured to be rated using Rating Engine.
4. Make sure that the ratingaplications value is correct.
5. Inspect the Rating Engine POD execution logs.

Kubernetes and Helm Commands for Troubleshooting

• To check that a POD is running:

  kubectl get pods | grep ratingengine | awk '{ print $3; }'

  Where ratingengine is the value of the ${SERVICE_NAME} variable from deploy.sh

• To obtain the kube-dns service IP address:

  kubectl get service --namespace kube-system | grep kube-dns | awk '{ print $3; };'

• To obtain the logs from an executed POD:

  kubectl logs -f `kubectl get pods | grep ratingengine | awk '{ print $1; }' | head -n 1`

  Where ratingengine is the value of the ${SERVICE_NAME} variable from deploy.sh

• To uninstall a POD:

  helm delete ratingengine --purge

  Where ratingengine is the value of the ${SERVICE_NAME} variable from deploy.sh