Deploying Reporting and Data Export

High-Level Architecture

The RDE is a microservice with its own release cycle. It is not included in the CloudBlue Commerce distribution package by default and must be installed on the platform as an APS application.

The RDE microservice consists of two components:

RDE UI, hosted on the management node
RDE backend, hosted in a separate container on the service node with installed Kubernetes

Installation Prerequisites

Before importing the RDE APS package, ensure that:

Odin Automation version 8.0, 8.2, or 8.3, or CloudBlue Commerce version 20.5 or later is installed.
A service node with a Kubernetes cluster is configured and registered in the CloudBlue Commerce platform.
Dynamic provisioning of Persistent Volumes is configured.

Note: Persistent Volumes are only required to upgrade RDE from 1.x and 2.x to 3.5.8 or later versions.
The RDE service database is prepared.
The RDE has read access to BSS database. For configuration instructions, refer to Allowing a Microservice to Access the BSS Database PostgreSQL Server.

Note: As RDE needs only read access, for better load balancing, using a BSS DB replica with RDE is recommended.
The RDE has read access to OSS database. To configure the access, use the instructions in Allowing a Microservice to Access the BSS Database PostgreSQL Server, where the name OSS replaces BSS for this configuration.

Note: As RDE needs only read access, for better load balancing, using a OSS DB replica with RDE is recommended.

The OSS database secret exists. Use this command to check it.

# kubectl get secret oss-db

If it does not exist, create a secret using this command:

# kubectl create secret generic oss-db --from-literal=dbname={{ oss_db_name }} --from-literal=host={{ oss_db_host }} --from-literal=password={{ oss_db_password }} --from-literal=username={{ oss_db_username }}

The RDE database secret exists. If it does not, create a secret using this command:

# kubectl create secret generic rateddataexport-db --from-literal=dbname={{ rde.dsdbname }} --from-literal=host={{ rde.dshost }}-postgresql --from-literal=password={{ rde.dspassword }} --from-literal=username={{ rde.dslogin }}

The BSS database secret exists. If it does not, create a secret using this command:

# kubectl create secret generic bss-db --from-literal=dbname={{ bss_db_name }} --from-literal=host={{ bss_db_host }}--from-literal=password={{  bss_db_password }} --from-literal=username={{ bss_db_username}}

Note: The creation of a BSS database secret is only required to upgrade RDE from 1.x and 2.x to 3.x.

Infrastructure Requirements

The minimum Kubernetes resources required for the microservice deployment, including on-premise Kubernetes cluster deployment, are:

Resources for PVC:
CPU:	2
Memory:	1Gi
Requests for PVC:
CPU:	100m
Memory:	512Mi
Resources for service:
CPU:	500m
Memory:	1280Mi

Firewall Requirements

The following ports must be open on the micro-service node and the BSS database replica node:

From	To	Protocol	Port	Purpose
CloudBlue Commerce Management Node	Micro-service node	TCP	8081	For access to the RDE service back end (HTTPS is used for communication)
Micro-service node	BSS database replica node	TCP	5432	For connection to the PostgreSQL database
Micro-service node	OSS database replica node	TCP	5432	For connection to the PostgreSQL database

How to Install RDE

Before installing the RDE application, select from one option:

Set up MS Azure cloud storage to store generated reports, or
Set up SFTP storage to store generated reports.

You also need to restart the following services on the management node before RDE installation:

pau
pa-agent

Deployment with MS Azure storage

To install the RDE application, run the following command:

For a cloud-based infrastructure, using internet access:

# helm install  a8n/rateddataexport --name rateddataexport --set oauthkey=`uuidgen` --set oauthsecret=`uuidgen` --set dslogin=<BSS_DB_login> --set dspassword=`echo -n <BSS_DB_password> | base64` --set dsdbname=<BSS_DB_name> --set dshost=<BSS_DB_IP_address>  --set dstimezone=<MN_timezone> --set azconnstr=<Azure_connection_string> --version <version> --set persistence.storageClass=azure --set dockerrepo=odindevops-a8n-docker.jfrog.io

where

<BSS_DB_login>: the user login to the BSS database.
<BSS_DB_password>: the password to the BSS database.
<BSS_DB_IP_address>: the BSS database IP address.
<MN_timezone>: the time zone the MN is set to. Use the region-based zone ID format area/city. For example, Europe/Moscow or America/New_York.
<Azure_connection_string>: the MS Azure Cloud Storage connection string.

Or using proxy:

# helm install  a8n/rateddataexport --set proxyhost=<Proxy_IP_address> --set proxyport="8008" --name rateddataexport --set oauthkey=`uuidgen` --set oauthsecret=`uuidgen` --set dslogin=<BSS_DB_login> --set dspassword=`echo -n <BSS_DB_password> | base64` --set dsdbname=<BSS_DB_name> --set dshost=<BSS_DB_IP_address>  --set dstimezone=<MN_timezone> --set azconnstr=<Azure_connection_string> --version <version> --set persistence.storageClass=azure --set dockerrepo=odindevops-a8n-docker.jfrog.io

where:

<Proxy_IP_address>: the proxy server IP address.

For an on-premises infrastructure:

# helm install  a8n/rateddataexport --name rateddataexport --set oauthkey=`uuidgen` --set oauthsecret=`uuidgen` --set dslogin=<BSS_DB_login> --set dspassword=`echo -n <BSS_DB_password> | base64` --set dsdbname=<BSS_DB_name> --set dshost=<BSS_DB_IP_address>  --set dstimezone=<MN_timezone> --set azconnstr=<Azure_connection_string> --version <version> --set dockerrepo=odindevops-a8n-docker.jfrog.io --set persistence.storageClass=nfs-share

After installing the RDE application, the following scenarios will be available for resellers:

Generating a one-time report with aggregated billing data
Configuring generation of periodic reports with aggregated billing data

Additional Configuration Parameters

Parameter	Default value	Description
`result_set_fetch_size`	1000	Note: This parameter is available only in RDE version 3.12 or later. The number of rows that can be retrieved at once from the database. If an RDE request to the database returns 50,000 rows, they will be processed in batches, each consisting of 1000 rows.
`sftp_download_block_size`	20971520 bytes (20MB)	Note: This parameter is available only in RDE version 3.12 or later. The maximum size of a block that can be downloaded through SFTP. If the report file exceeds the specified number, it will be downloaded in several batches. The downloaded file will include all the batches.
`azure_upload_block_size`	5242880 bytes (5MB)	Note: This parameter is available only in RDE version 3.12 or later. The maximum size of a block that can be uploaded to Azure. If, for example, the report size is 100MB, and the specified value is 5MB, the report will be uploaded in 20 chunks.
`xlsx_max_row_size`	250,000 rows	Note: This parameter is available only in RDE version 3.12 or later. The maximum number of rows that a report file can include. If, for example, the report file includes 500,000 rows and the specified value is 250,000, then the report will be split to two files and zipped.
`xlsx_force_zip_files`	false	Note: This parameter is available only in RDE version 3.12 or later. The parameter that indicates whether to zip report files or not. If set to true, the report files will always be zipped. Otherwise, only reports split to several files will be zipped.
`persistence.enabled`	true	Note: This parameter was added in RDE version 3.12.28. This parameter enables persistence for the component. If RDE fails to upgrade to the latest version with an error that relates to NFS share, set this parameter to false in the config.yaml file.

How to Upgrade RDE to the Latest Version

Important:
- Before upgrading RDE, ensure that the Installation Prerequisites are met. To upgrade RDE from 1.x and 2.x to 3.x, there are additional requirements.
- If Reporting and Data Export (RDE) fails to upgrade to the latest version with an error related to NFS share, run the upgrade setting the persistence.enabled parameter to false.

The upgrade procedure is different depending on your currently installed RDE version.

Your current RDE version is 1.3-117 or later

For a cloud-based infrastructure, to upgrade the existing application with MS Azure cloud storage, run this command:

# helm upgrade rateddataexport a8n/rateddataexport --version <version> --set jdbcconnectionparams='?sslmode=require&ApplicationName=rateddataexport' --set devmode.enabled=false --set persistence.storageClass==azure --set dockerrepo=odindevops-a8n-docker.jfrog.io

For an on-premises infrastructure, to upgrade the existing application with MS Azure cloud storage, run this command:

# helm upgrade rateddataexport a8n/rateddataexport --version <version> --set jdbcconnectionparams='?sslmode=require&ApplicationName=rateddataexport' --set devmode.enabled=false --set dockerrepo=odindevops-a8n-docker.jfrog.io --set persistence.storageClass=nfs-share

To upgrade the existing application to store new reports in SFTP storage instead of MS Azure, run this command:

helm upgrade rateddataexport a8n/rateddataexport --version <version> --set sftp_host=<SFTP_HOST> --set sftp_port=<SFTP_PORT> --set sftp_directory=<SFTP_DIRECTORY> --set sftp_login=<SFTP_LOGIN> --set sftp_password=<SFTP_PASSWORD> --set azconnstr=' ' --reuse-values

For an on-premises infrastructure, to upgrade the existing application with SFTP storage, run this command:

# helm upgrade rateddataexport a8n/rateddataexport --version <version>--set jdbcconnectionparams='?sslmode=require&ApplicationName=rateddataexport' --set devmode.enabled=false --set dockerrepo=odindevops-a8n-docker.jfrog.io --set persistence.storageClass=nfs-share --set sftp_directory=<SFTP_DIRECTORY> --reuse-values

Your current RDE version is earlier than 1.3-117

To upgrade the existing application, complete the following steps:

On the CloudBlue Commerce management node, run the following command:
```
# kubectl get nodes -o yaml | grep address:
```
For each IP address in the response, run the following command:
```
# ssh <user>@<IP_address> sudo mkdir -p -m 777 /opt/rde_sql
```
where
- <user>: root if local Kubernetes is used, or autodeploy if AKS is used.
- <IP_address>: the IP address from the output above.

Update the RDE endpoint in the OSS DB:

begin;
update aps_application_instances set uri='https://rateddataexport:8081/rest' where uri='https://rateddataexport-com-odin-billing:8081/rest';
update aps_application set endpoint_uri='https://rateddataexport:8081/rest' where endpoint_uri='https://rateddataexport-com-odin-billing:8081/rest';
delete from aps_container where container_id='rateddataexport-backend.com.odin.billing';
commit;

Reconfigure the RDE:
1. Write the existing settings for the RDE back end to the rateddataexport.yaml file.
```
# helm get values rateddataexport-backend.com.odin.billing -a > rateddataexport.yaml
```
2. Decode base64 values (oauthkey, oauthsecret, dslogin) in the rateddataexport.yaml using the following commands:
```
# grep oauthkey rateddataexport.yaml | cut -f2 -d' ' | base64 -d
```
```
# grep oauthsecret rateddataexport.yaml | cut -f2 -d' ' | base64 -d
```
```
# grep dslogin rateddataexport.yaml | cut -f2 -d' ' | base64 -d
```
3. Replace the oauthkey, oauthsecret and dslogin values in the rateddataexport.yaml file with the respective decoded values from the previous step.
4. Remove the following fields in the rateddataexport.yaml:
  - dsport
  - maxdbpoolsize
  - mindbpoolsize
Remove the previous version of RDE by running the following command on the MN:
```
# helm delete rateddataexport-backend.com.odin.billing --purge
```
Install the new version of RDE (the necessary settings will be taken from the .yaml file):
```
# helm install a8n/rateddataexport -n rateddataexport -f rateddataexport.yaml --set azconnstr=<AZURE_CONNECTION_STRING>
```
where
- <Azure_connection_string>: the MS Azure Cloud Storage connection string.