Monitoring and Troubleshooting

Overview: Monitoring and Troubleshooting the Vendor-rated Data Manager Data Flow

You can monitor and troubleshoot the Vendor-rated Data Manager data flow in the following way:

Stage 1: Inflow

Usage data is collected and imported to Vendor-rated Data Manager for processing in one of these ways:

  • Automatically by VRD Collector:
    1. The Collect rated data task collects usage data from APS applications using API.
    2. The Transfer rated data to BSS Rating task enriches the collected data and submits it to Vendor-rated Data Manager.

  • Automatically from CloudBlue Connect using API.

  • Manually from the CloudBlue Commerce UI.

For monitoring and troubleshooting:

  • Usage data delivered by VRD Collector:

    Note: Usage report processing results are available at the Provider Control Panel > Billing > System > Logs > the Vendor Rated Data tab. To troubleshoot errors related to the processing results, please go to Stage 2: Processing by Vendor-rated Data Manager.

    1. In the Provider Control Panel, go to Operations > Operations > Tasks > Periodic. Filter and inspect the Collect rated data and Transfer rated data to BSS Rating tasks. By default, each task is launched every thirty minutes.
    2. Make sure that each task is scheduled (the task status is Rescheduled).
    3. Click the necessary task and inspect the task profile and logs. Troubleshoot as necessary. For further troubleshooting, go to Stage 2: Processing by Vendor-rated Data Manager.
    4. For errors in the imported data, contact the APS application owner or the vendor to correct the data on the APS application or vendor side:
      • For data that has been rejected by Vendor-rated Data Manager (and has not reached the processing by Billing), corrections must be made with subsequent resubmission to VRD Collector.
      • For errors in the successfully processed data, corrections must be made in the form of submitting the corresponding positive or negative difference between the wrong data and the correct data.

  • Usage reports delivered from CloudBlue Connect:

    Note: Usage report processing results are available at the Provider Control Panel > Billing > System > Logs > the Vendor Rated Data tab. To troubleshoot errors related to the processing results, please go to Stage 2: Processing by Vendor-rated Data Manager.

    1. In the Provider Control Panel, go to Operations > Operations > Tasks > Periodic. Filter and inspect the CloudBlue Connect Extension <uuid> Connect Chunk Usage Files retrieval task. By default, the task is launched every 24 hours.
    2. Make sure that each task is scheduled (the task status is Rescheduled).
    3. Click the necessary task and inspect the task profile and logs. Troubleshoot as necessary. For further troubleshooting, go to Stage 2: Processing by Vendor-rated Data Manager. If there is an unidentified error, contact CloudBlue Connect Support.
    4. In the Provider Control Panel, go to Services > Usage Reports. The successfully imported usage reports are processed automatically as batches (one usage report is one batch). Reports that fail data validation (the Status is Validation Failed) are rejected as a whole and are not processed by Vendor-rated Data Manager.
    5. For reports that failed data validation, in the Error Report column, click Get Error Report. For detailed information about errors and troubleshooting actions, please see Usage Report Import Troubleshooting.
    6. For data that has been rejected by Vendor-rated Data Manager (and has not reached the processing by Billing), contact the vendor to make the necessary corrections to the usage report on the CloudBlue Connect side or the vendor system side and resubmit the corrected report for processing using API.

  • Usage reports uploaded from the CloudBlue Commerce UI:

    1. In the Provider Control Panel, go to Services > Usage Reports. The successfully imported usage reports are processed automatically as batches (one usage report is one batch). Reports that fail data validation (the Status is Validation Failed) are rejected as a whole and are not processed by Vendor-rated Data Manager.
    2. For reports that fail data validation:
    3. For data that has been rejected by Vendor-rated Data Manager (and has not reached the processing by Billing), make the necessary corrections to the usage report and resubmit it to Vendor-rated Data Manager.

Other troubleshooting scenarios:

Stage 2: Processing by Vendor-rated Data Manager

The Charges Rate Task processes the imported usage data in batches and calculates the remaining charges based on the imported data and configured margins.

For monitoring and troubleshooting:

  1. In the Provider Control Panel, go to Operations > Operations > Tasks > Periodic. Search the Charges Rate Task. For each usage data type, there is a separate task. By default, each task is launched every thirty minutes.
  2. Make sure that each task is scheduled (the task status is Rescheduled).
  3. Click the necessary task and inspect the task profile and logs. For detailed information about errors and troubleshooting actions, please see Monitoring and Troubleshooting the Charge Processing by Vendor-rated Data Manager.

  4. For data that has been rejected by Vendor-rated Data Manager (and has not reached the processing by Billing), make the necessary corrections to the usage report and resubmit it to Vendor-rated Data Manager.

Other troubleshooting scenarios:

Stage 3: Outflow. Processing by BSS

The ChargesProcessing task generates billing orders and reseller transactions based on charges in the Processed status. For charges in other statuses, please see the previous stage.

For monitoring and troubleshooting:

  1. In the Provider Control Panel, go to Billing > Operations > Tasks. Search the ChargesProcessing task. By default, the task is launched every thirty minutes.
  2. Make sure that the task is properly scheduled:
    1. Click the task, then click the ChargesProcessing event type.
    2. In the event type profile, switch to the Scheduler tab. Inspect the Next Execution Time column. For more information, please see Configuring Event Settings in Billing.
  3. For detailed information about errors and troubleshooting actions, please see Troubleshooting Charge Processing by Billing.

  4. For data that has been rejected by Vendor-rated Data Manager (and has not reached the processing by Billing), make the necessary corrections to the usage report and resubmit it to Vendor-rated Data Manager.

Other troubleshooting scenarios:

Note: If you cannot find a solution, please submit a ticket to CloudBlue Commerce Support.

Typical Troubleshooting Scenarios

This section contains typical error scenarios that may occur during usage data import and processing.

Note: The batch and charge statuses can be found at the usage report processing results page at the Provider Control Panel > Billing > System > Logs > Vendor Rated Data tab.

  Symptoms Troubleshooting
    Main Procedure Options Following Actions
Issues that occur during the processing of charges by Vendor-rated Data Manager
1. A usage report has been uploaded but the processing failed (the batch status is Failed) Download and inspect the error report linked to the reports log table or obtain the batch ID from the corresponding table row and inspect the charges included in the batch. You find related charges in the Processing Failed status. See 6.
Charges related to the batch ID are absent. Fix and reimport the usage report file.
Other cases. Submit a ticket toCloudBlue Commerce Support.
2. A usage report has been uploaded but the Processing batch status hangs for a long time
  1. Obtain the batch ID from the reports log table.
  2. Inspect the batch status.
 The batch status is Processing. The batch may contain too many charges. Wait for the processing to complete.
The batch status is Ready.
  1. Make sure that Charges Rate Task is scheduled and its status is not Failed. You can launch the task manually, if necessary.
  2. If the Charges Rate Task status is Failed, submit a ticket toCloudBlue Commerce Support.
The batch hangs in the New status for a long time.

  1. Make sure that charge processing is enabled. If disabled, make sure that the related issues are resolved and enable charge processing.

  2. Make sure that Charges Rate Task is enabled and its status is not Failed.

  3. Execute Charges Rate Task.
Other cases. Submit a ticket toCloudBlue Commerce Support.
Issues that occur during the processing of charges by Billing
3. An invoice contains wrong amounts.
  1. Go to Vendor Rated Data.
  2. Verify the amounts of the charges related to the subscription.
 All Tiers (TR) charges have incorrect amounts. See 11.
Price-Rated (PR) charges have incorrect amounts. See 12.
Cost-Rated (CR) charges have incorrect amounts. See 13.
4. No invoice generated Make sure that a billing order has been created. A billing order has been created. Submit a ticket to CloudBlue Commerce Support.
A billing order is absent. See 5.
5. No billing order created.
  1. Go to Vendor Rated Data.
  2. Make sure that charges related to the subscription exist and are in the correct statuses.
 Charges exist and are in the Processed status. See 6.
Charges exist and are in the Processing Failed status. See 7.
Charges exist and are in the New status. See 8.
Charges exist and are in the Reporting Failed status. See 9.
Charges do not exist. See 10.
6. Charges exist and are in the Processed status. Inspect the batch status. The batch status is Processed but there is no billing order.
  1. Make sure that the ChargesProcessing task is scheduled and is not in the Failed status.
  2. Wait for the ChargesProcessing task to complete.
The batch status is Processing failed. Inspect the failed charges in the batch (see 7.)
The batch status is Report failed. Inspect the charges which failed to be reported (see 9.)
The batch status is Processing. The batch may contain too many charges. Wait for the processing to complete.
The batch status is Ready. Make sure that the Charges Rate Task is scheduled and its status is not Failed.
7. Charges exist and are in the Processing Failed status.

View the charges and check error descriptions. Fix the error causes and resubmit the batch for processing.

If you cannot fix the error, issue a ticket.

    
8. Charges exist and are in the New status Inspect the batch status. The batch status is Processing. The batch may contain too many charges. Wait for the processing to complete.
The batch status is Ready. Make sure that Charges Rate Task is scheduled and its status is not Failed.
The batch status is New. The vendor rated data import has probably not been completed. Wait for the import to complete.
The batch hangs in the New status for a long time.

The vendor rated data import has probably failed.Re-upload the usage report manually or issue a ticket.

  1. Make sure that charge processing is enabled. If disabled, make sure that the related issues are resolved and enable charge processing.
  2. Make sure that Charges Rate Task is enabled and its status is not Failed.
  3. Execute Charges Rate Task. If it fails, re-upload the usage report manually or issue a ticket.
9. Charges exist and are in the Report Failed status.

View the charges and inspect error descriptions. Fix the root cause.

    
10. Charges do not exist. Inspect the vendor-rated data source.    
11. All Tiers (TR) charges have incorrect amounts. Make sure that the usage report file has the correct data. The usage report file has incorrect data.
The usage report data is correct. Submit a ticket to CloudBlue Commerce Support.
12. Price-Rated (PR) charges have incorrect amounts. Inspect the source and derivative charges (see the 'Derivative' column in the usage report file). The source charges have incorrect amounts.
The source charges have correct amounts while the derivative charges have incorrect amounts.
13. Cost-Rated (CR) charges have incorrect amounts. See 12. See 12. See 12.

Usage Report Import Troubleshooting

Note: The usage report format guidelines can be found in the usage report template.

This section applies to usage reports delivered from CloudBlue Connect or uploaded from the CloudBlue Commerce UI.

During the usage report file import, the following errors can occur:

Error Type Error Message Troubleshooting

Runtime exception (ErrorCode 500)

These messages are visible in the UI.

Cannot find an account for one of the tiers for {record_id} = %s. Make sure that the corresponding provider or reseller account exists.

Check the usage report data. If there are no data errors, submit a ticket to CloudBlue Commerce Support.

Note: For each usage value (per MPN per subscription), in the usage report file, there are as many records as there are distribution tiers. Each tier 0 (customer) record has corresponding tier 1, 2 and so on records for the whole distribution chain, from the lowest-line reseller to the provider. The <subscription ID> field for all tiers has the same value: the customer's subscription ID. Tier X accounts (reseller accounts of the corresponding distribution levels) are determined based on the customer subscription ID and the tier number. If a tier X account cannot be determined for a customer's subscription, this error occurs.

Typical distribution chain tiers (levels):

  • tier 3 = provider
  • tier 2 = L1 reseller (provider's operating unit)
  • tier 1 = L2 reseller
  • tier 0 = customer

Unsupported Usage Schema value: {%s}.

Make sure that the Usage Schema field has one of these values:

Cannot find fields {%s} in the {general} tab.

Make sure that the fields specified in the error are configured in the ‘general’ tab of the report file.

Cannot parse the date. The date format must be 'yyyy-MM-dd HH:mm:ss'.

Make sure that all ‘date’ fields have the necessary format.

Invalid format: A tab with the name %s does not exist.

The 'records’ and ‘general’ tabs are required for the usage report.

Empty {records} tab.

Make sure that the 'records’ tab of the usage report is not empty.

One of these usage report format validation errors:

  • The 'record_id' field cannot be blank.
  • The 'record_note' value cannot exceed 255 characters.
  • The 'record_id' field cannot be blank.
  • The 'record_note' value cannot exceed 255 characters.
  • The 'start_time_utc' field cannot be blank.
  • The 'start_time_utc' value cannot be a future date.
  • The 'end_time_utc' cannot be blank.
  • The 'end_time_utc' value cannot be a future date.
  • The 'item_mpn' field cannot be blank.
  • The 'item_unit' field cannot be blank.
  • The 'quantity' field cannot be blank.
  • The 'report_start_time_utc' value cannot be a future date.

  • The 'report_end_time_utc' value cannot be a future date.

Make sure that the usage report file meets the format requirements outlined in the instructions inside the downloaded report template.

Error Report (guiding issues).

These messages are visible in the error report.

Cannot map a subscription by its Reconciliation ID: {reconciliationId}.

The subscription reconciliation ID was not propagated to CloudBlue Commerce. Specify it in the UI.

Cannot map a resource by its MPN: {itemMpn}.

A resource with the MPN returned in the error does not exist in CloudBlue Commerce.

Error Report (Batch Import)

These messages are visible in the error report.

Cannot retrieve the account, subscription, or resource by the APS uuid: '{uuid}'.

Make sure that an account, subscription, or resource with APS uuid {uuid} exists: Send a GET request with the with APS uuid to the APS bus and identify the entity in BSS by the ID returned in the response.

Cannot retrieve Vendor Contract ID for subscription {subscriptionId}.

  1. Make sure that the reseller profile is configured and available on the APS bus.

  2. Configure Vendor Contract ID through custom account attributes.

Duplicate charges found in the packet.

Validate the usage report file against the data source. If the duplication is a result of human error, remove the duplication from the usage report and re-upload it. If the duplication is a result of an unknown system error, submit a ticket to CloudBlue Commerce Support.

Monitoring and Troubleshooting the Charge Processing by Vendor-rated Data Manager

To monitor the processing of charges by Vendor-rated Data Manager, along with the Charges Rate Task, you need to monitor charges and batches.

Monitoring Batch and Charge Statuses. Resubmitting a Failed Batch for Processing

Vendor-rated charges are imported, processed in batches by Vendor-rated Data Manager (one usage report is one batch), and reported to BSS for further processing.

To monitor batches and resubmit failed batches for processing:

  1. In the Provider Control Panel, go to Billing > System > Logs. Switch to the Vendor Rated Data tab.

  2. In the Batch ID column, click the necessary batch ID.

    Note: Make sure that the Batch ID column is visible.

  3. Click View Batch. Inspect the batch status:
    • Processed: This status means that the batch is successfully processed.
    • Processing Failed: This status is assigned to a batch when any of the charges fails to be rated. In this case, you can resubmit the batch for processing by clicking Resubmit Batch; if it does not help, troubleshoot the charge processing errors.
    • Report Failed: This status is assigned to a batch after the batch goes through the Processed status but one of the already rated charges fails further processing by Billing. For troubleshooting, please see Typical Troubleshooting Scenarios.
    • Ready: This status means that the batch is pending processing.
    • Processing: This status means that the batch is currently being processed. Usually, batches are processed fast and do not stay in this status for a long time.

To monitor a charge:

  1. In the Provider Control Panel, go to Billing > System > Logs. Switch to the Vendor Rated Data tab.
  2. In the Charge ID column, click the necessary charge and inspect the charge profile. For detailed information on the charge statuses, please see Troubleshooting Charge Processing by Billing.

Troubleshooting Vendor-rated Data Manager Charge Processing Errors

Note: Both in the Usage Report Template and error messages and troubleshooting actions below, the following abbreviations from CloudBlue Connect terminology are used for rating models (or rating schemas):
• TR - for Usage data rated by the vendor for all parties (Tiers Rated)
• PR - for Usage data rated by the vendor for the End Customer (Price Rated)
• CR - for Usage data rated by the vendor for the Provider (Cost Rated)

Error Message Explanation Troubleshooting

Margin value not found for resource {BSS resource APS ID} of service plan {service plan APS ID} and account {account APS ID}.

new: Margin/markup configuration not found for resource {BSS resource APS ID} for service plan {service plan APS ID} for reseller account {account APS ID}.

 For the PR model: There is no margin configuration for the resource for the service plan.

Under the service plan owner, configure margins for the resource for the service plan.

If there is a custom margin configuration for the reseller account for the service plan, configure custom margins for the resource.
For the CR model: There is no markup configuration for the resource for the service plan.

Under the service plan owner, configure markups for the resource for the service plan.

If there is a custom markup configuration for the reseller account for the service plan, configure custom margins for the resource.

Margin configurations for resource {BSS resource APS ID} of service plan {service plan APS ID} and account {account APS ID} conflict for charge {charge ID}.

new: There are multiple margin/markup configurations for resource {BSS resource APS ID} for service plan {service plan APS ID} for account {account APS ID} for charge {charge ID}.

For the PR model: There is a margin configuration where more than one attributes or attribute values match the reported usage detail.

 Eliminate the margin configuration conflict.

For the CR model: There is a markup configuration where more than one attributes or attribute values match the reported usage detail.

Eliminate the markup configuration conflict.

The Your Margin value is not set for resource {BSS resource APS ID} for service plan {service plan APS ID} for account {account APS ID}.

 For the PR model: The Your Margin value is not set for the resource for the service plan.

Under the service plan owner, set the Your Margin value for the resource for the service plan.

If there is a custom margin configuration for the reseller account for the service plan, set a custom Your Margin value for the resource of the service plan.
For the CR model: The Your Markup value is not set for the resource for the service plan.

Under the service plan owner, set the Your Markup value for the resource for the service plan.

If there is a custom markup configuration for the reseller account for the service plan, set a custom Your Markup value for the resource for the service plan.

The Recommended Total Margin/Markup value is not set for resource {BSS resource APS ID} for service plan {service plan APS ID} for account {account APS ID}.

 For the PR model: The Recommended Total Margin value is not set for the resource for the service plan. Under the service plan owner, set the Recommended Total Margin value for the resource for the service plan.
For the CR model: Your Markup and Recommended Total Markup are not set for the resource for the service plan for the account.

Use one of these options:

  • (for an individual reseller) Under the reseller who distributes directly to the end customer, configure markups for the resource for the service plan for the account.

  • (for all resellers) Under the service plan owner, set the Recommended Total Markup value for the resource for the service plan.

Default margin configuration not found for service plan {service plan APS ID}.

 For the PR model: There is no default margin configuration for the service plan. Under the service plan owner, create the default margin configuration for the service plan.
For the CR model: There is no default markup configuration for the service plan. Under the service plan owner, create the default markup configuration for the service plan.

Account {account APS ID} is not the owner of service plan {service plan APS ID}.

 Applicable only for the CR model. The account APS ID must be either the distribution contract owner or its operating unit ("OPCO"). Fix and reimport the usage report.

Troubleshooting Charge Processing by Billing

To monitor and troubleshoot the processing of charges by Billing, along with the ChargesProcessing task, you need to monitor the charge processing logs:

  1. In the Provider Control Panel, go to Billing > System > Logs.
  2. Switch to the Vendor Rated Data tab. The list includes the charges imported to or created by Vendor-rated Data Manager in different statuses. Among the Status column values are the following:
    • Reported: This status means that the charge has been fully processed by Vendor-rated Data Manager and sent to Billing for further processing (to be included in an invoice or a billing order).
    • Processed: This status means that the charge awaits the ChargesProcessing task execution, which generates billing orders based on charges in this status.
    • Preserved: This status is the terminal status assigned to imported charges of the Price Rated type where the end customer is serviced outside CloudBlue Commerce. Such charges are used only to calculate the charges for the rest of the distribution chain; they cannot be included in end-customer billing orders and invoices because those are not generated by CloudBlue Commerce.
    • Processing Failed and Report Failed: Please see Typical Troubleshooting Scenarios.
    • Ready: This status means that the charge is pending processing.

Submitting a Ticket to CloudBlue Commerce Support

When submitting a ticket to CloudBlue Commerce Support:

  1. Add a clear description of what happened and what is expected.
  2. Attach the usage report document, if available.
  3. To report importing and processing issues, attach logs for the necessary period from the Vendor-rated Data Manager microservice (Docker container re-payg).
  4. To report invoicing issues, attach billing logs for the necessary period from the Billing application node (/var/log/pa/billing.log and /var/log/pa/billing.scheduler.log).

  5. Collect and attach records from the Billing DB using the following requests:

    SELECT uri.*, er."ErrorReport" FROM "REUsageReportInfo" uri INNER JOIN "REErrorReport" er ON er."ReportID" = uri."ReportID" AND er."ProductID" = uri."ProductID" WHERE uri."ReportID" = <Report ID>
    SELECT * FROM "Batch" WHERE "BatchID" = <Batch ID>;
    SELECT * FROM "Charge" WHERE "BatchID" = <Batch ID>;

Applying Data Corrections to a Successfully Uploaded Usage Report

There can be situations where there are no technical errors during the usage data import but there are factual errors in the already imported data.

To apply corrections to data of an already uploaded usage report, you need to create and upload another report with respective corrections (a correcting report).

Warning:
• To avoid double-charging for the same usage, include in the report only the rows that need correction.
• To ensure correct charges, complete the quantity and amount fields with the corresponding positive or negative difference with the already uploaded values.

To apply corrections to an already uploaded usage report:

  1. Create another report whose report_id is the original report ID + a suffix, for example: UF-2019-12-9797-0764-correction-01.
  2. Include in the report only the rows that need correction:
    1. Complete the quantity and amount fields with the corresponding delta. For example, if the imported usage needs to be decreased by 5 units and the total by $10, enter '-5' in the quantity field and '-10' in the amount field. If the imported usage needs to be increased by 5 units and the total by $10, enter '5' in the quantity field and '10' in the amount field.
    2. Complete all the other fields with the same values as in the original report.
  3. Upload the correcting report.

© 2021 Ingram Micro, Inc. All Rights Reserved. Privacy Policy, Terms of Use and Legal