Coherent Research - DCS API Importer

The DCS API (JSON) Importer allows consumption data to be imported into Sigma by connecting to the Coherent Research Public API for a specific data point reference.

The API specification is documented here: dcs-documentation/DcsPublicApiDescription.md at master · coherent-research/dcs-documentation

Channel Configuration

There are numerous parameters that can be sent in a request to the API which are configurable at a Channel level. These are passed in as parameters on the Channel ID string, which is of the following format:

coh://<UniqueId>/<BackStop_datetime>/<BackTo_Datetime>/<interpolated>/<calibrated>

The parameters are:

Unique ID - the identifier from the source DCS system (this will be prefixed with either an "R" or "VM")
BackStop Datetime - whether there is a date and time that no data should be imported before that overrides schedule execution start date that has been determined
BackTo Datetime - whether there is a date and time that no data should be imported after that overrides schedule execution end date that has been determined
Calibrated Flag - a true/false flag to indicate whether the request asks that any calibrated readings associated with the register should adjust the value returned. If omitted the default is FALSE.
Interpolated Flag - a true/false flag to indicate whether the request asks that any missing readings missing in the source system are estimated and returned in the response. If omitted, the default is FALSE.
- When set to TRUE, the API will return the interpolated reads for data gaps, so gaps have reads with a status "Estimated by Collector" in Sigma
- When set to FALSE, the API will not return interpolated reads for data gaps, so the gaps will have reads of 0 with a status "NULL" in Sigma

There is a wizard in the Sigma screens when setting up the Channel ID to allow these parameters to be specified as required:

Of the parameters than can be specified:

All parameters are optional apart from the Unique ID – this must be specified.
If the other Parameters are not specified, the applicable forward slashes should still be added, e.g. if only specifying the Unique ID, the Channel ID would look like:
- coh://r1234////
The Datetimes specified in the Channel will be of the format: ddMMyyyyTHHmm
- coh://r1234/01012022T0000/31012022T2359/
The interpolated and calibrated flags will wither be “true” or “false”
- coh://r1234///true/false

Schedule Execution

The Sigma Schedule will make the request for each Channel it needs to run for. Any channel starting with “coh://” is included in the list to run for.

The request will be made to the API using the information on the Channel.

Where the BackStop Datetime is specified, this will override the start date determined by the schedule and will import data FROM this date
Where the BackTo Datetime is specified, this will override the end date determined bu the schedule and will import data TO this date.

API Response

The readings provided in the response will be interpreted and stored in Sigma as follows:

Readings that are not a number e.g. NaN/Infinity/-Infinity, these will be captured as a “Zero” value read with a “NULL” status.
For valid readings, the status of the reading will be interpreted and set in Sigma as follows:
- 0 = No Errors
- 1 = Estimated by Data Collector
- 2 = Data Reset
- 3 = Data Reset

There is some other background configuration which controls the system behaviour if the API request received a “429” response due to implemented rate limiting. There are two parameters controlled by TEAM in the configuration of the product (not user configurable):

maxWaitTimeOn429Response – this is set to 120 seconds and is the maximum amount of time the system will permit to wait before failing the schedule. This is to ensure resources are not tied up for significant amounts of times if there are processing issue with the API
maxNumberOf429Responses – this is the maximum number of “429” responses the system will allow in a single Schedule Execution. This is set to 20.

Setting up a Schedule

This is achieved using the “Importing” activity. The importer is called “DCS API (JSON) Schedule” (note the legacy importer is called “DCS Schedule”).

The Importer Options allows the date range to imported to be specified. Note, this will be in the context of the run date.

For example, the below date range (15^th March to 24^th March) will be in the context of the schedule start date of 25^th March based on a setting which will need to be enabled in a subsequent screen. When it next runs on 26^th March, the date range will be 16^th March to 25^th March, and so on.

In the next “Source Options” screen, the API connection User credentials will need to be specified.

In the “Schedule Options” screen, the frequency of the Schedule execution will need to be set.

Please note the Start Time – to allow a rolling importing date range window to be determined, ensure the “Adjust dates relative to date of execution” to be ticked. This will ensure as the execution date moves forwards, so foes the date range for the data being imported.

After clicking “Next” and then “Schedule” on the subsequent screen, the Schedule has been stored and will now run as specified.

Failed Schedules

The system will log out failures as part of each applicable Schedule Execution. Such details will be visible in the “Schedule Management” activity.

The schedule can be selected, and the “Run History” button will show a pop-up with the past few months of execution for the Schedule, along with their status. A particular Schedule Execution can be drilled into (if the status is “Success” or “Partial”) to see the individual channel activity and the date range of data for each one that has been imported.

Where a Schedule has “Failed” it will be possible to download a Log file to view the associated errors. Some of the common expected failures and how they will appear are listed below.

More information on these screens cane be found here: Managing Schedules | ManagingSchedules ViewtheRunHistory

Incorrect Webservice User Credentials

Incorrect Channel ID

Exceeded the “Too Many Requests” Responses Received Threshold

Exceeded the “Too Many Requests Wait Time” Response Threshold