This Airbyte Connector uses the HTTP API provided by the British Columbia MOT in Python. With this connector, data is easily pulled from British Columbia MOT and loaded into the ViaWarn Postgres database, and are available in the ViaWarn app.
The connector is built on top of the Airbyte open-source platform, which provides a standardized way of integrating data sources and destinations. It is designed to be simple and easy to use. The required configuration is show in the the sections below.
The pulled resources from the British Columbia 511 API are the following:
After creating the Airbyte connector template, modify the next files to retrieve data from British Columbia MOT .
documentationUrl: https://docsurl.com
connectionSpecification:
$schema: http://json-schema.org/draft-07/schema#
title: VIAWARN CAN BC MOT - Spec
type: object
properties:
request_limit:
type: integer
description: Request limit as described here https://api.open511.gov.bc.ca/help (Default=50 (Leave empty), Maximum=500)
{
"request_limit":"[limit]"
}
{
"streams": [
{
"stream": {
"name": "event",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}
Location: source-viawarn-ca-bc-mot/source.py
This field is the common part of the URL for all streams.
String
This is a constructor method used to initialize the class. It uses the ‘config’ object to define the ‘request_limit” attribute.
return config[]
Retrieve ‘offset’ parameter from the response if the ‘pagination’ key exists, otherwise return None.
return dictionary
The API requires two parameters: the ‘request_limit’ and the ‘offset’ in which the data will be passed.
return a dictionary
It is a placeholder to store the request.response object and constructs the appropriate mapping object for each parsed element by each Stream class.
yields an empty dictionary.
Unique field required in the stream associate to each output record.
String
subdomain to be passed into the URL.
return a string
Definition of how a response is parsed.
Set a definition to: primary key and ewkt fields.
return an iterable containing each record in the response.
Set up a test to verify the connection between the connector and the API. This test should return two variables: the first one is a boolean that indicates the status of the connection, while the second one shows a log message if the connection fails.
return True, None
The displayed occurrences are limited. To access the complete dataset, please join us in the ViaWarn app.