VIAWARN

Arizona DOT

Fetching data from Arizona DOT with Airbyte and Python.

Description

This Airbyte Connector uses the HTTP API provided by the Arizona DOT in Python. With this connector, data is easily pulled from Arizona DOT and loaded into the ViaWarn Postgres database, and are available in the ViaWarn app.

The connector is built on top of the Airbyte open-source platform, which provides a standardized way of integrating data sources and destinations. It is designed to be simple and easy to use. The required configuration is show in the the sections below. 

The pulled resources from the AZ 511 API are the following:

Requirements and Installation

Find the requirements and steps to create the connector in the Airbyte documentation.

Customized files

After creating the Airbyte connector template, modify the next files to retrieve data from Arizona DOT .

spec.yaml

Location: source-viawarn-us-az-dot/spec.yaml
Access API: SECURED
In this case, the connector password is specified in the spec file.
spec.yaml:
 
				
					documentationUrl: https://….rvice
connectionSpecification: $schema:http://json-schema.org/draft-07/schema# 
    title: VIAWARN US AZ DOT Spec 
    type: object 
    required: -developer_api_key
        properties: developer_api_key:
        type: string 
        airbyte_secret: true 
        description: Developer API Key obtained from https:...
				
			

config.json

Location: secrets/config.json
This file is ignore by git.
Content:
				
					{
"developer_api_key":"[key]"
}
				
			

configure_catalog.json

Location: integration_tests/configure_catalog.json
List of Streams that describes the data in the data store that a Source represents. It also declares the sync modes supported by each stream (full refresh or incremental).
						
				
					{ 
"streams": 
    [
    {
        "stream": {
            "name": [STREAM_NAME],
            "json_schema": {},
            "supported_sync_modes": ["full_refresh"] 
            },
        "sync_mode": "full_refresh", "destination_sync_mode": "overwrite" 
        } 
    ]
}
				
			

source.py

Location: source-viawarn-viawarn-us-az-dot/source.py

class ViawarnUsAzDotStream()

Create a stream to represent the data that will be pull from the API. It contains all the common API functionalities:
  •  url_base

This field is the common part of the URL for all streams, if it does not exist, add it in each stream method.

String

  • __init__()

This is a constructor method used to initialize the class. It uses the ‘config’ object to define the ‘developer_api_key’ attribute.

return config[]

  • next_page_token()

Pagination is not provided by this API.

return None

  • request_params()

The API requires two parameters: the ‘Developer Key’ and the ‘format’ in which the data will be passed.

return a dictionary

  • parse_response()

It is a placeholder to store the request.response object and constructs the appropriate mapping object for each parsed element by each Stream class.

yields an empty dictionary.

class EVENT()

This method extends the ViawarnUsAzDotStream() method to specified unique behavior for this stream.
  • primary_key

Unique field required in the stream associate to each output record.

String

  • path()

subdomain and parameters to pass into the URL.

return a string

  • parse_response()

Definition of how a response is parsed.

Set a definition to: primary key and ewkt fields.

return an iterable containing each record in the response.

[STREAM_NAME].json

Location: source-viawarn-viawarn-us-az-dot/schemas/[STREAM_NAME].json
Declare schemas for each stream by adding json files. Each file is a JsonSchema describing the output from that stream.
Streams name: event, sign
Previous
Next

Resources

support@viawarn.com