Country Wide NWS
Fetching data from Country Wide NWS with Airbyte and Python.
Description
This Airbyte Connector uses the HTTP API provided by the Country Wide NWS in Python. With this connector, data is easily pulled from Country Wide NWS and loaded into the ViaWarn Postgres database, and are available in the ViaWarn app.
The connector is built on top of the Airbyte open-source platform, which provides a standardized way of integrating data sources and destinations. It is designed to be simple and easy to use. The required configuration is show in the the sections below.
The pulled resources from the NOAA NWS API are the following:
Requirements and Installation
Customized files
After creating the Airbyte connector template, modify the next files to retrieve data from NOAA-NWS.
spec.yaml
documentationUrl: http://docs.oasis-open.org/emergency/cap/v1.2/CAP-v1.2-os.pdf
connectionSpecification:
$schema: http://json-schema.org/draft-07/schema#
title: Viawarn US CW NWS Spec
type: object
properties:
documentationUrl:
type: string
description: You can consult https://alerts.weather.gov/
config.json
{}
configure_catalog.json
{
"streams": [
{
"stream": {
"name": "weath",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}
source.py
Location: source-viawarn-us-cw-nws/source.py
class ViawarnUsCwNwsStream()
- url_base
This field is the first part of the URL.
String
- next_page_token()
Pagination is not provided by this API.
return None
- request_params()
The API does not require any parameter.
return None
- parse_response()
It is a placeholder to store the request.response object and constructs the appropriate mapping object for each parsed element by each Stream class.
yields an empty dictionary.
class Weath()
- primary_key
Unique field required in the stream associate to each output record.
String
- path()
subdomain and parameters to pass into the URL.
return a string
- parse_response()
Definition of how a response is parsed.
Set a definition to: primary key and ewkt fields.
return an iterable containing each record in the response.
class SourceViawarnUsCwNws()
- check_connection()
Set up a test to verify the connection between the connector and the API. This test should return two variables: the first one is a boolean that indicates the status of the connection, while the second one shows a log message if the connection fails.
return True, None
- streams()
