API Connector

Connector that will enrich data based on API connector. This package uses requests package.

Examples

This example implements the use of the IBGE API and the conversion between the data entry format and the standard used by the Framework. In the definition of the URL_PATTERN the parameters that will be used to collect in the API are defined, these PARAMETERS must be in the standard used by the API. As the Framework works with the JSON format it is necessary to use Convert to transform the input from CSV to JSON. The Converter takes as a parameter the path to the file to be converted (CIDADES_DIR_CSV) and the path of the converted file (CIDADES_DIR_JSON).

# name of file: api-ibge.py

from moredata.enricher import EnricherBuilder, Enricher
from moredata.enricher.api_connector import ApiConnector
from moredata.models.data import Data
from moredata.parser import parse_document
from moredata.utils.util import read_json_from_file, Converter


URL_PATTERN = r'https://servicodados.ibge.gov.br/api/v1/pesquisas/-/indicadores/47001/resultados/{localidade}'

PARAMETERS = {
  "fields": [
    {
      "key": "localidade",
      "name": "Código da Região Geográfica Imediata"
    }
  ]
}

Converter.csv_to_json(CIDADES_DIR_CSV, CIDADES_DIR_JSON)
cidades = Data(data_file=CIDADES_DIR_JSON, parser_func=parse_document, data_type="csv")

api_ibge_enricher = Enricher(connector=ApiConnector(response_parser=response_parser, url_pattern=URL_PATTERN, params=PARAMETERS))

cidades_enriched = \
EnricherBuilder(cidades) \
.with_enrichment(api_ibge_enricher) \
.get_result()

Remember to edit the file path (CIDADES_DIR_CSV and CIDADES_DIR_JSON) for the correct behavior and run the example file.

$ python api-ibge.py

API Connector

class moredata.enricher.api_connector.api_connector.ApiConnector(response_parser, url_pattern, params)

ApiConnector implements interface IEnricherConnector, so this connector can be used to enrich some data.

Parameters

• response_parser (Callable) – a function that will parse the response of the request and returns a dict with the values which will enrich the data.

• url_pattern (str) – url_pattern is the route of the api to retrieve values which will enrich the data.

• params (Dict) – params are a dict that has key as the variable in route and name is where to retrieve the data value to assign in route.

response_parser

Type

Callable

url_pattern

Type

str

params

Type

dict

enrich(data, **kwargs)

Method overrided of interface. This interface do enrichment using API as a enricher and return all data enriched as Json. This method has a cache that will store as dict the url as key and the response parsed as value for don’t spend time with requests that are already done previously.

Parameters

data (Data) –

Yields

data (Dict)