API Tutorial

Tutorial: APIs for Open Government Data

Collecting data is essential for data curators. In order to learn how to collect data, this tutorial introduces the concept of an API, and will show you examples of and practical guidelines for accessing data and metadata from public sources using APIs. Specifically, this tutorial first teaches you how to work with SODA API, which is provided by an open government data portal provider, Socrata. Once the tutorial is complete, you will know how to retrieve open government datasets and their metadata through the SODA API, Discovery API, and Metadata API.

Learning Objectives

  • Understand the concept of an API.
  • Use an API to collect data from public data sources such as open government data portals and open research data repositories.

Prerequisites

  • Computer and internet access
  • Basic understanding of open data and data repositories
  • Basic knowledge of popular open data portal software providers (CKAN, Socrata, Dataverse, etc.)
  • Basic knowledge of common web exchange file formats, such as JSON, XML, and CSV
  • Basic knowledge of any programming language is helpful, but not required

Introduction to API

API stands for Application Programming Interface. An API defines how different applications or different software components talk to each other, providing a way for developers to simplify programming. Just as a graphical user interface makes it easier for people to use programs, APIs make it easier for developers to use functions or data from another application (or another component of an application) without needing to know the implementation details [1][2].

In this tutorial, we will focus on web APIs. Applications such as Facebook, Twitter, and Google Maps provide access to some of their services and data through web APIs, which receive requests and send responses. For example, if you’re an iPhone user and you use Yelp to search for a restaurant, you will see an Apple Map embedded in Yelp showing you the location of the restaurant. Yelp doesn't provide map services; instead, it uses Apple’s API to integrate maps into its own functions. Another example of API usage is the ability to use Google or Facebook to log into other platforms. When you sign up for an account on a non-Google, non-Facebook website, you may be asked to sign in with your Facebook or Google account. When you do so, the website uses a Facebook or Google API to retrieve your account information.

The APIs used in this tutorial adhere to an architectural style called REST, which stands for Representational State Transfer. As blogger Sarah Maddox explains, “REST services tend to offer an easy-to-parse URL structure consisting of nouns that reflect the logical hierarchical categories of the data on offer” [3]. An example of a REST API web URL is: http://api.us.socrata.com/api/catalog/v1/domains. When using a REST API, you will need to know how to structure your query according to rules, and these rules can usually be found in the API documentation for the web application.

Why should we learn about API? Software developers use APIs to create well-structured applications, and data scientists and data curators use APIs to collect large amounts of data automatically. Thus, learning to access data and metadata using an API will be the main objective of this tutorial.

Example Overview

We will walk through two examples of retrieving datasets from the municipal open data portal of City of Seattle using an API:

  • Example 1: Retrieve open data from open government data platforms using an API
  • Example 2: Retrieve open government data metadata using API

The underlying software, Socrata, provides APIs for accessing, filtering, and downloading data and metadata. To learn more about Socrata's API, visit their API starter guide.

Example 1: Retrieve data from Socrata Platforms

Step 1: First, take some time to explore Seattle Open Data Portal datasets and choose a dataset that interests you. For example, Road Weather Information Stations. API documentation is in the upper right corner of the data landing page, as shown in the image below:

Click “API” and copy the “API Endpoint” using the default JSON data format. Then, open a new tab or window and paste the link into the browser. Then, try it again, selecting a different data format, such as CSV. Finally, click “API” again, but this time, click “API Docs," instead of using an endpoint.

Step 2: On the “API Docs” page, scroll down to “Fields." Read through this section, and then try different fields, and their subsequent filters. To do so, click the “+” next to a field. For example, when you expand “stationname” (see image below), you can see a sample filter by clicking “try it.” This shows you what type of response (data) you will get.

Step 3: Now, go to https://dev.socrata.com/docs/endpoints.html. Review the API endpoint concept, and try the examples on the page.

Example 2: Retrieve catalog information and metadata using Socrata's API

Example 2a: Explore catalog-level metadata using Discovery API

Step 1: Go to Discovery API, read the introduction, and scroll through the rest of the content. Try a few examples by clicking on the text boxes that begin with “Search by . . .” You can also view different code examples by choosing from the drop-down menu under “Request” that is set by default to “Raw.” Take a look at a few code examples, especially if you are familiar with any of the provided languages.

Step 2: After trying some of the examples in the documentation, try customizing your own API call in a browser window. Here are some examples:

About

Open Data Literacy is a collaborative project led by the Information School at the University of Washington in partnership with public institutions that create, manage, and publish open data. Open Data Literacy is funded by a generous grant from the Institute of Museum and Library Services.

Contact

The Information School
University of Washington
Seattle, WA 98105
odluw@uw.edu

twitter medium github