Big Data and Open Data Sources

CRD 150 - Quantitative Methods in Community Research

This guide lists and describes online sources that provide Big Data or Open Data. Remember to look at the meta data to figure out what kinds of information you can download.

• Airbnb: Provides csv files containing detailed information on data on airbnb hosts. The data are in longitude/latitude. They don’t provide historical data.

• Bikesharing: Web sites providing public use data on bikesharing. Provides station-to-station data.

  • San Francisco
    
  • New York City
  • Chicago
  • Washington D.C.

• OpenStreetMaps. osmdata is an R package for downloading OpenStreetMaps data. The site provides a couple of vignettes on using the package.

• Array of things. The City of Chicago installed modular sensor boxes around Chicago to collect real-time data on the city’s environment, infrastructure, and activity for research and public use. Other cities have followed.

• Zillow. Provides housing price data at the metro, city and zipcode levels. R has a package for downloading Zillow data directly.

• Yelp. A public use dataset put together by Yelp specifically for personal and educational purposes, but has been used in academic and applied research. You can use the Yelp API, and here is a tutorial, and another, but there are some restrictions, specifically getting an access ID and creating your own app. Here is another tutorial for a specific R package that uses the Yelp API.

• Twitter. Twitter provides access to a sample of their tweets. You’ll need to register for an API. Here are some guides to collect and manage tweets in R: here, here, and here.

Open data portals

Many city, county and even state governments maintain open data portals. These portals provide various datasets held and maintained by the public sectors. Some of the data are measured at a fine spatial scale, going doing to longitude/latitude.

There are a couple of sites that maintain open data portal directories, including

• Data.gov

• Open Data Inception

• Open Knowledge Foundation

Here are links to various open data portals in US cities (updated 01/08/24)

California

• Southern California Association of Governments

• Sacramento Area Council of Governments

• Sacramento City

• Sacramento County

• San Francisco City

• Alameda County

• Oakland City

• Oakland City

• Long Beach City

• San Jose City

• Kern County

• Chula Vista City

• Anaheim City

• Riverside County

• Fresno City

• Los Angeles City

• Los Angeles City

• Los Angeles County

• Los Angeles County

• Orange County

• San Francisco City/County

• Santa Clara County

• San Diego City

• California State

Major Cities

• Boston

• New York

• New York

• Chicago

• Philadelphia

• Phoenix

• Detroit

• Baltimore

• Portland

• Seattle

• Washington D.C.

• Milwaukee

• Houston

Your city/county not listed above? Use Google. It’s your friend.

Looking for data?

• Kaggle is a crowd-sourced platform for all things data science. This includes competitions, discussion forums, online tutorials, and most importantly, at least for the purpose of this guide, a repository of big data sources. A lot of these data are not pertinent to this class, but some are; specifically, those with geographic information that allows you to connect data to geographic locations. Check out their datasets here.

• Esri provides a repository that many of its members use to store various big and open data all in shapefile format. Check out what’s available here.

• Awesome Public Datasets

• Amazon AWS Public Data Sets

---

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Website created and maintained by Noli Brazil