ARSC 5040

Level

Graduate (one-credit mini-course)

Prerequisites

None. The course is for researchers with limited or no programming experience.

Length

5 weeks, one 2.5-hour class a week (1 credit)

Tools

Python, Jupyter, requests, BeautifulSoup, pandas, Selenium, PRAW, Mastodon.py

Taught

3 terms, Spring 2019 to Spring 2023

License

CC BY-NC-SA 4.0

Overview

Web Data Scraping teaches researchers with limited or no programming experience to retrieve (“scrape”) data from the web, and to weigh the ethics and law of each method as they learn it. I taught it as ARSC 5040, a five-week, one-credit mini-course for the Interdisciplinary Training in the Social Sciences (ITSS) program at CU Boulder. It is for people in the social sciences and the humanities who think computationally but have written little or no code. The course uses Python and its data analysis libraries.

The classes move from web pages to programming interfaces. Students first read HTML, XML, and JSON and parse web pages into tables. Then they control a web browser with code, and in the last two classes they collect data from application programming interfaces (APIs). Consent, privacy, terms of service, robots.txt, the Computer Fraud and Abuse Act, spoofed headers, and the load that scrapers put on servers come up all along the way. Each class is 2.5 hours with a break in the middle, and it mixes lecture-by-notebook with hands-on exercises. Every class ends with links to resources and take-home exercises, and students can present their solutions at the start of the next one.

The materials build on resources that Allison Morgan and I made for the 2018 Summer Institute for Computational Social Science in Boulder, which in turn came from resources by Simon Munzert and Chris Bail. The API notebooks adapt Cody Buntain’s seminar on Social Media and Crisis Informatics and the PRAW documentation.

Learning objectives

1. Navigate and access structured web data in HTML, XML, and JSON.
2. Identify the relevant structures in a semi-structured web page with the browser’s developer tools (“Inspect”).
3. Use Python libraries to request and parse web data, and turn it into tables.
4. Retrieve data from application programming interfaces (APIs).
5. Reflect critically on the technical and ethical limits of web scraping.

Topics

Each class is built around one Jupyter notebook. I work through it with the students, and they stop to do short exercises in the same notebook.

Readings

There is no textbook. The notebooks link to documentation and short readings, and the open ones are listed here by week.

• Introductions: Bail, “Screen-Scraping in R”; “To Scrape or Not to Scrape: The Technical and Ethical Challenges of Collecting Data off the Web” (Storybench); Densmore, “Ethics in Web Scraping”; “On the Ethics of Web Scraping”; the Computer Fraud and Abuse Act, 18 U.S.C. § 1030; ACLU, “Sandvig v. Sessions”; Jupyter documentation
• Scraping HTML: Beautiful Soup documentation; HTML tables (W3Schools); Python exceptions (Real Python)
• Selenium: Selenium with Python; XPath syntax (W3Schools)
• Internet Archive and Wikipedia: the Wayback Machine; the Wikipedia API; the Wikimedia pageviews API and its page view definitions
• Reddit and Mastodon: PRAW documentation; PMAW (archived Reddit data from Pushshift); Mastodon.py documentation; Mastodon’s user preferences, which let users opt out of search indexing; Academics on Mastodon

Assignments and assessment

The one credit is graded pass/fail, and students pass by submitting the assignments. There are no weights or letter grades; what I watch for is regular attendance, participation, and growth in skill and confidence.

In-class exercises are short tasks inside each notebook: count the members of the U.S. House who represent at-large districts from the House’s XML data, find fields in a JSON object, or change an XPath so that the scraper clicks a different button.

Take-home exercises come at the end of each class. Students can present their solutions at the start of the next class.

Individual projects get class time of their own. Each student applies the methods to a website they want to scrape and debugs the scraper with me.

Adopt this course

Students need the Anaconda distribution of Python (3.9 or later) with Jupyter. The notebooks use requests, BeautifulSoup, pandas, matplotlib, and seaborn. Week 3 needs Selenium and a web driver for the browser, and installing them is the hardest setup step in the course, so plan time for it. Week 5 needs PRAW, PMAW, and Mastodon.py, a Reddit account with an app registration, and an account on a Mastodon server. The optional Wikipedia analysis notebook also uses networkx and NLTK.

Students collect the data live from public websites: House member data in XML, the Academy Awards nominee pages, box office tables from The Numbers, Wikipedia tables and the Wikipedia API, xkcd, a YouTube channel, Wayback Machine snapshots of Facebook’s terms of service, Reddit, and Mastodon. The Week 4 folder includes facebook_tos_archive.json, so the whole class does not need to send the same requests to the Internet Archive. Websites change. Run each notebook before class, because some scrapers break when a site is redesigned, and the Pushshift service is often down.

The Spring 2023 repository has one folder for each week: Introductions, Scraping HTML, Scraping with Selenium, Internet Archive and Wikipedia APIs, and Reddit and Mastodon APIs. The Week 4 folder also has a longer notebook of Wikipedia functions and analyses (revision histories, pageviews, links, and networks), and my wikifunctions library has the same functions.

The plan is five weekly classes of 2.5 hours, each with a break in the middle. The Selenium week and the API weeks can trade places: in Spring 2019, I taught the Wikipedia and Reddit APIs before Selenium. The repositories are under the MIT License; the material on this page is under CC BY-NC-SA 4.0.

Past offerings

Each term had five classes, but their order and the platforms changed. Twitter had the last class in 2019 and 2021. The Twitter material came out of those repositories in 2022, and in Spring 2023 Mastodon took its place, because access to Twitter’s API had become uncertain.

Course materials on this page are licensed under CC BY-NC-SA 4.0: reuse and adapt them with credit, not for commercial use, and share adaptations under the same license. Last reviewed October 8, 2026.