Tools, datasets and studies for mining software repositories:
GitHub, GitLab, package registries, Q&A sites and beyond.
- github-proxy-server: proxy for massive data collection from the GitHub REST and GraphQL APIs, managing access tokens and requests to stay within the API limits.
- github-token-donation: web app where trusted people donate GitHub tokens to support our data collection.
- geocoder: turns free-form location text, such as what developers write on their GitHub profiles, into structured places (city, state, country). Ships as a TypeScript library, an HTTP API and a bulk CLI over OpenStreetMap Nominatim, Photon and LocationIQ.
- libpostal-rest-docker: libpostal address parsing as a REST service in a Docker container.
- repo-insights: web app for visualizing repository insights and trends. No longer maintained.
Large-scale studies quickly hit GitHub's per-token rate limits. If you trust this work, you can donate a token through github-token-donation: you authorize a GitHub OAuth App, see exactly which scopes it requests, and can revoke it at any time in your GitHub settings.
Maintained by Hudson Borges.