Skip to main content Link Menu Expand (external link) Document Search Copy Copied
FOSS Foundations Directory

Since all of our primary data (foundations, sponsorships, budgets) is updated regularly, we need to ensure some data consistency to make it valuable for researchers. As new foundations are modeled, we also want to download or visualize various secondary data sources, for comparison to other foundations.

Workflows / Help Wanted

GitHub Pages and Just-The-Docs handle the static site build. But we need a lot more tools to ensure data consistency and automate basic visualizations.

OpenAPI Generated Linting

Current plans are to complete the OpenAPI schemas and endpoints, and then use that definition to generate linting tools for data quality, and to integrate into PR and site build processes.

  • Complete OpenAPI schema spec for our primary data: foundations, sponsorships, entities, budgets.
  • Build OpenAPI schemas to data file linter shim or tool for each dataset:
    • Transform OpenAPI schema into some sort of automated lint tool.
    • Linter reads .md files, and uses schema to validate frontmatter.
    • Linter should also warn on formatting and Markdown issues with .md document bodies (which are displayed as GitHub Pages).
    • Integrate linter into PRs/merges and site builds.
  • Complete OpenAPI definitions for read-only data.
    • We can model datasets with OpenAPI GET operations to retrieve data, similar to:

OpenAPI GET Data Example 

require 'yaml'
require 'open-uri'
asf = YAML.load(URI.open("https://raw.githubusercontent.com/Punderthings/fossfoundation/main/_foundations/asf.md"))
puts asf['legalName']
# YAML Hash should match _data/foundations-schema.json
  • GET operations need to be mocked up to present listings of available data.
    • Run openapi_builder.rb to create foundations/list.json listing with crossref.
    • Build OpenAPI for other primary datasets.

Automate Secondary Datasets

  • Automate updating various secondary datasets.
    • When adding a new foundation, or periodically over time (because tax forms get updated periodically):
      • Scan _foundation/*.md where addressCountry = US and taxID is present
      • Run foundation_reporter.rb -r to download any new _data/p990/*.json files, and generate foundations_990_common.csv
    • When adding or updating a sponsorship:
      • Run sponsor_utils.rb --check to see which entities are stale, changed, or failing to parse.
      • Run sponsor_utils.rb to parse all sponsorship models and websites, and generate _data/sponsorships/*.json sponsor data. Review any WARNING/ERROR output on stderr: orgs that fail to parse keep their previous data, and the script exits nonzero.
      • Changed sponsor lists are added to history/sponsorships/*.json automatically; commit those with the data.
      • Occasionally run sponsor_archive.rb collect to refresh historical lists from each model’s sources (coverage shows gaps).
      • Run sponsor_reports.rb to generate org-funding.json, sponsor-counts.json, sponsor-history.json, and sponsor-income.json (the yearly estimates shown on foundation pages)

Define Historical Modeling Tools

To better understand how the FOSS ecosystem has changed, we should evaluate sponsorships over time, perhaps on an annual basis in the past. We could use the Internet Archive Wayback machine to download sponsorship prospectus and listings from the past. If we go far enough in the past, we’d need to have dated sponsorship models to account for older data. For example, the ASF’s Platinum sponsorship level was originally 100K USD, but was raised to 125K USD.