Скачать или смотреть Keynote: From a spreadsheet to critical data infrastructure: building The COVID Tracking Project

Presenters: Julia Kodysh, Michal Mart, Kevin Miller, Kara Schechtman

The COVID Tracking Project (https://covidtracking.com/about) began in March 2020 as a stopgap spreadsheet maintained by a handful of journalists, hoping to provide some information on COVID across the US until the federal government stepped in. But that day never came. Powered by over a thousand volunteers collecting data from disparate state and federal systems every day, the project accidentally became an indispensable source of data used by governments, individuals, and institutions to make critical decisions. The decentralized nature of public health infrastructure in the United States, which is mostly managed by overstretched health departments at the local and state level, made it impossible to automate the collection and normalization of data on the pandemic. States, which suddenly found themselves needing to produce and report data out of underfunded and overstretched systems, produced COVID dashboards that were all different from each other, didn't provide APIs and used technologies that are difficult to scrape. Human gleaning of the data from these systems allowed us to identify sudden changes in reporting, keep an eye on data definitions, and develop a deep well of experience and metadata that informed how we reported every state's data. Volunteers not only do the critical work of data collection, but through their experience in working with the data, are empowered to make key decisions in our analysis and reporting. Our tooling has matured as we have tested the edges of the possible with Google Sheets. We still use spreadsheets, but have developed more powerful tools to ensure data quality and improve our publishing process. This infrastructure allows teams working on our website and API to use a reliable dataset that serves millions of users a day. The COVID Tracking Project became the de-facto source of COVID data for so many because of a community built in Slack channels by strangers. The tools and sheets and websites we have built are impressive and useful for others to learn from. But the biggest legacy of the project will be the thousands of people who caught a glimpse of the best of themselves during a terrible time.