COVID-19 Data Project

We have assembled executive orders on coronavirus policies from all states to create a machine readable data set for the public. Our interns have created a data set of these policies by type and date as detailed in this code book spreadsheet. Our data is updated regularly, and text files are now available for those doing research and plugging them into a data base.

The COVID-19 Data Project: Executive Orders & Policies

 

Background

 

The COVID-19 pandemic has unveiled unprecedented challenges for the majority of the countries in the world. In the United States, an estimated 90% of Americans experienced lock-downs and social distancing orders.1 As the novel coronavirus disease 2019 pandemic threatens countries throughout the world, the United States became an epicenter of the outbreak.2 Systematically tracking and assessing policy responses to the public health crisis in each state is essential to better understand how to manage each outbreak and slow the growth of positive cases. Within the current healthcare system, good care is almost unobtainable for many individuals; eligibility is limited, and coverage is costly and too uncertain, resulting in an increasing number of uninsured individuals.3 While researchers and health care professionals are racing to develop a vaccine, officials are implementing social distancing, stay at home orders, restricting travel, and closing businesses and schools to try and slow the spread of the outbreak.4 With positive case numbers on the rise, what responsibilities do governments have? What are the roles of policy makers? How can we ensure the safety and security of citizens throughout the country?

BroadStreet and The COVID-19 Data Project Policy Track have partnered with Temple University’s Center for Public Health Law Research (CPHLR) to look longitudinally at executive orders, health directives, proclamations, and policies related to COVID-19. The COVID-19 Data Project has adapted CPHLR’s process, enabling up-to-date collection of order releases for each state and US territory, conversion into machine readable formats with visually quantifiable and qualitative data. On a weekly basis, new releases and their original sources are gathered, converted, read, and coded for the quality assurance process. Through a review and coding process, aligned with the scope criteria set in place by the CPHLR, the teams of the Policy Track clean and prepare the data for quantitative analysis and further research projects.

 

Methods

 

Collaboration with CPHLR 

 

Starting on April 6th, 2020, Temple University’s CPHLR began collecting, coding and analyzing state statutes, regulations, emergency declarations, and mitigation policies related to the COVID-19 pandemic. The project was divided into phases, with the Phase 1 longitudinal dataset covering legal action from January 20, 2020, through June 1, 2020, for the following states: Alabama, Alaska, Arkansas, California, Colorado, Connecticut, Georgia, Hawaii, Iowa, and Kansas. The start date of January 20, 2020, was selected as it is the day prior to the first case of COVID-19 confirmed by the Centers for Disease Control and Prevention on January 21, 2020. Data from the following sources was used to examine, research, and collect relevant state actions: American Enterprise Institute COVID-19 Action Tracker, American Network of Community Options and Resources COVID-19 State Policy Resources, ASTHO Coronavirus Disease 2019 (COVID-19) Responses Hub, Boston University School of Public Health COVID-19 US State Policy Database, Council of State Governments COVID-19 Resources for State Leaders, Kaiser Family Foundation State Data and Policy Actions to Address Coronavirus, National Conference of State Legislatures State Action on Coronavirus (COVID-19), and The National Governors Association (NGA).5 After beginning Phase 1, the CPHLR team quickly recognized the need for help as the number of legal actions increased dramatically and rapidly. With help of BroadStreet volunteers, some of whom were also students at Temple University, a partnership was formed to collaborate and work side by side. Phases 2 through 4 ramped up including the remaining 40 states, the District of Columbia, and Puerto Rico, with the COVID-19 Data Project Policy Track teams helping to identify which orders were within scope and assisting in building the longitudinal dataset.

 

The Policy Track teams meticulously reviewed the COVID-19 statutes, regulations, emergency declarations, and mitigation policies for three of the ten Phase 2 states: Michigan, Wyoming, and Washington; and assisted the CPHLR team in converting the documents into machine readable format. The leaders of the Policy Track developed a Google Sheet to record, analyze, and code the documents in line with CPHLR’s protocol (see complete process below). Comparing the CPHLR scoping notes with the Policy Track document, the Collaboration Team was able to identify outliers and adjust the process to create the final process document. The Policy Track teams proceeded with the updated process adjustments when working with documents for the remaining states and territories. 

 

Data Analysis Process

 

In order to analyze and generate data each week, the COVID-19 Data Project Policy Track organized into teams. Four of the teams complete “Data Bursts,” a team meeting where individuals work independently and simultaneously to review the documents, generate the dataset following CPHLR’s protocol, while exchanging ideas about how to resolve any issues that arise. Each Burst Team works on a range of a preformatted spreadsheet, unique to each week. The Flight Team, the fifth team, contributes to the data analysis process throughout the week by navigating preparatory and finalization steps. To ensure a smooth and manageable Data Burst for each team, recently released state documents are collected and converted into Google Doc format; this conversion allows the teams to easily edit the documents and efficiently collaborate with one-another throughout the data analysis process.

Before the Data Bursts can begin each week, several tasks must be performed to equip the teams with the necessary documents and data. First, the COVID-19 related official documents for each state are gathered from The Council of State Governments website. Specific dates of the gathering(s) are listed on a Policy Track document, which identifies the states to be analyzed for a given week. Gatherings occur at least once per week. After the gathering for the week is complete, a leader in the Policy Track organizes the documents by state onto a shared spreadsheet. On the shared spreadsheet, the leader enters their name, the source from which they gathered the document(s), and the date each document was gathered. 

On the shared spreadsheet, the documents for each week are separated onto tabs for ease of use during the Data Bursts. As weeks are completed, the completed tabs are combined and passed to the Quality Assurance team for further review. For organization and management purposes, a count of the total number of documents analyzed each week and a count of the total number of documents per state/territory is maintained.

The original format of the documents is either HTML or PDF. The conversion of the documents into Google Doc format is essential to the steps completed during each Data Burst. The HTML formatted documents are converted by copying and pasting the text into a blank Google Doc. The majority of the documents gathered are in PDF format, which requires a more complex conversion process. The PDF documents must be downloaded, converted to a Microsoft Word Document (which requires specific software), uploaded to Google Drive, and then opened as a Google Doc. Once the documents have been converted into the Google Doc format, the direct links are incorporated to the shared spreadsheet for the Teams to access. However, both conversion methods generate spelling and formatting inconsistencies which must be corrected manually. Manual correction may include “auto spell check,” image removal, border removal, and/or hand-typing documents in full. When spelling and formatting has been completed on the documents, it is labeled as “Cleaned” for Teams to recognize.

The deep read is the part of the process which all Policy Track team members are trained to complete during a Data Burst. The review and coding, or “deep read,” of the documents following the CPHLR protocols is the main step of the data analysis process.  The deep read includes the analysis and mark-up of the documents and the entering of binary variables pertaining to the inclusion of “in-scope” topics. The marked-up document provides the Track leaders and the Quality Assurance (QA) team members with important information and creates a track record of the logged binary variables. 

 

Data Burst Outline 

 

Each Data Burst includes the deep read of the COVID-19 state documents released following a procedure intended to review, analyze, and code them as accurately as possible. Each Burst Team Lead hosts a regular, weekly meeting with their team for a 90-minute Data Burst. The Data Bursts are completed in a team-setting in order to provide a group space for the deep reads and to reinforce the community’s bond. It has been found that the analysis of these complex documents is most successful when individuals can actively communicate. 

During the Burst, each team member tracks and documents their work on the shared spreadsheet. The shared spreadsheet is formatted to guide individuals through the analysis of each document, identifying content that is within scope by following the scope descriptions on the spreadsheet. The individuals place a 0 (zero) in cells to signify a document does not contain content about the given scope topics; a 1 (one) is placed in cells to signify a document does contain content about the given scope topics. Cells that have a 1 also have a comment attached to the cell to describe the content found in the document for that topic.

The scope topics and descriptions have been created by Policy Track leaders based on CPHLR’s “protocol notes.” The topics are grouped into color-coded categories, which allows for the mark-up of the document(s) to align with the binary code and attached comments on the spreadsheet. This alignment enables the QA team to easily confirm the content is applicable to the scope topic. Any ambiguous information in the documents is noted and a detailed review is conducted by the QA team to determine if the content is applicable to the scope topics.

The following are the categories of scope topics:

  • The official number or name of the document.
  • The amendment status and effective time period of the document.
  • The State of Emergency status pertaining to the document.
  • Content related to individuals.
  • Content related to entities and public services.
  • Content related to medical practices.
  • Content related to travel restrictions and correctional facilities.

The individuals may also enter notes and/or miscellaneous information about the document that may be of interest to researchers and reviewers. At the conclusion of the Data Burst, the Burst Team lead reviews the work completed by the team members for completion. At the conclusion of each week, the content created by the teams is reviewed by the QA Team before integration into longitudinal dataset performed by a Policy Track leader.

 

Quality Assurance Process

The QA Team completes an in-depth review of the analysis completed by the Burst Teams each week. The QA Team uses a combination of the completed spreadsheet(s) and the mark-up of the individual documents to review the completed analysis. Their process includes review of:

    • All of the 1’s (ones) on the document and the associated comments, which indicate and explain the scope topic’s presence within the body of the documents. 
    • The amendment status and details of the documents.
    • The effective time period of the documents, whether a specific date or a length of time.
    • The content pertaining to the medical practices, as this topic is the most diverse when referenced by the states.
    • The notes and/or miscellaneous information included by the individual, as it may be incorporated into and/or influence the binary coding of the scope topics. 

The review status of each document is labeled on the shared spreadsheet to track and monitor the completion of QA. When the QA Team has completed their in-depth review, a Policy Track leader is alerted and the shared spreadsheet is integrated into the longitudinal dataset. This dataset includes the code of all documents that have been completed, separated by state/territory.

 

Suggested Citation

 

When using data images, downloaded data, or shared document formats, please attribute BroadStreet as well as the original source, when applicable. For examples and more information, review this article which answers the question "How do I cite BroadStreet?".  

 

Report Contributors

 

Robert Birds

Samin Charepoo

Alessandro Malave

Allison Muir, MHA

References

 

  1.  Chan AT, Brownstein JS. Putting the Public Back in Public Health — Surveying Symptoms of Covid-19. N Engl J Med. 2020;0(0):null. doi:10.1056/NEJMp201625

  2.  Monroe M. Florida emerges as world’s new epicenter for COVID-19. TheHill. Published July 8, 2020. Accessed July 20, 2020. https://thehill.com/homenews/state-watch/506492-florida-emerges-as-worlds-new-epicenter-for-covid-1

  3.  Fuchs VR. Health Care Policy After the COVID-19 Pandemic. JAMA. Published online June 12, 2020. doi:10.1001/jama.2020.10777

  4.  Gostin LO, Wiley LF. Governmental Public Health Powers During the COVID-19 Pandemic: Stay-at-home Orders, Business Closures, and Travel Restrictions. JAMA. 2020;323(21):2137-2138. doi:10.1001/jama.2020.5460

  5.  Coronavirus Resources. National Governors Association. Accessed August 6, 2020. https://www.nga.org/coronavirus-resources/