Advancing Evidence-Based Decision-Making in Colorado Policymaking:
A 5-Year Vision Focused on Culture and Structure
MEMO PURPOSE
The purpose of this document is to articulate a commonly accepted vision for Colorado’s approach to evidence-based decision-making in policymaking and to align roles and responsibilities across branches of government with this vision. By making explicit our shared understanding of the work to be done and our approach to it, we can accelerate progress and build on the good work that has come before.
WHY EVIDENCE-BASED DECISION-MAKING?
Evidence-Based Decision-Making (EBDM) in Policymaking
EBDM is the intersection of the best available research evidence, decision-makers’ expertise, and community needs and context. EBDM recognizes that research evidence is not the only contributing factor to policy and budget decisions. Other equally important contextual factors include resourcing, cultural values, community voice, and feasibility of implementation.
Shared Tenets of an EBDM Culture in Policymaking:
EBDM is critical to driving smart state investments, continuous quality improvement, innovation, and outcomes. A successful culture of EBDM is anchored in shared tenets that cross stakeholders, time, and space.
How do these shared tenets show up in practice? Consistent with being outcomes-driven, evidence-building should inform continuous quality improvement and learning to strengthen implementation. Research evidence on outcomes is also critical to ensuring efficient and effective use of resources when measuring whether target goals are being met or prioritizing services for scaling. Finally, generating and using research evidence can help cross-system partners better align and integrate as they work to understand service gaps in reach and access, and then innovate and evaluate solutions to identified problems.
Promoting an EBDM culture requires collective effort across levels and branches of government, including
, appointed, and career leadership and staff. A common understanding of the why, what, and how of EBDM allows leadership from diverse systems to identify shared goals and develop strategies that contribute to achieving them. This, in turn, promotes consistency among public sector staff, regardless of branch of government or state agency, in what it means to fulfill Colorado’s commitment to EBDM. As a result, research evidence use can help strengthen outcomes and return on investments for Coloradans.
Memo Foundations: The vision articulated in this memo builds off a strong foundation of evidence-based practice and policy in Colorado. In fall 2018, the Joint Budget Committee (JBC) of the General Assembly supported the use of evidence standards recommended by the Colorado Evidence-Based Policy Collaborative, which subsequently informed SB21-284 (opens in a new window) (Evidence-Based Evaluations for Budget). SB21-284 describes how research evidence should be used to inform funding decisions for programs implemented and delivered by state agencies; however, funding decisions are just one use case in EBDM. Without an explicit framework for EBDM in Colorado, including agreed-upon definitions of terms and clear roles and responsibilities, there is the potential for miscommunication and inconsistency across, and even within, branches of government. This memo aims to provide that framework.
This memo leverages the robust body of literature on the use of research evidence in policy decision-making to help ensure that purpose, vision, and strategies are aligned with the “evidence of using evidence.” Across policy areas, rigorous research evidence has identified the conditions needed to achieve an EBDM culture that is meaningful, efficient, and effective. These studies were used in crafting vision, content, and approach.
Finally, this vision was co-developed with members of the executive and legislative branches as well as non-governmental partners, including JBC members and staff; the Governor’s Office of State Planning and Budgeting, Office of Operations, and Office of Information Technology; General Assembly members; representatives of several state Departments; and the Colorado Evidence-Based Policy Collaborative. Together, these stakeholders helped to shape content and articulate the value of Colorado’s approach to EBDM in policymaking.
WHAT DOES A SUCCESSFUL EBDM CULTURE LOOK LIKE? A SHARED VISION
It is important to articulate how we will know when we have achieved a successful EBDM culture. By defining the key features of a successful EBDM culture in Colorado, we can build capacity for successful implementation, measure progress over time, invest resources in areas that are proving difficult, and ensure every department, agency, and decision-maker has the support necessary to meaningfully contribute.
Using a “best available evidence” approach can help decision-makers overcome three known EBDM challenges, as illustrated in the table below.
| The Challenge | The Solution | Example |
|---|---|---|
| Taking a narrow or one-size-fits all approach to evaluation | Meet programs and practices where they are, matching evaluation approach to need. An EBDM culture embraces the reality that not every policy area can approach evaluation the same way so what constitutes the “best available evidence” will vary widely during budget and policy decision-making. | Randomized controlled trials may be appropriate for established programs, but ill-advised in other situations, such as for new programs and populationlevel practices. |
| Biased uses of research evidence and managing conflicting evidence | Develop and implement transparent processes that leverage existing community and scientific expertise to promote a non-partisan approach to identifying the best available evidence. | Using results from evidence-based clearinghouses to source, understand, and apply the most rigorous evidence. |
| Confusing the absence of research evidence with evidence of ineffectiveness. | Recognize that the best available evidence may be limited. In these cases, the default assumption cannot be that the program is ineffective. Rather, the program or practice should be assessed for evaluation readiness and evidence-building. | County-designed programs may measure outputs like reach or access, which may suggest program potential, but on their own, do not illustrate effectiveness. |
Four Essential Tasks:There are four essential tasks that support decision-makers in using the best available evidence.
- Acquire (find and access) the best available evidence on the topic
- Critically appraise and summarize the best available evidence
- Make sense of (interpret) the best available evidence in relation to relevant context
- Apply the best available evidence and contextual factors to make decisions
In the tasks above, decision-makers work with the best available evidence that already exists to make a decision. In the course of the decision-making process, it is vital that opportunities to build evidence are also identified, incentivized, and rewarded as appropriate. This includes determining what additional evaluation is needed to inform future decisions, measuring the impacts of decisions made, and strengthening implementation over time. Pairing the evidencebuilding process with evidence application is critical to achieving maximum value of EBDM.
Roles and Responsibilities:Together, the roles outlined below contribute to both (a) making today’s deci
: There are four essential tasks that support decision-makers in using the best available evidence.
- Acquire (find and access) the best available evidence on the topic
- Critically appraise and summarize the best available evidence
- Make sense of (interpret) the best available evidence in relation to relevant context
- Apply the best available evidence and contextual factors to make decisions
In the tasks above, decision-makers work with the best available evidence that already exists to make a decision. In the course of the decision-making process, it is vital that opportunities to build evidence are also identified, incentivized, and rewarded as appropriate. This includes determining what additional evaluation is needed to inform future decisions, measuring the impacts of decisions made, and strengthening implementation over time. Pairing the evidencebuilding process with evidence application is critical to achieving maximum value of EBDM.
Roles and Responsibilities:Together, the roles outlined below contribute to both (a) making today’s decisions using the best available evidence, and (b) generating the new research evidence necessary to make even better decisions in the future. Each branch of government has their own unique perspectives and needs as well as legally assigned responsibilities in which EBDM can be embedded. As such, application of EBDM in each branch will look different. In the table below, EBDM best practices are summarized. The list is not exhaustive, but rather, illustrative of the major areas for embedding EBDM. It is important to have transparency in process as well as sufficient cross-checks (trust with accountability) when implementing EBDM systemwide. Together, the roles outlined below contribute to both (a) making today’s decisions using the best available evidence, and (b) generating the new research evidence necessary to make even better decisions in the future. Each branch of government has their own unique perspectives and needs as well as legally assigned responsibilities in which EBDM can be embedded. As such, application of EBDM in each branch will look different. In the table below, EBDM best practices are summarized. The list is not exhaustive, but rather, illustrative of the major areas for embedding EBDM. It is important to have transparency in process as well as sufficient cross-checks (trust with accountability) when implementing EBDM systemwide.
| Branch of Government | Role | Role Summary of Responsibilities |
|---|---|---|
| Legislative Branch | Joint Budget Committee (JBC) Budget and Policy Analysts | Provide JBC members the best available evidence in a nonpartisan, comprehensible way so research evidence can appropriately inform budget and policy decisions. Analyze budget requests from the Executive Branch and make recommendations to the JBC concerning departmental resource allocation. As requested, support legislators in incorporating the best available evidence into legislation. |
| Legislators, including JBC members | Use the best available evidence, as applicable, to inform and prioritize budget and policy decisions, while also considering context factors like community needs, resource availability, and feasibility. Identify opportunities to build the capacity of decisionmakers, staff, and agency leadership to generate and use research evidence. Fund agencies to build research evidence for programs and practices matched to the needs of Colorado residents, including sustainability plans for pilot programs. Use the best available evidence in developing and considering legislation. | |
| Executive Branch | Governor’s Office | Make requests to the JBC on resource allocation for programs and practices aligned with the Governor’s priority issue areas and, as applicable, informed by the best available evidence. Coordinate performance management for existing and new programs and practices. As applicable, support agencies in building research evidence for programs and practices matched to the needs of Colorado residents, including program design, implementation, and evaluation priorities. Identify opportunities to build the capacity of agency staff and leadership to use research evidence in decision-making. |
| Executive Branch (cont’d) | Agency Leadership, Staff, and Legislative Liaisons | Participate in capacity-building opportunities to improve the use of research evidence in decision-making. As applicable, understand and use the best available evidence to inform and prioritize budget and policy decisions on issue areas aligned with agency and administration priorities. Make budget requests to the Governor’s Office of State Planning and Budgeting that includes the best available evidence, as applicable. |
| NonGovernmental Partners | Colorado Evaluation and Action Lab | Serve as boundary spanning leadership for decision-makers across the policy process, upholding an independent, nonpartisan commitment to capacity-building, coordination, and strategy that supports growth of Colorado’s EBDM culture. Measure progress in developing and executing this EBDM vision over time. Support development of EBDM best practices and tools for decisions makers, agency, and leadership to be effective and efficient in research evidence use and evidence-building. |
| Researchers/ Evaluators | Support evidence-building for programs and practices matched to the needs of Colorado residents. Together with executive and legislative decision-makers, identify research priorities to inform policy decisions. | |
| Clearinghouses | Through a rigorous review process, synthesize existing research evidence on programs and practices of interest to policymakers. | |
| Community Voice and Constituents | Identify priorities of relevance with which evidencebuilding goals and investments should align. Contribute to meaning making and contextualizing as research evidence is applied to make decisions. |
NEXT STEPS – FROM VISION TO EXECUTION
Moving toward a broad-based culture of EBDM in Colorado is a long game. The next step is to co-develop a 5-year strategic plan for executing the vision with representatives from the various roles outlined in this memo. The strategic plan will also include communications, change management, and implementation plans that roll out over the next 5 years. This memo will anchor us to the agreed-upon vision and serve as the “evergreen” as we develop the full suite of tools, structures, and best practices needed to build capacity for an EBDM culture and execute this vision. We invite you to join the conversation by contacting Name Blocked.