Factors influencing evaluation use has been a primary concern for evaluators. However, little is known about the current conceptualizations of evaluation use including what counts as use, what efforts encourage use, and how to measure use. This article identifies enablers and constraints to evaluation use based on a scoping review of literature published since 2009 ( = 47). A fulsome examination to map factors influencing evaluation use identified in extant literature informs further study and captures its evolution over time. Five factors were identified that influence evaluation use: (1) resources; (2) stakeholder characteristics; (3) evaluation characteristics; (4) social and political environment; and (5) evaluators characteristics. Also examined is a synthesis of practical and theoretical implications as well as implications for future research. Importantly, our work builds upon two previous and impactful scoping reviews to provide a contemporary assessment of the factors influencing evaluation use and inform consequential evaluator practice.

Organizational evaluation policies describe how evaluation practices should be structured and implemented. As such, they provide key insights into organizational priorities and values regarding evaluation. However, the link between evaluation policies and how evaluation policies translate into concrete practices has seldom been explored until now. Our study examines the implementation of two Canadian federal government evaluation policies over a 10-year timespan, through the secondary analysis of reports produced on behalf of governmental evaluation functions. Our findings show that some policy elements have been fully implemented, but the way in which these have been implemented varies between organizations. Further, we observed that the level of control of various organizational members responsible for implementing policy elements, as well as time, can influence implementation of certain policy requirements. We conclude by proposing further directions for research to examine the policy-practice link.

Program evaluations that lack experimental design often fail to produce evidence of impact because there is no available control group. Theory-based evaluations can generate evidence of a program's causal effects if evaluators collect evidence along the theorized causal chain and identify possible competing causes. However, few methods are available for assessing competing causes in the program environment. Effect Modifier Assessment (EMA) is a method previously used in smaller-scale studies to assess possible competing causes of observed changes following an intervention. In our case study of a university gender equity intervention, EMA generated useful evidence of competing causes to augment program evaluation. Top-down administrative culture, poor experiences with hiring and promotion, and workload were identified as impeding forces that might have reduced program benefits. The EMA addresses a methodological gap in theory-based evaluation and might be useful in a variety of program settings.

Evaluation capacity building (ECB) continues to attract the attention and interest of scholars and practitioners. Over the years, models, frameworks, strategies, and practices related to ECB have been developed and implemented. Although ECB is highly contextual, the evolution of knowledge in this area depends on learning from past efforts in a structured approach. The purpose of the present article is to integrate the ECB literature in evaluation journals. More specifically, the article aims to answer three questions: What types of articles and themes comprise the current literature on ECB? How are current practices of ECB described in the literature? And what is the current status of research on ECB? Informed by the findings of the review, the article concludes with suggestions for future ECB practice and scholarship.

Knowledge tests used to evaluate child protection training program effectiveness for early childhood education providers may suffer from threats to construct validity given the contextual variability inherent within state-specific regulations around mandated reporting requirements. Unfortunately, guidance on instrument revision that accounts for such state-specific mandated reporting requirements is lacking across research on evaluation practices. This study, therefore, explored how collection and integration of validity evidence using a mixed methods framework can guide the instrument revision process to arrive at a more valid program outcome measure.

This article describes our experience of conducting a 5-year, culturally responsive evaluation of a federal program with Indigenous communities. It describes how we adapted tenets from "participatory evaluation models" to ensure cultural relevance and empowerment. We provide recommendations for evaluators engaged in similar efforts. The evaluation included through a Steering Committee and an Evaluation Working Group in designing and implementing the evaluation. That engagement facilitated in developing a program logic model and medicine wheel and in gathering local perspectives through storytelling to facilitate understanding of community traditions. Our ongoing assessment of program grantees' needs shaped our approach to and development of a diverse array of experiential learning opportunities and user-friendly tools and resources. We present practical strategies from lessons learned during the evaluation design and implementation phases of our project that might be useful for other evaluators.

This article shares lessons learned in applying system evaluation theory (SET) to evaluate a Clinical and Translational Research Center (CTR) funded by the National Institutes of Health. After describing how CTR support cores are intended to work interdependently as a system, the case is made for SET as the best fit for evaluating this evaluand. The article then details how the evaluation was also challenged to facilitate a CTR culture shift, helping support cores to move from working autonomously to working together and understanding how the cores' individual operating processes impact each other. This was achieved by incorporating the Homeland Security Exercise and Evaluation Program (HSEEP) building block approach to implement SET. Each of the seven HSEEP building blocks is examined for alignment with each of SET's three steps and the ability to systematically support the goal of moving CTR cores toward working interdependently. The implications of using HSEEP to support SET implementation for future evaluations are discussed.

The flexibility federal block grants provide recipients poses challenges for evaluation. These challenges include aggregating data on wide-ranging activities grant recipients implement and the outcomes they achieve. In 2014, we began designing an evaluation to address the challenges of assessing outcomes and to improve outcome accountability for the Preventive Health and Health Services Block Grant. Through the use of evaluability assessment methodology, review of existing data and the literature, and key informant interviews, we developed a measurement framework to assess outcomes resulting from recipients' ability to use grant funds to meet their locally prioritized needs. We argue our evaluation approach demonstrates that block grants, and other similarly flexible programs, can be evaluated through appropriately designed measures. Our efforts challenge the idea that flexibility presents an insurmountable barrier to evaluation and outcome accountability for federal block grants.

Despite advances in the sexual violence (SV) prevention field, practitioners still face challenges with identifying indicators to measure the impact of their prevention strategies. Public data, such as existing administrative and surveillance system data, may be a good option for organizations to examine trends in indicators for the purpose of program evaluation. In this article, we describe a framework and a process for identifying indicators with public data. Specifically, we present the SV Indicator Framework and a five-step indicator review process, which we used to identify indicators for a national SV prevention program. We present the findings of the indicator review and explain how the process could be used by evaluators and program planners within other developing topic areas. Tracking indicators with public data, in conjunction with other evaluation methods, may be a viable option for state-level program evaluations. We discuss limitations and implications for practice and research.

HIV continues to significantly impact the health of communities, particularly affecting racially and ethnically diverse men who have sex with men and transgender women. In response, health departments often fund a number of community organizations to provide each of these subgroups with comprehensive and culturally responsive services. To this point, evaluators have focused on individual interventions, but have largely overlooked the complex environment in which these interventions are implemented, including other programs funded to do similar work. The Evaluation Center was funded by the City of Chicago in 2015 to conduct a city-wide evaluation of all HIV prevention programming. This article will describe our novel approach to adapt the principles and methods of the Empowerment Evaluation approach, to effectively engage with 20 city-funded prevention programs to collect and synthesize multi-site evaluation data, and ultimately build capacity at these organizations to foster a learning-focused community.

Interest from Congress, executive branch leadership, and various other stakeholders for greater accountability in government continues to gain momentum today with government-wide efforts. However, measuring the impact of research programs has proven particularly difficult. Cause and effect linkages between research findings and changes to morbidity and mortality are difficult to prove. To address this challenge, the National Institute for Occupational Safety and Health program evaluators used a modified version of contribution analysis (CA) to evaluate two research programs. CA proved to be a useful framework for assessing research impact, and both programs received valuable, actionable feedback. Although there is room to further refine our approach, this was a promising step toward moving beyond bibiliometrics to more robust assessment of research impact.

Despite growing interest in data visualization and graphically aided reporting, the evaluation literature could benefit from additional guidance on systematically integrating visual communication design and marketing into comprehensive communication strategies to improve data dissemination. This article describes the role of targeted communication strategies-based on visual communications, design, and marketing theory-in producing more effective reports. In evaluation practice, well-synthesized and translated reports often require the integration of data from multiple sources, methods, and/or time points to communicate complex findings in ways that elicit productive responses. Visual communication strategies, such as project branding or designing actionable tools with marketing principles in mind, can be applied to optimize effective reporting of complex evaluation findings. This article references a longitudinal, mixed-method evaluation of public school administrators in Michigan to illustrate the application of a systematic communication design framework to produce several graphically aided project materials and subsequent findings reports.

Theory of change (ToC) is currently approach for the evaluation and planning of international development programs. This approach is considered especially suitable for complex interventions. We question this assumption and argue that ToC's focus on cause-effect logic and intended outcomes does not do justice to the recursive nature of complex interventions such as advocacy. Supported by our work as evaluators, and specifically our case study of an advocacy program on child rights, we illustrate how advocacy evolves through recursive interactions, with outcomes that are emergent rather than predictable. We propose putting "practices of change" at the center by emphasizing human interactions, using the analytical lenses of strategies as practice and recursiveness. This provides room for emergent outcomes and implies a different use of ToC. In this article, we make a clear distinction between theoretical reality models and the real world of practices.

Evidence-based interventions hold promise for reducing gaps in health equity across diverse populations, but evidence about effectiveness within these populations lags behind the mainstream, often leaving opportunities to fulfill this promise unrealized. Mismatch between standard intervention outcomes research methods and the cultural and community contexts of populations at greatest risk presents additional challenges in designing and implementing rigorous studies; these challenges too often impede efforts to generate needed evidence. We draw on experiences with American Indian and Alaska Native (AIAN) communities to illustrate how consideration of culture and context can constructively shape intervention research and improve the quality of evidence produced. Case examples from a partnership with one AI community highlight opportunities for increasing alignment in intervention development, research design, and study implementation to maximize both validity and feasibility. We suggest that responsively tailoring intervention outcome research to cultural and community contexts is fundamental to supporting health equity.

Heterogeneity between and within people necessitates the need for sequential personalized interventions to optimize individual outcomes. Personalized or adaptive interventions (AIs) are relevant for diseases and maladaptive behavioral trajectories when one intervention is not curative and success of a subsequent intervention may depend on individual characteristics or response. AIs may be applied to medical settings and to investigate best prevention, education, and community-based practices. AIs can begin with low-cost or low-burden interventions and followed with intensified or alternative interventions for those who need it most. AIs that guide practice over the course of a disease, program, or school year can be investigated through sequential multiple assignment randomized trials (SMARTs). To promote the use of SMARTs, we provide a hypothetical SMART in a Head Start program to address child behavior problems. We describe the advantages and limitations of SMARTs, particularly as they may be applied to the field of evaluation.

Evaluation has become expected within the nonprofit sector, including HIV prevention service delivery through community-based organizations (CBOs). While staff and directors at CBOs may acknowledge the potential contribution of evaluation data to the improvement of agency services, the results of evaluation are often used to demonstrate fiscal prudence, efficiency, and accountability to funders and the public, rather than to produce information for the organization's benefit. We conducted 22 in-depth, semistructured interviews with service providers from four agencies implementing the same evidence-based HIV prevention intervention. We use the lens of "audit culture" to understand how the evaluation and accountability mandates of evidence-based program implementation within HIV prevention service provision affect provider-client relations, staff members' daily work, and organizational focus in natural settings, or contexts without continuous support and implementation monitoring. We conclude with recommendations for improving the use and methods of evaluation within HIV prevention service delivery.

The National Institutes of Health (NIH) is the largest source of funding for biomedical research in the world. Funding decisions are made largely based on the outcome of a peer review process that is intended to provide a fair, equitable, timely, and unbiased review of the quality, scientific merit, and potential impact of the research. There have been concerns about the criteria reviewers are using, and recent changes in review procedures at the NIH now make it possible to conduct an analysis of how reviewers evaluate applications for funding. This study examined the criteria and overall impact scores recorded by assigned reviewers for R01 grant applications. The results suggest that all the scored review criteria, including innovation, are related to the overall impact score. Further, good scores are necessary on all five scored review criteria, not just the score for research methodology, in order to achieve a good overall impact score.

This article describes lessons learned about implementing evaluations in hospital settings. In order to overcome the methodological dilemmas inherent in this environment, we used a practical participatory evaluation strategy to engage as many stakeholders as possible in the process of evaluating a clinical demonstration project. Demonstration projects, in this context, push the envelope about what is known about effectiveness in novel settings, and turnover of staff and patient populations can present challenges to gathering optimal data. By using P-PE, we built capacity in the environment while expanding possibilities for data collection. Suggestions are made based on our experience.

The move to build capacity for internal evaluation is a common organizational theme in social service delivery, and in many settings, the evaluator is also the practitioner who delivers the service. The goal of the present study was to extend our limited knowledge of practitioner evaluation. Specifically, the authors examined practitioner concerns about administering pretest and posttest evaluations within the context of a multisite 7-week family strengthening program and compared those concerns with self-reported attitudes of the parents who completed evaluations. The authors found that program participants ( = 105) were significantly less likely to find the evaluation process intrusive, and more likely to hold positive beliefs about the evaluation process, than practitioners ( = 140) expected. Results of the study may address a potential barrier to effective practitioner evaluation-the belief that having to administer evaluations interferes with establishing a good relationship with program participants.