Geographically-related outcomes of U.S. funding for small business research and development: Results of the research grant programs of a component of the National Institutes of Health

This article examines the geographic distribution of funding for the U.S. Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs sponsored by the National Institute of General Medical Sciences (NIGMS). Despite a significant investment in SBIR/STTR and an interest in increasing geographic diversity in the institute’s research portfolio, there has not been an assessment of the distribution of NIGMS’s SBIR/STTR funding, outcomes associated with that investment, and relationships between the two. The geographic distribution of NIGMS’ SBIR/STTR funding was highly concentrated in a small number of states, with a high correlation between each state’s funding and its number of small scientific research and development businesses. Affiliation with a major research university was correlated with several measures of innovation and firm success. Our findings are consistent with earlier research showing that economic activity in research and development and research output tend to cluster in geographic regions where knowledge can be generated and shared more efficiently. These findings lend support to an investment strategy for small business research and development that creates networks between major research universities and small businesses.     

Evaluation approaches in mental health: Implications of the new federalism

Since the introduction of federal mental health legislation in 1963, there has been a changing emphasis on evaluation and accountability. With direct federal funding of community mental health services, accountability demands were met through expectations for local agency evaluation activities which were overseen by federal authorities. The advent of the New Federalism and the shift to block grant funding of mental health services to state mental health authorities have shifted responsibility for evaluation to the states and local programs. This paper reviews federal mental health statutes to trace the extent and locus of required evaluation activities and discusses two approaches to carrying out program evaluation: "top-down" where the evaluation topic, method, and data collection are mandated by an administering or funding body; and "bottom-up" approaches where the subject, method of study, and data to be collected are developed in response to a felt need at the local agency level. A case study of each approach as used at the state level in mental health is examined. Based on the literature and the case studies, conclusions are presented on the pro's and con's of each method in meeting accountability demands and the barriers which must be overcome for either method to be successful.     

Redistributive Policy and Devolution: Is State Administration a Road Block (Grant) to Equitable Access to Federal Funds?

Over the last several decades, block grants have been a keyinstrument by which policy authority is devolved from federalto state governments. Whether the block grant approach producesmore effective redistributive policy management (on equity orefficiency grounds) is an open and central question. We focuson the equity dimension by positing an explanatory frameworkthat links access to federal block grant funding to the transactioncosts associated with intergovernmental grant contracting. Weargue that grant access is a function of both state-level institutionalchoices and the administrative capacity of local governments—whichcombine to influence the salient transaction costs associatedwith grant contracting. We apply our theoretic framework tothe non-entitlement portion of the Community Development BlockGrant (CDBG) program and analyze granting data across distinctinstitutional settings. We find that local administrative capacityis a key to access, indicating that some localities most inneed are least equipped to capture grants, which raises equityconcerns. However, the specific way in which states structurearenas of grant competition can make access more equitable.In short, we show that state-level institutional arrangementscritically affect access to federal block grant funds when thosefunds are allocated through competitive grant contracting tolocal governments.     

As You Sow,So Shall You Reap?

Foundations’ capacity‐building grant programs strive to bolster performance and outcomes for their nonprofit grantees. Yet with few outcome evaluations of such programs, we have limited understanding of whether these capacity‐building efforts achieve their intended result. This study evaluates fifteen years of data for one foundation's capacity‐building grant program to understand whether targeted capacity building for financial management and development contributes to nonprofit financial growth. The authors examine the management–performance link in this context and inform sector leaders who dedicate resources to capacity‐building programs about the outcomes of these efforts.     

Evaluating the integration of strategic priorities within a complex research-for-development funding program

This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.     

THE IMPACTS OF THE USDA BROADBAND LOAN AND GRANT PROGRAMS: MOVING TOWARD ESTIMATING A RATE OF RETURN

In this article, we evaluate the rate of return to government efforts to promote broadband. Specifically, we estimate the impact of USDA's broadband loan and grant programs on the average payroll per worker using zip code level data from the Zip Code Business Patterns for the period from 1997 to 2007. Our results indicate that two of the smaller broadband programs (the Pilot loan program and the broadband grants program) likely had no effect on local payroll per workers. On the other hand, the largest program in terms of funding and coverage (the current broadband loan program) likely had a positive impact. Our estimate implies that a $1 per capita increase in a particular zip code's one-time receipt of the current program broadband loan results in a $0.92 increase in payroll per worker annually. Our calculated point estimates of the benefit: cost ratios for this broadband program range from 1.98 to 2.99, depending on assumptions about the time frame over which benefits accrue. However, the confidence intervals are wide enough to include the possibility that the costs outweigh the benefits.(JEL L86, J30, O18)     

Evaluating Scholarship Productivity in COAMFTE-Accredited PhD Programs

Due to an increasing trend among states to cut higher education funds, many universities are relying more on private donations and federal funding to keep programs afloat. Scholarship productivity in general has become an integral factor in terms of universities granting tenure to faculty, allocating resources, and supporting program goals due to the fact that more research in a particular area tends to increase the likelihood that one will obtain funding from federal, state, and private sources. In the past, ranking systems have also been used to evaluate programs. However, most ranking systems use methodologies that do not quantify research productivity or evaluate factors that match current university trends. The purpose of this article is to explore current scholarship productivity trends among COAMFTE-accredited doctoral programs through the use of several evaluation methods. Specifically, productivity was examined in regard to the following areas: (a) family therapy journal publications; (b) family science journal publications; (c) historic journal publication trends; and (d) recent journal publication trends.     

Connecting the dots: implementing and evaluating a network intervention to foster scientific collaboration and productivity

This article presents the design and implementation of a network intervention to foster scientific collaboration at a research university, and describes an experimental framework for rigorous evaluation of the intervention’s impact. Based on social network analysis of publication and grant data, an innovative type of research funding program was developed as a form of alteration of the university’s collaboration network. The intervention consisted in identifying research communities in the network and creating a new collaborative relation between pairs of unconnected researchers in selected communities. The new collaboration was created to maximally increase the overall cohesion of the target research community. In order to evaluate the impact of the program, we designed a randomized experiment with treatment and control communities based on the Rubin Causal Model approach. The paper describes the intervention design, reports findings from the program implementation, and discusses the statistical framework for future evaluation of the intervention.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Evaluation of community-based AIDS education and risk reduction projects in ethnic and racial minority communities: A survey of projects funded by the U.S. public health service

The purpose of this study was to describe proposed program evaluation methods used to determine effectiveness of selected federally funded AIDS education and risk reduction interventions focused on ethnic and racial minority populations. Funding for AIDS prevention programs has grown significantly since 1985, when the first allocation of $11 million was earmarked for general AIDS prevention and education. In 1987, funds specifically targeting minority populations were set aside from AIDS prevention and education programs. Nine agencies/divisions within the Department of Health and Human Services (DHHS) were identified as having funded AIDS prevention and education projects targeted toward ethnic and racial minority populations. The grants and contracts reviewed for this study were limited to those programs receiving Federal funding through DHHS. Few programs provided information within the broader context of sex education and drug education. The modification of needle-sharing behavior among IV drug users and unprotected sex with an IV drug user were the two most frequent risk factors targeted. All 63 programs used at least one formative evaluation method. The most frequently used formative techniques were monitoring achievement of project objectives and counting the number of people participating in program activities. Forty programs counted the amount of AIDS literature distributed. Thirty-three programs conducted a baseline assessment of AIDS knowledge, and 33 programs conducted a baseline assessment of risk behaviors for HIV infection. Results from this study support the National Academy of Sciences recommendation that while summative evaluation will ultimately be valuable, it would be premature to begin developing outcome evaluation strategies at the present time. It is incumbent upon program planners and evaluators to ensure that premature program evaluations are not made in an effort to eliminate direct federal funding of minority community-based strategies to stop the spread of HIV infection.     

Evaluation readiness: improved evaluation planning using a data inventory framework

Factors intrinsic to many programs, such as ambiguously stated objectives, inadequately defined performance measures, and incomplete or unreliable databases, often conspire to limit the evaluability of these programs. Current evaluation planning approaches are somewhat constrained in their ability to overcome these obstacles and to achieve full preparedness for evaluation. In this paper, the concept of evaluation readiness is introduced as a complement to other evaluation planning approaches, most notably that of evaluability assessment. The basic products of evaluation readiness--the formal program definition and the data inventory framework--are described, along with a guide for assuring more timely and appropriate evaluation response capability to support the decision making needs of program managers. The utility of evaluation readiness for program planning, as well as for effective management, is also discussed.     

Legislative history and evaluation

Legislative histories can be done with those evaluative studies which are based on programs which have a history in the legislature. ESEA Title I, Title IV, and the Title I Technical Assistance Centers are examples of programs which have their direct source in federal legislation. The evaluator could ask whether the intent (aims and objectives) of a program is the same as the intent of the legislation or could ask whether the outcomes of a program were the same as those intended by the legislature. The purpose of this paper is to introduce the reader to the doing of legislative history in an evaluation study. The paper introduces the reader to five major sources for legislative history. The paper then discusses seven acceptable sources of information. Following this is a five-step guide to doing a legislative history.     

The Longevity of Community Aging Initiatives: A Framework for Describing NORC Programs’ Sustainability Goals and Strategies

Naturally Occurring Retirement Community Supportive Services programs (NORC programs) aim to bring together stakeholders within a local area with a large number of older adults (e.g., a neighborhood or apartment building) to facilitate activities and services that promote aging in place. Despite enthusiasm for these programs, there is ongoing concern regarding their long-term sustainability, especially as federal funding to support NORC programs nationally has dwindled in recent years. This exploratory study aimed to develop a framework to describe NORC programs’ sustainability goals and strategies. As part of a statewide study in New Jersey, this study drew on data from semi-structured in-depth interviews with staff of 10 lead organizations that represented 15 NORC programs. Using a grounded theory analytic approach, findings revealed three types of sustainability goals: continuing the program indefinitely within the lead agency (described by eight organizations); integrating the program functions across other organizations (described by one organization); and defining the program as a finite project (described by one organization). Lead organizations implemented and developed their programs in ways that reflected their goals. This framework can be used to guide efforts to maintain, expand, and evaluate NORC programs and related community aging initiatives.     

Intergenerational Programs Review: Study Design and Characteristics of Intervention,Outcomes, and Effectiveness

There is a growing interest in intergenerational connectivity through intergenerational programs. In this work, a review of intergenerational programs was performed, with focus on the program design and objectives as well as in their outcomes.

We used a systematic review method in which we screened 3,796 articles. After analyzing titles, abstracts, and full paper analysis, 16 articles were retained. Each study was reviewed, and data were extracted related to target population, study design, characteristics of intervention, outcomes, and effectiveness.

Intergenerational programs included educational programs and art, Information technology development, cultural heritage, health education, and therapeutic activities. Most of the programs collected both quantitative and/or qualitative data. Seven studies collected data in the beginning and at the end of the program. Significant diversity in sample size and intervention length was found. Measurement of outcomes was performed in both young and/or elderly group of participants. Programs impact evaluation varied between studies, including validated scales, interviews, observation, focus groups, and conversation analysis, narratives, videotaped sessions, and field notes.

Our study highlighted the diversity in the design of studies and in the program’s effectiveness evaluation. More randomized design studies are required to support researchers and practitioners in the development of future intergenerational programs.     

Avoiding type III error in program evaluation : Results from a field experiment

A common, yet questionable assumption underlying many evaluations of service intervention programs is that program clients uniformly receive the services purportedly available. The authors draw upon the experience of a randomized field experiment to point out the hazards of that assumption. They found marked differences among clients in the amount of actual service received during participation in the program evaluated. Moreover, the data suggest that program outcomes varied as a function of the amount of service received. These findings are offered as a cautionary note to other evaluators; the amount of service actually received by clients should be accurately recorded and incorporated into the analyses of program outcomes.     

Program decisions : Evaluating a bird in the hand versus two in the bush

In new programs that have not reached full effectiveness, or in programs that are designed to produce future benefits (i.e., become “institutionalized”), evaluations that fail properly to assess future performance may bias decisions against the programs. For such programs, consideration of future benefits may significantly enhance the assessment of their cost-effectiveness. This paper describes an evaluation methodology based on decision theory that can take future benefits into account in evaluating a program. The methodology was applied to the Metropolitan Atlanta Crime Commission (MACC) Community Anti-Crime Project and indicated the potential for substantial future benefits subsequent to cessation of federal funding. The implications of use of this methodology and the MACC results for the overall Community Anti-Crime Program are also discussed.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.