Developing a methodology to assess the impact of research grant funding: A mixed methods approach

This paper discusses the development of a mixed methods approach to analyse research funding. Research policy has taken on an increasingly prominent role in the broader political scene, where research is seen as a critical factor in maintaining and improving growth, welfare and international competitiveness. This has motivated growing emphasis on the impacts of science funding, and how funding can best be designed to promote socio-economic progress. Meeting these demands for impact assessment involves a number of complex issues that are difficult to fully address in a single study or in the design of a single methodology. However, they point to some general principles that can be explored in methodological design. We draw on a recent evaluation of the impacts of research grant funding, discussing both key issues in developing a methodology for the analysis and subsequent results. The case of research grant funding, involving a complex mix of direct and intermediate effects that contribute to the overall impact of funding on research performance, illustrates the value of a mixed methods approach to provide a more robust and complete analysis of policy impacts. Reflections on the strengths and weaknesses of the methodology are used to examine refinements for future work.     

A case study of action research in probation : The improved correctional field services project

This case study of the Improved Correctional Field Services Project describes the pre-implementation history of an action research project—in this case a project to improve probation services. That history encompasses a period of about five years (1974–1978). ICFS represented a major initiative in probation programming by the now defunct Law Enforcement Assistance Administration. It was intended to be a test of some of the principle ideas for making probation more effective. Its history illustrates the pitfalls that may confront a program developer and evaluator. Further, it illustrates how concepts and objectives can change within the life-time of a given project.     

Evaluating health services research capacity building programs: Implications for health services and human resource development

Evaluation of research capacity building programs presents challenges, including defining “research capacity building” and measuring it. The present study evaluates infrastructure capacity-building strategies and barriers and facilitators of two federally funded health-related research capacity building programs within the United States. We developed a conceptual framework and abstracted 223 data elements from existing project files. We administered questionnaires to 17 principal investigators and conducted 2-h interviews of 15 principal investigators. Finally, we conducted one-and-a-half-day site visits with five projects. We aggregated data and analysis results across the two programs. Impacts and strategies were also analyzed as to whether host institutions had little or no existing research infrastructure (“seed projects”) or had well-developed research infrastructure (“fertilizer projects”). Projects were associated with increases in publication and presentation rates by Principal Investigators and staff, increases in research funding, changes in organizational research culture, and positive impacts on the community. Project characteristics and activities associated with each of these outcomes were identified. It was also noted that the correlations between these outcomes were small and non-significant.     

Geographically-related outcomes of U.S. funding for small business research and development: Results of the research grant programs of a component of the National Institutes of Health

This article examines the geographic distribution of funding for the U.S. Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs sponsored by the National Institute of General Medical Sciences (NIGMS). Despite a significant investment in SBIR/STTR and an interest in increasing geographic diversity in the institute’s research portfolio, there has not been an assessment of the distribution of NIGMS’s SBIR/STTR funding, outcomes associated with that investment, and relationships between the two. The geographic distribution of NIGMS’ SBIR/STTR funding was highly concentrated in a small number of states, with a high correlation between each state’s funding and its number of small scientific research and development businesses. Affiliation with a major research university was correlated with several measures of innovation and firm success. Our findings are consistent with earlier research showing that economic activity in research and development and research output tend to cluster in geographic regions where knowledge can be generated and shared more efficiently. These findings lend support to an investment strategy for small business research and development that creates networks between major research universities and small businesses.     

Professional development laboratory: center for literacy and community services

We developed three Professional Development Laboratories: Centers for Literacy and Community Services that addressed four pressing needs in the current urban school crisis: the preparation of in-service teachers to be literacy specialists, the dire need for qualified teachers in urban settings, the professional development of uncertified urban teachers, and the literacy needs of the children and families these teachers serve. These centers were housed in three inner-city schools rather than at the university and thus were able to serve larger numbers of children. Since many of the teachers in these schools are novice or uncertified teachers, we offered on-site in-service courses in order to enhance their ability to help their students. By housing the Centers for Literacy and Community Services on site in the schools, we sought to reach students and families who might otherwise not be able to avail themselves of these services. Unique to this project was that our graduate literacy education students tutored students on site in schools, the university faculty worked alongside the graduate students, and both students and faculty provided services to enhance the professional development of teachers and administrators in the area of literacy. Additionally, this project was distinguished by the inclusion of a component designed to provide literacy education services to the families and caregivers of the children.     

Enhancing participatory research with the community oriented primary care model: A case study in community mobilization

Refining Participatory Research with the methodology of Community Oriented Primary Care (COPC), sociologists, medical providers and other “professionals” were able to assist the people of a small, rural county in western Appalachian North Carolina create a “community health consortium.” The consortium grew out of a research process funded by the W.K. Kellogg Foundation, which included some 40 interviews with community groups that articulated health problems and barriers to health care. The consortium then undertook action projects, which in turn enhanced networking between community groups and agencies while reducing mistrust between them. The consortium empowered people to articulate their realities, needs and became a force for change able to make demands on political office holders and decision makers. Despite its democratic origins and ideology, the consortium needs to do more to extend its outreach to include the marginal and disenfranchised.     

Staffing,retention, and government funding: A case study

This study analyzes the implications of government‐contract funding on the staffing pattern of a nonprofit agency, the Canadian Red Cross, Toronto Region. Furthermore, the study explains the implications of the staffing pattern on services and on the agency's organization. Staffing is one area that the literature on nonprofit organizations has not adequately addressed. The findings indicate that contract‐based funding leads to the hiring of temporary staff and affects the retention of employees. Although contract funding has some benefits, temporary staffing is detrimental to the agency's services. It affects not only employee recruitment and retention but also training. The results highlight how change in government funding from grants to contracts resulted in the agency's new staffing strategy and ultimately reduced the effectiveness of the services the agency provided to the community.     

First Nations and community economic development: a case study

A training project in a northern Canadian community providedan opportunity to examine participatory planning approachesand the meaning of work in First Nations communities. Focusgroups conducted three years after the unsuccessful interventionof a community economic development (CED) project suggest thatcomplex factors such as lack of support from community leadersand rate of pay for workers determine whether CED is alwaysappropriate in northern, First Nations contexts.     

Case management for child protection services: A multi-level evaluation study

This article presents an evaluation study of a case management method for child protection services, the Delta Method for Family Supervision, in terms of supervision order duration and occurrence and duration of out-of-home placements. Additionally, case and case manager characteristics were examined. Data was collected about 224 cases, 58 case managers and 30 team managers of all 15 offices of the Child and Youth Protection Services in the Netherlands. In all cases the Delta Method was applied. Data were obtained by interviews, questionnaires and case files. Multi-level analysis was performed to study the influence of independent variables on supervision order duration, and the occurrence and duration of out-of-home placements. Case characteristics related to 87% of the differences in the duration of supervision order, case manager characteristics to 13% of the differences. Some case manager characteristics about applying the Delta Method were significantly related to shorter duration of the supervision order and the occurrence and duration of out-of-home placement. Case characteristics also showed strong relations. Together with the more general aspects of case management supported by this study, such as a one family and one worker approach, this contributes to a more effective practice of case management for child protection services.     

The community development potential of large health services organizations

Large health services organizations routinely support communitiesthrough provision of services. However, as these organizationshave also become important economic entities, they are increasinglyviewed as having significant potential to make additional contributionsto community development in a variety of ways. The incentivesfor large health services organizations to play broader communitydevelopment roles include the same business advantages thataccrue to other organizations that do so. Beyond this, uniquely,health services organizations can more fully fulfill their centralhealth-enhancing purposes through their community developmentefforts because, ultimately, the health of people is affectedby the quality of the communities in which they live.     

The contribution of research to community development

In this article I consider the relationship between differenttypes and methods of conducting research in the context of communitydevelopment. In particular I will consider the degree to whichdifferent approaches contribute to core objectives of communitydevelopment to empower communities and promote social justiceby engaging the most excluded. The contents will draw on a dialoguewithin my own agency, the Scottish Community Development Centre(SCDC).     

Goal-oriented evaluation for program planning: A case study

Utilizing a contextual model of evaluation, a goal-oriented method was applied to the Health Psychology program, a doctoral program in its early stages at the University of California, San Francisco. There were five stages involved in implementing this method: (1) clarification of the goals and objectives of the program, (2) prioritizing the objectives, (3) judging the attainment of the objectives, (4) organization of faculty/student input, and (5) feedback to the program management. All faculty members and students were invited to participate as selfevaluators in this evaluation effort. The results indicated that there were significant differences between the faculty group and the student group on their ranking and rating of the importance of specific educational and resource objectives. A one year follow-up was obtained by interviewing the director of the program to assess the impact of the project on program planning. The advantages and disadvantages of the approach were discussed in light of this attempt to analyze a new educational program.     

The manifestations of politics in evaluation: An exploratory study across the evaluation process

Evaluation has been described as a political act. Programs and policies are generated from a political process, and the decision to evaluate and how to use the evaluation are manifestations of the political dynamic. This exploratory study was conducted with practicing evaluators to understand what they view as political situations in the evaluation process and how they responded to these situations. Findings suggest that, in relation to the potential evaluation phases in which each respondent has been involved, evaluations are susceptible to politics when initially attempting to identify stakeholders and when it’s time to report the evaluation findings. Evaluators have also developed multiple strategies for dealing with these situations, including finding allies for the evaluation and working to explain the evaluation process and its implications. We hope that this study will help to inform novice and expert evaluators about the various political situations they may encounter in their practice.     

Interlakes community health centers: A case study

Interlakes Community Health Centers (ICHC) provides primary health care services to low-income people in a medium-size city. ICHC expanded from three to five service centers and soon ran into problems meeting payroll. Several strategic management dilemmas stem from this crisis.     

Midwifery in Colorado: A case study in the politics of professionalization

This paper uses documentary evidence, interviews, and participant observation to examine the professionalizing activities of lay midwives in Colorado. It shows that professionalization for midwives is primarily a political process. In order to gain state recognition and professional autonomy, lay midwives were forced into the political arena, where they encountered resistance from economic competitors. I argue that medical dominance over childbirth practices and professional segmentation between lay midwives and certified nurse midwives were the primary reasons Colorado lay midwives failed to achieve professional status.I would thank the Colorado midwives for their cooperation in this study. My thanks also to the reviewers for their comments on earlier drafts.     

The case study evaluation: A means for managing organizational and political tensions

This study of evaluation utilization identified organizational, political, and practical arrangements which facilitated wide use of The Interim Report Evaluation in policy making for California's Early Childhood Education program. In a program fraught by tensions, where evaluations had been political tools, this evaluation was special. The research used a field study approach. Analysis was guided by literature on organizations, policymaking, and evaluation utilization. It identified techniques for maintaining political support, marshalling organizational resources, and designing and disseminating an evaluation that used an ethnographic approach and that was directly applied to policy deliberations. Such techniques have significance when evaluators and researchers need to convince policymakers of their worth. This case study adds to knowledge of the research/policy intersect, on ethnographic evaluation, and on state education policymaking.     

The ethics of deception in social research: A case study

Using data from a study on courtship through personal advertisements, I argue that Kai Erikson's classic case against disguised observation is flawed. Certain kinds of deception are necessary to gather certain data in certain settings. I placed bogus ads in a personal column to obtain and analyze responses. The data would have remained inaccessible—indeed, many of the responses would not have existed in the first place—without some measure of deception. While deception was used, no risk whatsoever was posed to respondents. I further argue that several of Erikson's criteria of risk do not separate ethical from empirical questions; informants use very different criteria in evaluating the risk of harm to them posed by social research that sociologists use. The question of exploitation is more complex, since it has to be weighed against how much of an effort my respondents made and hence, what it is exactly that I took from them. A “panel of judges” decided that most of my male (but not my female) respondents would not have gotten dates with my hypothetical ad placers, and that the research method I used was not especially unethical.