Developing a methodology to assess the impact of research grant funding: A mixed methods approach

This paper discusses the development of a mixed methods approach to analyse research funding. Research policy has taken on an increasingly prominent role in the broader political scene, where research is seen as a critical factor in maintaining and improving growth, welfare and international competitiveness. This has motivated growing emphasis on the impacts of science funding, and how funding can best be designed to promote socio-economic progress. Meeting these demands for impact assessment involves a number of complex issues that are difficult to fully address in a single study or in the design of a single methodology. However, they point to some general principles that can be explored in methodological design. We draw on a recent evaluation of the impacts of research grant funding, discussing both key issues in developing a methodology for the analysis and subsequent results. The case of research grant funding, involving a complex mix of direct and intermediate effects that contribute to the overall impact of funding on research performance, illustrates the value of a mixed methods approach to provide a more robust and complete analysis of policy impacts. Reflections on the strengths and weaknesses of the methodology are used to examine refinements for future work.     

Households’ preferences for attributes of Conditional Cash Transfer programmes: A choice experiment in Ghana

Conditional cash transfer (CCT) is increasingly being integrated into poverty reduction strategies in developing countries. However, debate remains over key elements of CCT programme design. Using a discrete choice model, this article examines Ghanaian households’ preferences regarding key CCT programme design elements including conditionality, targeting, and payment method. The results revealed a preference for CCT designs that target beneficiaries with limited or no productive capacity and CCT designs that are conditional on beneficiaries either investing in children's human capital or performing communal service, relative to unconditional programmes. Also, direct bank deposit was revealed as the preferred payment mode relative to receipt of cash payments or use of mobile money systems.     

Disaster resilience: what it is and how it can engender a meaningful change in development policy

Disasters pose a growing threat to sustainable development. Disaster risk management efforts have largely failed to arrest the underlying drivers of increased global risk: uncontrolled urbanization and proliferation of assets in hazardous areas. Resilience provides an opportunity to confront the social‐ecological foundations of risk and development; yet it has been vaguely conceptualized, without offering a concrete approach to operationalization. We propose a conceptualization of disaster resilience centred on well‐being: ‘The ability of a system, community or society to pursue its social, ecological and economic development objectives, while managing its disaster risk over time in a mutually reinforcing way.’ We present a conceptual framework for understanding the interconnections between disasters and development, and outline how it is being operationalized in practice.     

Family support policies and child outcomes: a realist-scoping review*

Family support policies (FSP) refer to government policies that promote the well-being of families with children (e.g. job-protected paid leave, cash transfers, childcare). We developed an initial conceptual and theoretical framework of FSP and conducted a realist-scoping review to document the state of evidence regarding the influence of FSP on three child outcomes: poverty, development, and health. Based on the contexts and mechanisms through which FSP are hypothesized to affect child outcomes, we constructed a framework to guide a search of five electronic databases (OVID, ProQuest, PubMed, Scopus, and Web of Knowledge). We analysed articles meeting our inclusion criteria (i.e. conceptualized FSP as a function of the welfare state; conceptualized child poverty, development, and health as outcomes; and conducted empirical analyses) in relation to our mechanisms and thematic concerns (context, developmental periods, and differential impacts). A total of 22 studies met our inclusion criteria, which tested 25 independent child outcomes. Our findings identified initial mechanisms that explain the relationship between FSP and child outcomes through (1) increasing parents’ basic capabilities, (2) shaping parents’ childcare options, and (3) influencing parental leave-taking and shifting beliefs about gender relationships in the home and workplace. Future work will be to test these mechanisms and framework through a realist synthesis.     

Risk preferences after a typhoon: An artefactual field experiment with fishers in the Philippines

When are risk preferences stable and when do they change? In general, individual preferences tend to be consistent across time and space but extreme shocks, such as natural disasters, appear to change how people make economic decisions. We conduct an artefactual field experiment with fishers on a remote island in the Philippines and investigate the effect of Typhoon Bopha on risk preferences, along with fairness and time preferences. The typhoon destroyed coral reefs and reduced populations of fish, weakening food security. Comparing individuals from communities that were directly hit by the typhoon with those that were not, we observe evidence that those affected by the typhoon are less risk averse. Stratifying our sample by gender, we observe strong evidence that females affected by the typhoon are more risk-loving than females unaffected by the typhoon. These results are consistent with the hypothesis that extreme negative events cause individuals to be more willing to accept greater risk in return for a larger financial reward.     

Human research ethics committees and ethical review: The changing research culture for social workers

Social workers have a long-standing concern with developing codes of ethics, which govern and protect those involved in practice and research scenarios. As more social workers engage in research, successfully negotiating the ethical review process becomes imperative. In Australia, the task of ensuring ethical compliance of research protocols resides with human research ethics committees (HREC) mandated under the National Health and Medical Research Council. However, given the scope for discretion exercised by HREC, this process can be intimidating to novice and experienced researchers. The review process needs to be understood in the context of Australian ethics policy, the roles and responsibilities, and cultures of HREC. This paper outlines practical strategies in preparing ethics protocols to increase the likelihood of successful review. To do this, social workers need to recognise the way policy, legal developments and risk management strategies influence the ethical review process.     

Evaluating a community based mental health programme in West Bengal,India: Description of the methodology and lessons learned

Mental disorders impose an enormous burden on society. In developing countries like India, there is a lack of adequate number of trained mental health professionals to provide specialized care and 75–85 % of affected individuals do not have access to appropriate mental health services. The National Mental Health Programme (NMHP) is being implemented by the Government of India to support state governments in providing mental health services in the country. The Urban Mental Health Programme (UMHP) is a pilot initiative that has attempted the integration of mental health services in primary health care settings in two municipal wards in Kolkata, West Bengal, India. The overarching aim of this paper is to describe the methodology used for the evaluation of the community based mental health programme and to understand the processes of the programme in terms of barriers and facilitators. The current evaluation is based on a concurrent nested design, where qualitative and quantitative data are both collected at the same time but analysed separately and priority was given to qualitative data. This experience will contribute in helping other researchers to make some evaluations more effective, useful and manageable. Ethics approval was obtained from an institutional ethics committee of an organization (Ekjut) based in Ranchi, Jharkhand, India. The evaluation was undertaken by the George Institute for Global Health, New Delhi from February- June 2016.     

Informing family policies: The uses of social research

Like the topic of family policy itself, research informing family policies is difficult to characterize. This article discusses how ideology and values influence research agendas and then describes three types of studies informing family policies: research defining social issues, evaluation research, and research about the policy-making process. Two case studies illustrate how social research informs family policy: in promoting gender equality in Scandinavia and in reforming child support in the United States. Values of individualism and the sanctity of the family have traditionally focused policy makers' and, hence, researchers' attention on individuals, not families, as the units of analysis. But dramatic shifts in family structure and functioning along with renewed public concern about family disintegration are placing families high on the policy agenda. Both “basic” and “applied” family scholars can contribute to a research agenda examining the factors promoting strong, effective families. She conducts research on gender and the life course, as well as on aging, families, and social policy. She received her Ph.D. in Sociology from the University of Minnesota. Her research interests in stratification, social policy, gender, and the life course include sex segregation in occupations, fertility, and work decisions and family policy.     

Gender,conflict, and peace-building: how conflict can catalyse positive change for women

Although modern-day armed conflict is horrific for women, recent conflict and post-conflict periods have provided women with new platforms and opportunities to bring about change. The roles of women alter and expand during conflict as they participate in the struggles and take on more economic responsibilities and duties as heads of households. The trauma of the conflict experience also provides an opportunity for women to come together with a common agenda. In some contexts, these changes have led women to become activists, advocating for peace and long-term transformation in their societies. This article explores how women have seized on the opportunities available to them to drive this advocacy forward: including the establishment of an international framework on women, peace, and security that includes United Nations Security Council Resolution 1325 and other international agreements and commitments to involving women in post-conflict peace-building. The article is based on on-the-ground research and capacity-building activities carried out in the Great Lakes Region of Africa on the integration of international standards on gender equality and women's rights into post-conflict legal systems.     

Reassessing the Facebook experiment: critical thinking about the validity of Big Data research

The Facebook experiment of 2014 manipulated the contents of nearly 700,000 users’ News Feeds to induce changes in their emotions. This experiment was widely criticized on ethical grounds regarding informed consent. This controversy, however, diverted attention from a more important concern the experiment was intended to address, which is the impact of Facebook use on well-being. In this paper, I explore the well-being concerns raised by prior research and argue that the experiment does not alleviate them, owing to poor research design. As the question of Facebook's impact on well-being is of great importance, both to Facebook and to society overall, there is a pressing need for more experimental research that is both sensitive to informed consent and carefully designed to yield reliable results. In turn, the lessons of this case have implications for general issues of validity that emerge in Big Data research, now in vogue at major scientific venues.     

Listening to children: Evaluation of a positive parenting programme through art-based research

This paper presents data from a unique programme evaluation of the parenting programme titled ‘Learning together, growing as a family’ applied in 14 cities in Spain and targeting families at risk of neglectful behaviour. The programme evaluation is based on the voices of children using the qualitative methodology of art-based research. Eighty-six children 6–12 years of age were interviewed in groups. The outcomes of the evaluation reveal that children perceive improvements in the parental competencies of their parents and in themselves and that these changes serve as a catalyst for positive family change.     

Decolonising research in a Sub-Saharan African context: exploring Ubuntu as a foundation for research methodology,ethics and agenda

In all parts of the world, researchers are addressing the colonial legacy of research. This article aims to contribute to the decolonisation of research in a sub-Saharan African context by exploring Ubuntu as an indigenous Southern African research paradigm. Drawing on lessons learnt from participatory action research with South African science teachers and on Ubuntu research literature, I develop, and reflect on, characteristics of an Ubuntu research ethics, agenda and methodology. Understood as humanness, Ubuntu encompasses a dimension of becoming human and being human. Both dimensions are realised through lived community and respectful, caring relations with other living beings and the environment. Thus, ethical protocols evolve around relating positively to others. Ubuntu research agendas contribute to strengthening community and methodologies are community based, relational and participatory. The emphasis of the article is not on presenting Ubuntu research as categorically oppositional to conventional methodologies, but on an approach to research that is grounded in indigenous African epistemologies.     

The process of evaluating a capacity-building support initiative for HIV positive South Africans

This paper describes the process of evaluating a capacity-building initiative involving HIV positive South Africans. The evaluation was conducted within a participatory evaluation framework, which aims at empowering participants through social action. The rationale is that an understanding and acceptance of evaluation results promotes commitment to act. Forty four members of the Kudu Support Group and 23 non-members participated in the evaluation. Methods included in-depth interviews, a survey, repeated measures of the General Health Questionnaire-28 (GHQ-28), a SWOT analysis, interviews with health workers, field notes, and participatory observation. The capacity-building initiative resulted in empowerment through paid employment for support group members, the establishment of a vegetable garden, increased skills and efficacy, access to resources and networks, a perceived reduction of stigma, perceived improvement of social relations, and improved scores on the GHQ-28. Lessons that emerge out of this study are the importance of integrating evaluation as a part of ongoing activities throughout the research project; to ensure that methods and results are understandable and useful to for the participants; that the evaluator has an in-depth understanding of the project and context; and to introduce frequent and simple internal evaluation tools to guide efforts to improve community-based groups’ functioning and activities.     

Recognition as a framework for ethical participatory research: developing a methodology with looked after young people

This paper reflects on a project which involved the authors working in partnership with a working group of looked after children and young people established by a small unitary local authority to develop a pledge defining what support, services and care looked after children can expect from them. The authors aimed to develop an approach which ensured that the young people’s voices were heard and also that the well-being of those involved as participants remained at the centre of the process throughout. We argue that Honneth’s theory of recognition, which identifies three forms of recognition as important to ensure that human beings feel assured of their dignity or integrity – in brief love, rights and solidarity – offers a useful framework for achieving this. The main principles which emerged were: (i) building the research around the young people’s existing relationships, (ii) respecting the group’s decisions and (iii) honouring the views expressed.     

Oxford house: a review of research and implications for substance abuse recovery and community research

After treatment for substance abuse, whether it is in hospital-based treatment programs, therapeutic communities, or recovery homes, many patients return to former high-risk environments or stressful family situations. Returning to these settings without a network of people to support abstinence increases chances of a relapse. As a consequence, substance abuse recidivism following treatment is high for both men and women. Alternative approaches need to be explored, and there are some promising types of recovery homes. From a public health perspective, a series of studies conducted at DePaul University suggests that one type of recovery home for alcohol abuse recovery has much potential. For example, within this self-help communal living setting, recovering alcoholics were able to maintain employment, thereby reducing their need for government subsidies. Maintaining employment for recovering alcoholics may promote increased personal responsibility, which may impact self-efficacy beliefs. These pilot studies, then, raised both theoretical and practical issues needing further evaluation.     

Home for now: A mixed-methods evaluation of a short-term housing support program for homeless families

The use of short-term rental subsidy vouchers offers a new approach to addressing the housing needs of families facing homelessness. In Massachusetts, the Family Home pilot program placed homeless families in housing instead of shelter, providing two years of rental subsidy plus support services with the goal of enabling families to maintain market rate housing. This mixed-method case study complements staff and participant interview data with participant survey and administrative data to evaluate the implementation and short-term outcomes of Family Home in one region. Data point to improved family well-being in housing but also persistent barriers to achieving longer-term housing and economic stability. Of the families who had exited the program at the end of the study, one quarter were able to retain their housing at market rate, only 9% returned to shelter, and one in five moved in with families/friends. Lack of affordable housing in a high rental cost region and jobs that pay living wages were among the major reasons that families struggled to maintain housing. This research points to the need for integrating supportive services from the program's start, including targeted workforce development, to plan for the end of the short-term rental subsidy.     

Ethics of resource allocation: instruments for rational decision making in support of a sustainable health care

Objective: In all western countries health care budgets are under considerable constraint and therefore a reflection process has started on how to gain the most health benefit for the population within limited resource boundaries. The field of “ethics of resource allocation” has evolved only recently in order to bring some objectivity and rationality in the discussion. In this article it is argued that priority setting is the prerequisite of ethical resource allocation and that for purposes of operationalization, instruments such as “need assessment” and “health technology assessment (HTA)” are essential worktools for making more rational decisions. Thresholds (deduced from the need assessment and HTA) are—within this context—guiding but not binding principles. Method: Discussion of theoretical concepts of not only priority setting, need assessment and HTA complemented by practical examples for showing the challenges and the need, but also the chances of a more explicit and transparent policy of resource allocation in health care. Results: Priority setting in health care is based on the values of equity, justice and solidarity. Health packages decisions are determined from medical need (the severity of the condition) and/or the appropriateness of medical interventions (their cost-effectiveness). With growing awareness that originally effective and cost-effective services and programmes are eventually provided inappropriately, the focus is shifting towards the organisational aspects of provision and application. Therefore, need assessment is based on the distinction of health care needs from demand, supply, or actual care. Additionally HTA provides the evidence on health care interventions in a way that it becomes obvious who benefits from an intervention and who definitely does not benefit, but eventually is harmed. Conclusions: Health services research on effective and cost-effective interventions and research/monitoring of performance that the effective and cost-effective services are provided appropriately are of increasing importance for guiding the decision-making process on priority setting and need assessment. Effective healthcare for all is sustainable, if we start to put expenditures in perspective and focus health policies and research strategies on managing expectations through patient information and a more realistic notion of medical advancements and, on the other hand, on encouraging need-based and cost-effective innovations.

---

Zusammenfassung Ziel: In allen Ländern des Westens ist die Haushaltslage im Gesundheitswesen heute äußerst angespannt, weshalb nun ein Reflexionsprozess begonnen hat, wie mit begrenzten Haushaltsmitteln der größtmögliche gesundheitliche Nutzen für die Bevölkerung erzielt werden kann. Erst in jüngster Zeit ist die “Ethik der Mittelzuweisung” als Arbeitsgebiet entwickelt worden, mit dem Ziel, eine gewisse Objektivität und Rationalität in die Diskussion zu bringen. In diesem Artikel argumentieren wir, dass eine ethische Mittelzuweisung Prioritätssetzungen erfordert und dass Instrumente wie Bedarfeinschätzung (Need Assessment) und Bewertung medizinischer Verfahren [Health Technology Assessment (HTA)] wesentliche Arbeitswerkzeuge zur Erzielung rationaler Entscheidungen sind. Schwellenwerte (abgeleitet aus Need Assessment und HTA) dienen in diesem Zusammenhang als leitende, jedoch nicht bindende Prinzipien. Methode: Diskussion theoretischer Begriffe der Prioritätssetzung, Bedarfeinschätzung und HTA, ergänzt durch praktische Beispiele zur Darstellung der Herausforderungen des Bedarfs nach, aber auch der Chancen für eine explizitere und transparentere Politik der Mittelzuweisung im Gesundheitswesen. Ergebnisse: Prioritätssetzung im Gesundheitswesen basiert auf den Werten Gleichheit, Gerechtigkeit und Solidarität. Entscheidungen über Versorgungspakete richten sich nach der medizinischen Bedarfslage (Schwere der Erkrankung) und/oder der Angemessenheit medizinischer Eingriffe (Kosteneffektivität). Mit der wachsenden Erkenntnis, dass ursprünglich effektive und kosteneffektive Leistungen und Programme am Ende unangemessen bereitgestellt werden, richtet sich die Aufmerksamkeit mehr auf Organisationsaspekte der Bereitstellung und Anwendung. Need Assessment basiert daher auf der Unterscheidung zwischen Bedarf im Gesundheitswesen und Nachfrage, Angebot oder tatsächlicher Versorgung. HTA fasst dann die Evidenz zu Eingriffen seitens des Gesundheitswesens zusammen, sodass offenbar wird, wem ein Eingriff nützt und wem er mit Sicherheit nicht nützt, sondern am Ende schadet. Schlussfolgerungen: Forschungen der Gesundheitsdienste zu effektiven und kosteneffektiven Eingriffen sowie die Erforschung/Erfolgskontrolle einer angemessenen Bereitstellung effektiver und kosteneffektiver Dienste sind von wachsender Bedeutung für die Information des Entscheidungsprozesses zur Prioritätssetzung und Bedarfseinschätzung. Eine effektive Gesundheitsversorgung für alle bleibt tragbar, sofern wir beginnen, Ausgaben in Perspektive zu setzen und gesundheitspolitische Entscheidungen und Forschungsstrategien darauf ausrichten, Erwartungen zu lenken, indem wir die Patienten informieren und eine realistischere Einschätzung medizinischer Fortschritte fördern, und uns zugleich auf bedarfsorientierte und kosteneffektive Innovationen konzentrieren.

---

Résumé Objectif : Dans tous les pays occidentaux, les budgets de santé publique sont soumis à de fortes contraintes, ce pourquoi un processus de réflexion a commencé sur la question de savoir comment obtenir avec des moyens limités la plus grande utilité possible pour la population en termes de santé. Ce n’est que récemment que « l’éthique de l’allocation des ressources » a été développée pour apporter une certaine part d’objectivité et de rationalisme dans la discussion. Le présent article soutient que l’attribution de priorités est la condition préalable indispensable à l’allocation des ressources et que des instruments tels que l’évaluation des besoins (need assessment) et l’évaluation des technologies de la santé (ETS/HTA) sont des outils essentiels pour parvenir à des décisions plus rationnelles. Les valeurs seuils (déduites de l’évaluation des besoins et de l’ETS) servent dans ce contexte de principes directeurs, mais non contraignants. Méthode : Discussion sur les notions théoriques d’attribution de priorités, d’évaluation des besoins et d’ETS, complétée par des exemples pratiques mettant en évidence les défis et les besoins, mais aussi les chances d’une politique d’allocation des ressources de santé plus explicite et transparente. Résultats : L’attribution des priorités dans le domaine de la santé se base sur les valeurs d’équité, de justice et de solidarité. Les décisions relatives à des ensembles thérapeutiques sont déterminées par le besoin médical (gravité de la maladie) et/ou le caractère opportun des interventions médicales (efficacité des coûts). Avec la prise de conscience que des services et des programmes initialement opérationnels et efficaces en termes de coûts sont appliqués de manière inappropriée, l’attention se porte sur les aspects organisationnels de l’attribution et de l’application. L’évaluation des besoins se base par conséquent sur la distinction entre les besoins de santé provenant de la demande, de l’offre et de l’approvisionnement effectivement fourni. L’évaluation des technologies de la santé procède à une synthèse des preuves faisant apparaître qu’une intervention de santé est utile à un patient, lui est en toute certitude inutile, voire préjudiciable. Conclusions : Les recherches des services de santé sur les interventions efficaces en termes de thérapie et de coûts, de même que la recherche/le suivi des performances sur une fourniture appropriée de services efficaces en termes de thérapie et de coûts revêtent une importance croissante pour guider le processus décisionnel sur l’attribution des priorités et l’évaluation des besoins. Une couverture de santé efficace pour tous n’est viable que si nous commençons à mettre les dépenses en perspective et à concentrer les politiques de la santé et les stratégies de recherche sur la gestion des attentes, en informant les patients et en promouvant une estimation plus réaliste des progrès médicaux, et d’autre part en encourageant les innovations fondées sur les besoins et sur le rendement des coûts.

     

The context of employment discrimination: interpreting the findings of a field experiment

Although field experiments have documented the contemporary relevance of discrimination in employment, theories developed to explain the dynamics of differential treatment cannot account for differences across organizational and institutional contexts. In this article, I address this shortcoming by presenting the main empirical findings from a multi‐method research project, in which a field experiment of ethnic discrimination in the Norwegian labour market was complemented with forty‐two in‐depth interviews with employers who were observed in the first stage of the study. While the experimental data support earlier findings in documenting that ethnic discrimination indeed takes place, the qualitative material suggests that theorizing in the field experiment literature have been too concerned with individual and intra‐psychic explanations. Discriminatory outcomes in employment processes seems to be more dependent on contextual factors such as the number of applications received, whether requirements are specified, and the degree to which recruitment procedures are formalized. I argue that different contexts of employment provide different opportunity structures for discrimination, a finding with important theoretical and methodological implications.     

Child maltreatment in Greece: a review of research

Child maltreatment can be traced in Greek mythology, history and literature. Greek society has always used passive or active violence towards its dependent members, including children. In the last 17 years there have been a number of studies describing child maltreatment in contemporary Greece. Most have used small clinical samples; nevertheless, their findings are comparable to those of other countries and their results support the psychopathological causality of child maltreatment. A number of studies on child-rearing practices and on children's socialization have revealed an extensive use of physical punishment related to the child's opposition to parental authority and high parental expectations often linked with the child's school performance. Another small set of studies assessed the effectiveness of social policy and the appropriateness of the child protection system. On the basis of the studies on child maltreatment in Greece, the author proposes firstly the adoption of an operational definition which takes into consideration community attitudes and culture. A second proposal relates to the application of an ecological theoretical model stressing the mutual interaction of different systems and their cumulative effects. Lastly, it is suggested that social policy measures should be introduced in the field of primary, secondary and tertiary prevention. © 1997 John Wiley & Sons, Ltd.     

Using mixed method approach in measuring effects of training in firms: Case study of the European Social Fund support

Public support of training in firms corresponds to the long-term importance of the quality of human capital in the competitiveness of firms and nations. Thus, the EU supports such training via the European Social Fund (ESF). The evaluation community evaluates the support by using either qualitative or quantitative methods. The simultaneous application of these two approaches is rare. The purpose of this paper is to combine quantitative (counterfactual impact evaluation) and qualitative (qualitative comparative analysis) methods in order to fill the methodological gap. Based on the combination of both approaches, it explores their strengths, complementarity and disadvantages to evaluate public support for employee training in the Czech Republic. The combination of methods makes it possible to identify not only the impacts but also their causes. Linking the ESF support to corporate competitiveness is crucial for demonstrating the effectiveness of public spending.