The use of concept mapping in measurement development and evaluation: Application and future directions

The past decade has seen an increase of measurement development research in social and health sciences that featured the use of concept mapping as a core technique. The purpose, application, and utility of concept mapping have varied across this emerging literature. Despite the variety of uses and range of outputs, little has been done to critically review how researchers have approached the application of concept mapping in the measurement development and evaluation process. This article focuses on a review of the current state of practice regarding the use of concept mapping as methodological tool in this process. We systematically reviewed 23 scale or measure development and evaluation studies, and detail the application of concept mapping in the context of traditional measurement development and psychometric testing processes. Although several limitations surfaced, we found several strengths in the contemporary application of the method. We determined concept mapping provides (a) a solid method for establishing content validity, (b) facilitates researcher decision-making, (c) insight into target population perspectives that are integrated a priori, and (d) a foundation for analytical and interpretative choices. Based on these results, we outline how concept mapping can be situated in the measurement development and evaluation processes for new instrumentation.     

Engaging children and youth in research and evaluation using group concept mapping

Group concept mapping is a systematic methodology that recognizes participants’ expertise in their own experiences. In this approach, participants share their diverse knowledge and interpret the data generated by all group members. This approach uniquely blends quantitative and qualitative techniques to create illustrative, structured conceptualizations. In this article, we describe the steps in group concept mapping and report feedback on this method from 37 children and youth who participated in group concept mapping studies. We recommend group concept mapping as an engaging way to involve children and youth in research and evaluation activities.     

Extending systems thinking in planning and evaluation using group concept mapping and system dynamics to tackle complex problems

Group concept mapping (GCM) has been successfully employed in program planning and evaluation for over 25 years. The broader set of systems thinking methodologies (of which GCM is one), have only recently found their way into the field. We present an overview of systems thinking emerging from a system dynamics (SD) perspective, and illustrate the potential synergy between GCM and SD. As with GCM, participatory processes are frequently employed when building SD models; however, it can be challenging to engage a large and diverse group of stakeholders in the iterative cycles of divergent thinking and consensus building required, while maintaining a broad perspective on the issue being studied. GCM provides a compelling resource for overcoming this challenge, by richly engaging a diverse set of stakeholders in broad exploration, structuring, and prioritization. SD provides an opportunity to extend GCM findings by embedding constructs in a testable hypothesis (SD model) describing how system structure and changes in constructs affect outcomes over time. SD can be used to simulate the hypothesized dynamics inherent in GCM concept maps. We illustrate the potential of the marriage of these methodologies in a case study of BECOMING, a federally-funded program aimed at strengthening the cross-sector system of care for youth with severe emotional disturbances.     

From plan to action: Adapting evaluation to serve the developmental needs of a newly-funded multidisciplinary research center

The Southeastern Coastal Center for Agricultural Health and Safety (SCCAHS) is one of many newly-funded federal research centers, housing five multidisciplinary research projects and seven pilot projects, and serving a multi-state region. In the early stages of such a complex project, with multiple teams separated by geography and disciplines, the evaluation program has been integral in connecting internal and external stakeholders at the center and project levels. We used a developmental evaluation (DE) framework to respond to the complex political environment surrounding agricultural health and safety in the southeast; to engage external stakeholders in guiding the center’s research and outreach trajectories; to support center research teams in a co-creation process to develop logic models and tailored indicators; and to provide timely and feedback within the center to address communications gaps identified by the evaluation program. By using DE principles to shape monitoring and evaluation approaches, our evaluation program has adapted to the dynamic circumstances presented as our center’s progress has been translated from a plan in a grant proposal to implementation.     

Using developmental evaluation as a system of organizational learning: An example from San Francisco

In the last 20 years, developmental evaluation has emerged as a promising approach to support organizational learning in emergent social programs. Through a continuous system of inquiry, reflection, and application of knowledge, developmental evaluation serves as a system of tools, methods, and guiding principles intended to support constructive organizational learning. However, missing from the developmental evaluation literature is a nuanced framework to guide evaluators in how to elevate the organizational practices and concepts most relevant for emergent programs. In this article, we describe and reflect on work we did to develop, pilot, and refine an integrated pilot framework. Drawing on established developmental evaluation inquiry frameworks and incorporating lessons learned from applying the pilot framework, we put forward the Evaluation-led Learning framework to help fill that gap and encourage others to implement and refine it. We posit that without explicitly incorporating the assessments at the foundation of the Evaluation-led Learning framework, developmental evaluation’s ability to affect organizational learning in productive ways will likely be haphazard and limited.     

Partnership evaluation: An application of a developmental framework to the Governance and Local Development project in Senegal

This study explores the valued-added of USAID/Senegal’s Governance for Local Development (GOLD) project’s integrated approach to working with four sector projects: two in health, one in water and sanitation, and one in nutrition. The study builds on a partnership evaluation framework that identifies practices related to preconditions, factors linked with success, implementation structures and processes, contribution of collaboration to performance, and collaboration outcomes. A survey instrument developed statements of positive practices and asked respondents to agree or disagree with each one. The analysis includes exploration of differences in perception. Respondents generally expressed agreement with the statements in the survey instrument associated with the various dimensions of effective partnerships. All projects mentioned the presence of partnership champions. Problems with joint action (e.g., planning, budgeting, managing) were the most frequently cited constraints. The study confirms most of the hypotheses embedded in the evaluation framework regarding partnership’s potential for creating synergies and improved outcomes and affirms the framework’s usefulness as a developmental evaluation tool. Positive outcomes result from intentional efforts to build relationships, cultivate champions, and design systems and processes that support a partnership culture and contribute to each partners’ success as well as the success of the partnership itself.     

Quality and rigor of the concept mapping methodology: a pooled study analysis

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.     

Using developmental evaluation and community-based participatory research to develop a model of supportive housing

Community-based participatory research (CBPR) and developmental evaluation (DE) have emerged over recent decades as separate approaches for addressing complex social issues. Current literature offers little with respect to the use of CBPR and DE in combination, although the two approaches are complementary. Through the current paper, we outline how CBPR and DE were used to develop a model of supportive housing for teen families. More specifically, we describe the structures and processes that contributed to this development, including (1) our partnership approach, (2) pooled resources, (3) regular opportunities for collaboration and reflection, (4) integration of multiple data sources, (5) ongoing feedback and knowledge dissemination, and (6) adjustments to program practices. We end by providing insights into the lessons that we learned through this project. Through this paper, we describe how researchers and community partners can collaboratively use CBPR and DE to develop a program model in complex community settings. Insights are offered that will be important for researchers, evaluators, and practitioners seeking to develop programming in response to complex community issues.     

Using Concept Mapping to Develop a Statewide Kinship Care Coalition: A Case Study

Increasingly, community coalitions are being developed to address a myriad of social challenges. However, documented research efforts related to planning processes associated with coalition formation are few. This case-study documents the use of concept mapping (CM) to d elineate a conceptual framework for developing a statewide kinship care coalition in one southeastern state. CM is a mixed-method research approach that analyzes qualitative data via nonmetric multidimensional scaling and hierarchical cluster analyses. After a review of relevant literature, this article discusses CM processes associated with this endeavor and examines implications for future use of CM for coalition formation and development.     

Piloting online WellnessRx learning modules: Demonstration of developmental evaluation

WellnessRx is a health initiative focusing on healthy living through education, knowledge translation, and community engagement. Stakeholders of WellnessRx identified web-based education learning modules on nutrition and physical education as a priority to be integrated into existing health sciences curricula, as well as adapted for use by health professionals. Five learning modules were created with essential knowledge, skills, attitudes and resources or tools for health professional students and practitioners. As part of the ‘developmental evaluation framework’ for WellnessRx, two of these modules were piloted within two health professional student programs. This paper describes the pilot-evaluation experience involving student surveys, focus groups and interviews, and faculty perspectives. For both modules, student pre-post knowledge assessments indicated some improvements in post-module knowledge. Post module evaluations by students indicated benefits with the online delivery being flexible for access, self-health, case-based assessments and useful nutrition and physical activity guides. Challenges for students included their time to do the modules and the activity expectations. Instructors felt each module could be better targeted to different years within an undergraduate program. Through developmental evaluation, the pilot results along with recommendations and lessons learned provided the direction needed to further develop the WellnessRx logic model and proposed learning modules.     

Using concept mapping as a planning tool: Child welfare citizen review panels

Citizen Review Panels (CRPs) are groups of citizen volunteers authorized by U.S. federal law to examine state child welfare agencies. These groups inspect policies and practices related to child protection responsibilities and are tasked with making recommendations for systemic improvement. Despite the federal mandate for each state to develop a CRP and the potential of these groups to positively impact child welfare practices, there is a dearth in the literature related to CRPs. Consequently, planning and evaluation processes of these groups vary widely. This study reports on the use of concept mapping (CM) to outline a framework for planning and subsequently evaluating the CRP in one southeastern state. CM is a mixed-method research approach that uses multi-dimensional scaling and hierarchical cluster analyses to explore an area of study. Through these analyses, the method creates visual depictions of conceptual relationships between ideas. Data yielded a seven cluster concept map that CRP members (N = 36) utilized for planning processes, and subsequently for developing an internal evaluation tool. Results from this study offer a unifying framework by which CRPs, and similar groups in other areas can utilize for planning and evaluation purposes. After a review of pertinent literature on CRPs, this article explicates CM processes utilized in this study, describes results, discusses lessons learned, and outlines apposite areas for future CRP research.     

Using concept mapping to identify opportunities for HPV vaccination efforts: Perspectives from the Midwest and West Coast

Human papillomavirus (HPV) infection significantly contributes to the burden of cancer in the United States, despite the existence of a highly effective vaccine. While numerous interventions to address vaccination uptake exist, vaccination rates remain low. We conducted a concept mapping exercise to solicit perspectives on barriers and facilitators to HPV vaccination from state-level stakeholders in five states in the Midwest and West Coast of the U.S. We identified 10 clusters of barriers and facilitators based on participants’ statements. For rural areas specifically, clusters rated as most important included education and provider influence; those rated as most feasible were education and coordinated/consistent messaging. Our results suggest that a combination of important (but potentially more difficult to implement) strategies, combined with those rated as most feasible (but potentially less impactful) may be beneficial. Our findings highlight similarities across diverse states, suggesting that states can learn from each other and work together to improve HPV vaccination rates. Using concept mapping proved to be an efficient way to collect information from diverse, stakeholders in different locations, and is a methodology that could be used for program planning in areas beyond HPV vaccination.     

An overview of concept mapping in Dutch mental health care

About 25 years ago, concept mapping was introduced in the Netherlands and applied in different fields. A collection of concept mapping projects conducted in the Netherlands was identified, in part in the archive of the Netherlands Institute of Mental Health and Addiction (Trimbos Institute). Some of the 90 identified projects are internationally published. The 90 concept mapping projects reflect the changes in mental health care and can be grouped into 5-year periods and into five typologies. The studies range from conceptualizing the problems of the homeless to the specification of quality indicators for treatment programs for patients with cystic fibrosis. The number of concept mapping projects has varied over time. Growth has been considerable in the last 5 years compared to the previous 5 years. Three case studies are described in detail with 12 characteristics and graphical representations. Concept mapping aligns well with the typical Dutch approach of the “Poldermodel.” A broad introduction of concept mapping in European countries in cooperation with other countries, such as the United States and Canada, would strengthen the empirical basis for applying this approach in health care policy, quality, and clinical work.     

And then the internet happened: Thoughts on the future of concept mapping

Over 25 years ago, in the late twentieth century, concept mapping emerged as a mixed method approach to inquiry that enables a group of people to conceptualize their thinking about a specific topic. Since then, the application of concept mapping has spread widely and an easy prediction for the future is that this trend is likely to continue; a more important and greater challenge is to think about the ways in which concept mapping may and should evolve. Discussed here are thoughts about the future of concept mapping including some predictions of likely directions and suggestions for new possibilities. Thoughts on the future are grounded in concept mapping applications that have emerged and gained ground in recent years; these include exploring wicked problems in communities and integrating concept mapping with other methods of inquiry. Thoughts on the future are also grounded in the social and cultural milieu in which we find ourselves at this time. The influence of social media and internet technologies has led to the emergence peer production and crowdsourcing as approaches to co-create information, knowledge, products and services. These tactics may create fertile ground for the further spread of concept mapping. This same collaborative milieu has produced the open software movement which in turn, offers opportunities to enhancing the methodology of concept mapping.     

Effective elements of care-physical activity initiatives for adults with a low socioeconomic status: A concept mapping study with health promotion experts

In care-physical activity (care-PA) initiatives, primary care and sports are combined to stimulate PA among adults at risk of lifestyle related diseases. Preliminary results from Dutch care-PA initiatives for adults with a low socioeconomic status (SES) indicate a decrease in participants’ body weight and an improved quality of life, however, the elements that make these initiatives successful are yet to be identified. In total, 19 Dutch health promotion experts participated in our concept mapping (CM) of the effective elements of care-PA initiatives for adults with a low SES. The experts identified 111 effective elements of these initiatives, which were grouped into 11 clusters, focusing on: 1) approaching participants within the care-PA initiative, 2) barriers experienced throughout the initiative, 3) long-term implementation, 4) customizing the care-PA initiative to the target population, 5) social support, 6) structure and guidance, 7) the professionals within the care-PA initiative, 8) the accessibility of the care-PA initiative, 9) targeted behaviour and progression, 10) recruitment and administration, and 11) intersectoral collaboration. CM was useful for creating a valuable overview of these effective elements. Our results could be used to improve the development and implementation of future care-PA initiatives for adults with a low SES.     

Planning and implementing group supervision: a case study from homeless social care practice

ABSTRACT

Effective clinical supervision is increasingly recognised as essential in supporting staff who are working in the field of social care. Using a case study approach, this paper illustrates the processes involved in planning and implementing a group supervision pilot with a team working in a homeless service. The supervision models used to inform and guide this initiative are described and the style and structure of the supervision sessions are discussed. The paper additionally provides a reflective analysis of group dynamics and group processes observed during this work. Subsequent conclusions identify the key features of this process and may provide guidance for other practice settings.     

Hindsight is 20/20: Reflections on the evolution of concept mapping

This paper considers the origins and development of the concept mapping methodology, a summary of its growth, and its influence in a variety of fields. From initial discussions with graduate students, through the rise of the theory-driven approach to program evaluation and the development of a theoretical framework for conceptualization methodology, the paper highlights some of the key early efforts and pilot projects that culminated in a 1989 special issue on the method in Evaluation and Program Planning that brought the method to the attention of the field of evaluation. The paper details the thinking that led to the standard version of the method (the analytic sequence, “bridging” index, and pattern matching) and the development of the software for accomplishing it. A bibliometric analysis shows that the rate of citation continues to increase, where it has grown geographically and institutionally, that the method has been used in a wide variety of disciplines and specialties, and that the literature had an influence on the field. The article concludes with a critical appraisal of some of the key aspects of the approach that warrant further development.     

Perspectives on the strategic uses of concept mapping to address public health challenges

We examine the adaptation of approaches used to plan and implement the steps of concept mapping to meet specialized needs and requirements in several public health projects. Seven published concept mapping projects are detailed to document how each of the phases were modified to meet the specific aims of each project. Concept mapping was found to be a useful tool to complement public health roles such as assessment, program development, and priority setting. The phases of concept mapping allow for a blending of diverse perspectives, which is critical to public health efforts. The adaptability of concept mapping permits the use of multiple modalities such as the addition of face-to-face brainstorming; use of qualitative methods, including structured interviews; and review and use of published literature and guidelines. Another positive aspect of concept mapping for public health practice is its ability to identify program elements, provide a visual map of generated ideas and their relationships to one another, and assist in identifying priorities. Our reflections on the adaptability should help inform another generation in designing concept mapping projects and related products that may benefit from unique adaptations and the rapidly expanding social media technology and platforms.     

Quality and rigor of the concept mapping methodology: A pooled study analysis

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.     

Trustworthiness in evaluation practice: an emphasis on the relational

As canons for trustworthiness developed explicitly in the discourse of qualitative inquiry, the emphasis was on procedural matters rather than fundamentally relational ones. A nod was made to the relational in such strategies as “member checks” but the issues of how the evaluator actually relates to participants and to the larger communities of practice and discourse—matters subsumed under moral principles and ethical standards—were often marginalized. This chapter posits that the first consideration in designing and conducting rigorous evaluation inquiry, and in critiquing the results of any research, should be the study's trustworthiness. Judging a study's trustworthiness, however, should focus on much more than procedural matters; it should also rely on moral principles and ethical standards which specifically address how we relate to an evaluation's participants. Central to these considerations are cross-cultural sensibilities about the nuanced meanings associated with any principles and standards. The chapter provides a theoretical framework for this position and offers two illustrative examples in the form of dialogues.