Quality and rigor of the concept mapping methodology: a pooled study analysis

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.     

Quality and rigor of the concept mapping methodology: A pooled study analysis

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.     

The use of concept mapping for scale development and validation in evaluation

Evaluators often make key decisions about what content to include when designing new scales. However, without clear conceptual grounding, there is a risk these decisions may compromise the scale's validity. Techniques such as concept mapping are available to evaluators for the specification of conceptual frameworks, but have not been used as a fully integrated part of scale development. As part of a multi-site evaluation of family support programs, we integrated concept mapping with traditional scale-development processes to strengthen the creation of a scale for inclusion in an evaluation instrument. Using concept mapping, we engaged staff and managers in the development of a framework of intended benefits of program participation and used the information to systematically select the scale's content. The psychometric characteristics of the scale were then formally assessed using a sample of program participants. The implications of the approach for supporting construct validity, inclusion of staff and managers, and theory-driven evaluation are discussed.     

The use of concept mapping in measurement development and evaluation: Application and future directions

The past decade has seen an increase of measurement development research in social and health sciences that featured the use of concept mapping as a core technique. The purpose, application, and utility of concept mapping have varied across this emerging literature. Despite the variety of uses and range of outputs, little has been done to critically review how researchers have approached the application of concept mapping in the measurement development and evaluation process. This article focuses on a review of the current state of practice regarding the use of concept mapping as methodological tool in this process. We systematically reviewed 23 scale or measure development and evaluation studies, and detail the application of concept mapping in the context of traditional measurement development and psychometric testing processes. Although several limitations surfaced, we found several strengths in the contemporary application of the method. We determined concept mapping provides (a) a solid method for establishing content validity, (b) facilitates researcher decision-making, (c) insight into target population perspectives that are integrated a priori, and (d) a foundation for analytical and interpretative choices. Based on these results, we outline how concept mapping can be situated in the measurement development and evaluation processes for new instrumentation.     

Illuminating the Black Box From Within: Stakeholder Perspectives on Family Drug Court Best Practices

This article describes a mixed-methods, concept mapping study in an urban family drug court (FDC) designed to identify keys to FDC success from stakeholders’ perspectives. Participating FDC team members and clients developed a set of items they deemed integral to an FDC, thematically clustered the items, and then rated their relative importance. Using these data, cluster analysis and multidimensional scaling generated 6 themes perceived as contributing to positive outcomes. Resultant concept maps revealed that, compared to team members, clients view relational aspects of the FDC as more important. The findings point to implications for future research and social work practice.     

Social Workers Can e-Learn: Evaluation of a Pilot Post-Qualifying e-Learning Course in Research Methods and Critical Appraisal Skills for Social Workers

Social workers are increasingly encouraged to become research minded and integrate research findings into their practice. However, there is a paucity of high-quality accessible training for social workers in research methods and critical appraisal skills. We developed an e-learning version of an MSc degree module to widen access to this kind of training and this paper presents the findings of an evaluation of a pilot of this e-learning course. To test the effectiveness of the e-learning environment in delivering research methods and critical appraisal skills training to social workers in comparison with a classroom group, we adopted a mixed methods approach which comprised concept mapping exercises, semi-structured interviews and evaluation questionnaires administered to the e-learners (n = 3) and a classroom comparison group (n = 12). Our analysis of the students' concept maps revealed no conspicuous differences in the quality of student knowledge changes that were a consequence of the mode of teaching delivery. The e-learners met the learning outcomes of the module to the same extent as the classroom group and were highly satisfied with the mode of delivery. Although this was a small pilot, it provided tentative evidence that social workers can use e-learning to develop evidence-based practice skills.     

Using concept mapping as a planning tool: Child welfare citizen review panels

Citizen Review Panels (CRPs) are groups of citizen volunteers authorized by U.S. federal law to examine state child welfare agencies. These groups inspect policies and practices related to child protection responsibilities and are tasked with making recommendations for systemic improvement. Despite the federal mandate for each state to develop a CRP and the potential of these groups to positively impact child welfare practices, there is a dearth in the literature related to CRPs. Consequently, planning and evaluation processes of these groups vary widely. This study reports on the use of concept mapping (CM) to outline a framework for planning and subsequently evaluating the CRP in one southeastern state. CM is a mixed-method research approach that uses multi-dimensional scaling and hierarchical cluster analyses to explore an area of study. Through these analyses, the method creates visual depictions of conceptual relationships between ideas. Data yielded a seven cluster concept map that CRP members (N = 36) utilized for planning processes, and subsequently for developing an internal evaluation tool. Results from this study offer a unifying framework by which CRPs, and similar groups in other areas can utilize for planning and evaluation purposes. After a review of pertinent literature on CRPs, this article explicates CM processes utilized in this study, describes results, discusses lessons learned, and outlines apposite areas for future CRP research.     

Medicine and Social Work: An Exploration in Interprofessionalism,Edited by Helen Rehr. Neal Watson Academic Publications,New York, 1974. $3.95 Paper

This article presents the results of focus groups conducted with participants following the completion of a Child Welfare Practice Certificate Program. Concept mapping was used as the guiding research method for the data collection and analysis. Social work students and child welfare workers generated statements during the focus group session that were then sorted into conceptual piles. Participants then rated the statements as to their importance and practicality and their interest in learning about them. Results revealed that the concept Reporting, Assessment, and Case Planning in the Real World was rated highest in all three categories. Recommendations are provided regarding infusing a child welfare curriculum into social work programs, and the authors discuss how concept mapping can be used in evaluation research.     

Concept mapping as an approach for expert-guided model building: The example of health literacy

Concept mapping served as the starting point for the aim of capturing the comprehensive structure of the construct of ‘health literacy.’ Ideas about health literacy were generated by 99 experts and resulted in 105 statements that were subsequently organized by 27 experts in an unstructured card sorting. Multidimensional scaling was applied to the sorting data and a two and three-dimensional solution was computed. The three dimensional solution was used in subsequent cluster analysis and resulted in a concept map of nine “clusters”: (1) self-regulation, (2) self-perception, (3) proactive approach to health, (4) basic literacy and numeracy skills, (5) information appraisal, (6) information search, (7) health care system knowledge and acting, (8) communication and cooperation, and (9) beneficial personality traits. Subsequently, this concept map served as a starting point for developing a “qualitative” structural model of health literacy and a questionnaire for the measurement of health literacy. On the basis of questionnaire data, a “quantitative” structural model was created by first applying exploratory factor analyses (EFA) and then cross-validating the model with confirmatory factor analyses (CFA). Concept mapping proved to be a highly valuable tool for the process of model building up to translational research in the “real world”.     

Engaging children and youth in research and evaluation using group concept mapping

Group concept mapping is a systematic methodology that recognizes participants’ expertise in their own experiences. In this approach, participants share their diverse knowledge and interpret the data generated by all group members. This approach uniquely blends quantitative and qualitative techniques to create illustrative, structured conceptualizations. In this article, we describe the steps in group concept mapping and report feedback on this method from 37 children and youth who participated in group concept mapping studies. We recommend group concept mapping as an engaging way to involve children and youth in research and evaluation activities.     

R-CMap—An open-source software for concept mapping

Planning and evaluating projects often involves input from many stakeholders. Fusing and organizing many different ideas, opinions, and interpretations into a coherent and acceptable plan or project evaluation is challenging. This is especially true when seeking contributions from a large number of participants, especially when not all can participate in group discussions, or when some prefer to contribute their perspectives anonymously. One of the major breakthroughs in the area of evaluation and program planning has been the use of graphical tools to represent the brainstorming process. This provides a quantitative framework for organizing ideas and general concepts into simple-to-interpret graphs. We developed a new, open-source concept mapping software called R-CMap, which is implemented in R. This software provides a graphical user interface to guide users through the analytical process of concept mapping. The R-CMap software allows users to generate a variety of plots, including cluster maps, point rating and cluster rating maps, as well as pattern matching and go-zone plots. Additionally, R-CMap is capable of generating detailed reports that contain useful statistical summaries of the data. The plots and reports can be embedded in Microsoft Office tools such as Word and PowerPoint, where users may manually adjust various plot and table features to achieve the best visual results in their presentations and official reports. The graphical user interface of R-CMap allows users to define cluster names, change the number of clusters, select rating variables for relevant plots, and importantly, select subsets of respondents by demographic criteria. The latter is particularly useful to project managers in order to identify different patterns of preferences by subpopulations. R-CMap is user-friendly, and does not require any programming experience. However, proficient R users can add to its functionality by directly accessing built-in functions in R and sharing new features with the concept mapping community.     

The Role of Self-Conscious Emotions in the Sexual Health of Gay and Bisexual Men: Psychometric Properties and Theoretical Validation of the Sexual Shame and Pride Scale

Researchers have theorized about the role of sexual shame as a mechanism through which sexual minority stress manifests into mental health difficulties, such as sexual compulsivity for gay and bisexual men (GBM), and about the resilience-promoting effects of sexual pride. However, no validated measures to date have directly tapped into these constructs rather than using proxies for them, such as internalized homonegativity. We developed the Sexual Shame and Pride Scale (SSPS) and conducted a psychometric evaluation of it using a sample of 260 highly sexually active GBM. The scale had the expected structure in factor analysis and showed evidence of internal consistency and test–retest reliability. Correlational analyses demonstrated the convergent validity of sexual shame and sexual pride with relevant constructs. Regression analyses demonstrated the predictive validity of sexual shame in relation to sexual compulsivity, accounting for unique variability even after adjusting for previously demonstrated etiological factors, and the predictive validity of both shame and pride, which interacted to consistently predict four sexual behavior outcomes. Findings suggest the SSPS is a psychometrically valid and reliable measure that may be useful in future empirical work and highlight preliminary evidence for the role of these constructs in the sexual health of GBM.     

The validity of self-reported cost events by substance abusers. Limits, liabilities, and future directions

The following review considers data on the validity of self-reports in addict populations, and then it discusses (a) the types of cost-related questions and the assumptions underlying them that are useful to the evaluation of addictions treatment, (b) both internal and external sources of invalidity, (c) the limits on cost-related information that is gathered from administrative databases, (d) methods for assessing measure validity, and (e) the means for improving the validity of self-reports of cost events. With some important exceptions, addicts provide valid data about both medical and criminal cost events. Skilled socioeconomic researchers able to monetarize these events should be able to produce significant cost of illness, cost offset, cost-benefit, and cost-effectiveness research using self-report data.     

Use of concurrent mixed methods combining concept mapping and focus groups to adapt a health equity tool in Canada

The aim of this project was to identify and prioritize a set of conditions to be considered for incorporating a health equity tool into public health practice. Concept mapping and focus groups were implemented as complementary methods to investigate the conditions of use of a health equity tool by public health organizations in Quebec. Using a hybrid integrated research design is a richer way to address the complexity of questions emerging from intervention and planning settings. This approach provides a deeper, operational, and contextualized understanding of research results involving different professional and organizational cultures, and thereby supports the decision-making process. Concept mapping served to identify and prioritize in a limited timeframe the conditions to be considered for incorporation into a health equity tool into public health practices. Focus groups then provided a more refined understanding of the barriers, issues, and facilitating factors surrounding the tools adoption, helped distinguish among participants’ perspectives based on functional roles and organizational contexts, and clarified some apparently contradictory results from the concept map. The combined use of these two techniques brought the strengths of each approach to bear, thereby overcoming some of the respective limitations of concept mapping and focus groups. This design is appropriate for investigating targets with multiple levels of complexity.     

Processes of Religious and Spiritual Influence in Adolescence: A Systematic Review of 30 Years of Research

This is a systematic review of 30 years (1988‐2017) of empirical research on processes of religious/spiritual influence in adolescence. We followed a multi‐step process that resulted in 241 studies organized according to eight research questions and the corresponding methods and analyses typically used to address them. We coded these studies based on the dimensions of religiosity/spirituality and the youth outcomes involved. In some cases (quantitative studies of mediation and moderation, as well as qualitative studies) we also coded a third process variable. Results of the coding process revealed a number of interesting patterns. First, religiosity/spirituality is generally adaptive for adolescents, protecting them from negative outcomes (e.g., risk behaviors and mental illness), and promoting positive youth development and flourishing. Nevertheless, in some contexts, religiosity/spirituality may be at least partially maladaptive. Second, there is some evidence, from experimental and longitudinal studies, that relations between religiosity/spirituality and adaptive outcomes are causal. Third, there are numerous complex and dynamic processes by which religiosity/spirituality relate to youth outcomes. In terms of mediation studies, the most salient mediating processes seem to involve religiosity/spirituality dimensions, peers, values/attitudes, and social control/norms. Fourth, religiosity/spirituality is multidimensional, involving various interwoven facets at the individual and ecological levels. Private or personal aspects of religiosity/spirituality (e.g., religious/spiritual importance) tend to be more salient predictors of outcomes than public aspects of religiosity/spirituality (e.g., religious worship service attendance). The results of this systematic review point to promising directions for future research. First, more research is needed studying a broader range of dimensions of religiosity/spirituality, processes of influence, and outcomes. In terms of religiosity/spirituality, much of the prior work has focused on overall religiosity/spirituality, and religious/spiritual behaviors (e.g., worship service attendance). In terms of outcomes, the emphasis has been on religiosity/spirituality protecting against maladaptive outcomes (e.g., substance use). Second, more research is needed examining the role of culture, and using more rigorous methods (e.g., experience sampling, experimental design, longitudinal design, or mixed methods). This systematic review provides a detailed analysis of what is known regarding processes of religious/spiritual influence in the lives of adolescents, and hopefully better positions researchers to move the field forward.     

Concept maps as network data: Analysis of a concept map using the methods of social network analysis

Concept mapping is a method that creates a visual representation that illustrates the thoughts, ideas, or planned actions that arise from a group of stakeholders on a particular issue. Social network analysis is a method that likewise creates a visual representation of data; a network map typically represents people and the connections, or lack thereof, between these people regarding a particular issue. While the goals of these two methods differ, the underlying data structures are similar; a network of relationships between data elements. Social network analysis is explored here as a supplement to concept mapping. A secondary analysis of a concept map to define to leadership needs was conducted using social network analysis. The methods and the implications for supplementing the analysis of concept maps and debriefing results with stakeholders are discussed.     

Time matters: the potential and pitfalls of using mixed methods approaches in longitudinal program evaluation

While social programs are often assessed using short-term impact studies, longitudinal designs allow evaluators to capture change over time, identify longer-term outcomes, adapt instruments, and better understand participants in transition. A mixed methods design can be critical in understanding these dynamics; yet there is a lack of literature exploring the practical considerations of planning and conducting qualitative and quantitative data collection and analysis within longitudinal studies. This paper examines two different mixed methods frameworks used in a 5-year evaluation of three youth entrepreneurship programs in East Africa. We show how the evaluation team dealt with unique challenges across methods and over time, and how the design ultimately facilitated a richer understanding of program impacts and processes. Considerations for conducting this type of study are explored, related to the impact of longevity on analysis and research staff. Successfully using a longitudinal mixed methods approach requires researchers to be strategic and reflexive, and work in close collaboration.     

Measuring societal effects of transdisciplinary research projects: design and application of an evaluation method

Most Transdisciplinary Research (TdR) projects combine scientific research with the building of decision making capacity for the involved stakeholders. These projects usually deal with complex, societally relevant, real-world problems. This paper focuses on TdR projects, which integrate the knowledge of researchers and stakeholders in a collaborative transdisciplinary process through structured methods of mutual learning. Previous research on the evaluation of TdR has insufficiently explored the intended effects of transdisciplinary processes on the real world (societal effects). We developed an evaluation framework for assessing the societal effects of transdisciplinary processes. Outputs (measured as procedural and product-related involvement of the stakeholders), impacts (intermediate effects connecting outputs and outcomes) and outcomes (enhanced decision making capacity) are distinguished as three types of societal effects. Our model links outputs and outcomes of transdisciplinary processes via the impacts using a mediating variables approach. We applied this model in an ex post evaluation of a transdisciplinary process. 84 out of 188 agents participated in a survey. The results show significant mediation effects of the two impacts "network building" and "transformation knowledge". These results indicate an influence of a transdisciplinary process on the decision making capacity of stakeholders, especially through social network building and the generation of knowledge relevant for action.