Use of multiple cluster analysis methods to explore the validity of a community outcomes concept map

Concept mapping is now a commonly-used technique for articulating and evaluating programmatic outcomes. However, research regarding validity of knowledge and outcomes produced with concept mapping is sparse. The current study describes quantitative validity analyses using a concept mapping dataset. We sought to increase the validity of concept mapping evaluation results by running multiple cluster analysis methods and then using several metrics to choose from among solutions. We present four different clustering methods based on analyses using the R statistical software package: partitioning around medoids (PAM), fuzzy analysis (FANNY), agglomerative nesting (AGNES) and divisive analysis (DIANA). We then used the Dunn and Davies-Bouldin indices to assist in choosing a valid cluster solution for a concept mapping outcomes evaluation. We conclude that the validity of the outcomes map is high, based on the analyses described. Finally, we discuss areas for further concept mapping methods research.     

The use of concept mapping for scale development and validation in evaluation

Evaluators often make key decisions about what content to include when designing new scales. However, without clear conceptual grounding, there is a risk these decisions may compromise the scale's validity. Techniques such as concept mapping are available to evaluators for the specification of conceptual frameworks, but have not been used as a fully integrated part of scale development. As part of a multi-site evaluation of family support programs, we integrated concept mapping with traditional scale-development processes to strengthen the creation of a scale for inclusion in an evaluation instrument. Using concept mapping, we engaged staff and managers in the development of a framework of intended benefits of program participation and used the information to systematically select the scale's content. The psychometric characteristics of the scale were then formally assessed using a sample of program participants. The implications of the approach for supporting construct validity, inclusion of staff and managers, and theory-driven evaluation are discussed.     

Perspectives on the strategic uses of concept mapping to address public health challenges

We examine the adaptation of approaches used to plan and implement the steps of concept mapping to meet specialized needs and requirements in several public health projects. Seven published concept mapping projects are detailed to document how each of the phases were modified to meet the specific aims of each project. Concept mapping was found to be a useful tool to complement public health roles such as assessment, program development, and priority setting. The phases of concept mapping allow for a blending of diverse perspectives, which is critical to public health efforts. The adaptability of concept mapping permits the use of multiple modalities such as the addition of face-to-face brainstorming; use of qualitative methods, including structured interviews; and review and use of published literature and guidelines. Another positive aspect of concept mapping for public health practice is its ability to identify program elements, provide a visual map of generated ideas and their relationships to one another, and assist in identifying priorities. Our reflections on the adaptability should help inform another generation in designing concept mapping projects and related products that may benefit from unique adaptations and the rapidly expanding social media technology and platforms.     

Theory and method in treatment evaluation

We outline a conceptual model of evaluation research that has both guided and flowed from our research on the process and outcome of treatment for alcohol abuse and for depression. The conceptual model encompasses three basic processes: the process of treatment selection, the process of treatment, and the process underlying the onset and course of the disorder or problem being treated. A theory guiding a program evaluation can focus on one or more of these processes, and findings from evaluations can contribute to the development of better theories of these processes. We discuss how theories can guide both experimental and naturalistic evaluations, as well as aid in the estimation of treatment effects with either type of design. We then consider the role of theory and method in exploring patient-treatment matching. Finally, we discuss the implications of theory-guidance for the utility of evaluations—particularly the utility for evaluation researchers.     

Concept mapping as a method to enhance evidence-based public health

In this paper we explore the suitability of concept mapping as a method for integrating knowledge from science, practice, and policy. In earlier research we described and analysed five cases of concept mapping procedures in the Netherlands, serving different purposes and fields in public health. In the current paper, seven new concept mapping studies of co-produced work are added to extend this analysis.For each of these twelve studies we analysed: (1) how the method was able to integrate knowledge from practice with scientific knowledge by facilitating dialogue and collaboration between different stakeholders in the field of public health, such as academic researchers, practitioners, policy-makers and the public; (2) how the method was able to bring theory development a step further (scientific relevance); and (3) how the method was able to act as a sound basis for practical decision-making (practical relevance).Based on the answers to these research questions, all but one study was considered useful for building more evidence-based public health, even though the extent to which they underpinned actual decision-making varied. The chance of actually being implemented in practice seems strongly related to the extent to which the responsible decision-makers are involved in the way the concept map is prepared and executed.     

Social Capital in the LEADER Initiative: a methodological approach

This paper focuses on the introduction of a suitable method for the measurement of social capital in the context of rural development policies. We present an empirical application of the method to four case studies from the south of Italy. In order to overcome some limits affecting previous empirical research, we have grounded the measurement framework upon a clear decomposition of the concept of social capital characterizing three main dimensions: structural, relational and cognitive. This has allowed us to build five direct indicators for the core components of social capital created within the EU local agencies for rural development (the Local Action Groups). Moreover we have set a synthetic measure capturing the specific configuration of the internal social capital of the groups as a whole.     

Evaluation utilization research-Developing a theory and putting it to use

This article presents the findings of a two-stage study that had two key objectives: to develop a theory about evaluation utilization in an educational organization and to apply this theory to promote evaluation utilization within the organization. The first stage involved a theoretical conceptualization using a participatory method of concept mapping. This process identified the modes of evaluation utilization within the organization, produced a representation of the relationship between them and led to a theory. The second stage examined the practical implications of this conceptualization in terms of how different stakeholders in the organization perceive the actual and preferable state of evaluation utilization within the organization (i.e. to what extent is evaluation utilized and to what extent should it be utilized). The participatory process of the study promoted the evaluation utilization by involving stakeholders, thus giving them a sense of ownership and improving communication between the evaluation unit and the stakeholders. In addition, understanding the evaluation needs of the stakeholders in the organization helped generate relevant and realizable evaluation processes. On a practical level, the results are currently shaping the evaluation plan and the place of evaluations within the organization.     

Engaging children and youth in research and evaluation using group concept mapping

Group concept mapping is a systematic methodology that recognizes participants’ expertise in their own experiences. In this approach, participants share their diverse knowledge and interpret the data generated by all group members. This approach uniquely blends quantitative and qualitative techniques to create illustrative, structured conceptualizations. In this article, we describe the steps in group concept mapping and report feedback on this method from 37 children and youth who participated in group concept mapping studies. We recommend group concept mapping as an engaging way to involve children and youth in research and evaluation activities.     

The conceptualization and measurement of quality of life: implications for program planning and evaluation in the field of intellectual disabilities

The concept of quality of life (QOL) is increasingly being used in the field of intellectual disabilities as a conceptual and measurement framework for program planning and evaluation. This article describes the development of a QOL conceptual and measurement framework, and summarizes how this framework is currently being used both nationally and internationally to assess and report personal QOL-related outcomes, to guide quality improvement strategies, and to evaluate the effectiveness of those strategies. Implications of such use are discussed, including those related to understanding mental models, developing internal data systems, supporting organization change, and building on current public policies. The article concludes with reference to the evolving nature of the QOL concept and the impact of this on model development and transdisciplinary research.     

Using the evaluation process as a lever for improving health and healthcare accessibility: The case of HCV services organization in Quebec

BackgroundThe evaluation process can be a lever to improve pathways of access to healthcare. The objective of this article is to show how an evaluation strategy can both contribute to knowledge development and have direct impacts on health services provision. We use the case of hepatitis C (HCV) services organization to illustrate the use and the value of this evaluative approach.MethodInspired by empowerment evaluation, the transformative–participatory approach involved overlapping phases of knowledge development and discussion with stakeholders. We conducted several knowledge development activities to discern the needs of people with HCV, the resources available, and the facilitators and impediments along the care pathway, starting from prevention and screening, all the way through to treatment. Using an overlapping approach allowed us to regularly transfer acquired knowledge back to the participants in the study settings and also to gather their impressions, interpretations, and suggestions during periods of deliberation.ResultsThe knowledge development activities made it possible to document the needs, resources, and experiences of people affected by HCV. In the discussion sessions, viable solutions were identified to improve health and healthcare access for people with HCV and to prioritize certain actions. This project demonstrated that using the evaluation process can enable an instrumental, conceptual use of results and, in fact, can have a transformative impact on services organization.     

Using developmental evaluation and community-based participatory research to develop a model of supportive housing

Community-based participatory research (CBPR) and developmental evaluation (DE) have emerged over recent decades as separate approaches for addressing complex social issues. Current literature offers little with respect to the use of CBPR and DE in combination, although the two approaches are complementary. Through the current paper, we outline how CBPR and DE were used to develop a model of supportive housing for teen families. More specifically, we describe the structures and processes that contributed to this development, including (1) our partnership approach, (2) pooled resources, (3) regular opportunities for collaboration and reflection, (4) integration of multiple data sources, (5) ongoing feedback and knowledge dissemination, and (6) adjustments to program practices. We end by providing insights into the lessons that we learned through this project. Through this paper, we describe how researchers and community partners can collaboratively use CBPR and DE to develop a program model in complex community settings. Insights are offered that will be important for researchers, evaluators, and practitioners seeking to develop programming in response to complex community issues.     

Extending systems thinking in planning and evaluation using group concept mapping and system dynamics to tackle complex problems

Group concept mapping (GCM) has been successfully employed in program planning and evaluation for over 25 years. The broader set of systems thinking methodologies (of which GCM is one), have only recently found their way into the field. We present an overview of systems thinking emerging from a system dynamics (SD) perspective, and illustrate the potential synergy between GCM and SD. As with GCM, participatory processes are frequently employed when building SD models; however, it can be challenging to engage a large and diverse group of stakeholders in the iterative cycles of divergent thinking and consensus building required, while maintaining a broad perspective on the issue being studied. GCM provides a compelling resource for overcoming this challenge, by richly engaging a diverse set of stakeholders in broad exploration, structuring, and prioritization. SD provides an opportunity to extend GCM findings by embedding constructs in a testable hypothesis (SD model) describing how system structure and changes in constructs affect outcomes over time. SD can be used to simulate the hypothesized dynamics inherent in GCM concept maps. We illustrate the potential of the marriage of these methodologies in a case study of BECOMING, a federally-funded program aimed at strengthening the cross-sector system of care for youth with severe emotional disturbances.     

The participatory design of a performance oriented monitoring and evaluation system in an international development environment

This article illustrates the application of the impact monitoring and evaluation process for the design and development of a performance monitoring and evaluation framework in the context of human and institutional capacity development. This participative process facilitated stakeholder ownership in several areas including the design, development, and use of a new monitoring and evaluation system, as well their targeted results and accomplishments through the use of timely performance data gathered through ongoing monitoring and evaluation. The process produced a performance indicator map, a comprehensive monitoring and evaluation framework, and data collection templates to promote the development, implementation, and sustainability of the monitoring and evaluation system of a farmer's trade union in an African country.     

Quality and rigor of the concept mapping methodology: a pooled study analysis

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.     

Quality and rigor of the concept mapping methodology: A pooled study analysis

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.     

Spaces enabling game-changing and sustaining innovations: why space matters for knowledge creation and innovation

Abstract

Innovation has become one of the key drivers for growth. However, how do we bring about innovation which is both radical and respects the limits of the world? One of our key assumptions is that we have to take into consideration the epistemological and cognitive processes leading to (radically) new knowledge first. We propose an approach that establishes spaces enabling such processes of knowledge creation we refer to them as Enabling Spaces.

This article is concerned with the question of how innovation, and more specifically, profound, radical and sustainable innovation can be brought about in a more qualitative manner. What are the necessary concepts and attitudes which facilitate the processes of innovation? The notion of enabling as opposed to 'managing' or controlling innovation is to be developed. Furthermore the concept of situated/extended cognition is discussed as a key ingredient for Enabling Spaces.

The second part gives an overview of the concept of Enabling Spaces and of the design process leading to such spaces. Finally the concrete case of a knowledge-creating university is discussed.     

Bioenergy Futures: Framing Sociotechnical Imaginaries in Local Places

Sociotechnical imaginaries are collectively imagined forms of social life reflected in the design and fulfillment of technological projects. While it is implied that there may be contention around sociotechnical imaginaries, the literature on how that contention is manifested is scant. We use a frame‐analytic approach to demonstrate the potency of collective action frames for making sense of the national imaginaries underpinning siting proposals. As a case study, we use woody biomass bioenergy development in northern Michigan. After briefly outlining the multiple frames that are encompassed in the imaginary of bioenergy development, we focus on the “wood for energy” frame, employing the concept of “frame keys” to demonstrate how national imaginaries are interpreted differently by local and nonlocal actors involved in community sitings of proposed facilities. We find not only that frame keys are essential to how the national imaginary of bioenergy is interpreted, (re)produced, and responded to but also that framing processes are related to social movements that coalesce around competing collective memories of place.     

How can We Enhance Productivity in Social Work? Dynamically Reflective Structures,Dialogic Leadership and the Development of Transformative Expertise

This article examines how to enhance the ability of managers to promote productivity through reflective structures and appreciatively evaluative dialogues based on front-line expertise and knowledge in social work practice. In order to discuss these ideas, we review the ideas of dialogical leadership and reflective structures, and the concept of productivity in relation to reassessing social work core tasks and the work-related well-being of social workers. We will present and reflect on our experiences and ideas generated from research that developed reflective tools, known as the Mirror method, for peer evaluation and practice development in social work. The Mirror method is designed as a tool for leaders and managers to enhance, through trialogic processes and structures, both the well-being of employees at the individual level and the productivity of social work and learning for transformation at the organizational level.     

Winning to Learn,Learning to Win: Evaluative Frames and Practices in Urban Debate

Sociologists of (e)valuation have devoted considerable attention to understanding differences in evaluative practices across a number of fields. Yet, little is understood about how individuals learn about and navigate multivalent valid group styles within a single setting. As a social phenomenon, many accept how central processes of evaluation are to everyday life. Accordingly, scholars have attempted to link research on evaluation to processes of inequality. Nevertheless, the sociology of evaluation only has tenuous, often implicit connections to literature on inequality and disadvantage. This article addresses these two gaps. Drawing on over two-hundred hours of ethnographic fieldwork in an urban high school debate league, twenty-seven semi-structured interviews with league judges, and archival data, we illustrate how high school policy debate judges employ evaluative frames and link them to the implementation of evaluative practices in a disadvantaged setting. We show that the cultural meanings that emerge within the evaluation process—in this case, urban uplift and competition—stem from the conflicted context in which evaluation is occurring. We also make a first step toward applying the conceptual tools within the sociology of evaluation to a disadvantaged setting, and more broadly, suggest that micro-processes of evaluation are important to the study of urban inequality.     

Using concept mapping as a planning tool: Child welfare citizen review panels

Citizen Review Panels (CRPs) are groups of citizen volunteers authorized by U.S. federal law to examine state child welfare agencies. These groups inspect policies and practices related to child protection responsibilities and are tasked with making recommendations for systemic improvement. Despite the federal mandate for each state to develop a CRP and the potential of these groups to positively impact child welfare practices, there is a dearth in the literature related to CRPs. Consequently, planning and evaluation processes of these groups vary widely. This study reports on the use of concept mapping (CM) to outline a framework for planning and subsequently evaluating the CRP in one southeastern state. CM is a mixed-method research approach that uses multi-dimensional scaling and hierarchical cluster analyses to explore an area of study. Through these analyses, the method creates visual depictions of conceptual relationships between ideas. Data yielded a seven cluster concept map that CRP members (N = 36) utilized for planning processes, and subsequently for developing an internal evaluation tool. Results from this study offer a unifying framework by which CRPs, and similar groups in other areas can utilize for planning and evaluation purposes. After a review of pertinent literature on CRPs, this article explicates CM processes utilized in this study, describes results, discusses lessons learned, and outlines apposite areas for future CRP research.