The use of concept mapping in measurement development and evaluation: Application and future directions

The past decade has seen an increase of measurement development research in social and health sciences that featured the use of concept mapping as a core technique. The purpose, application, and utility of concept mapping have varied across this emerging literature. Despite the variety of uses and range of outputs, little has been done to critically review how researchers have approached the application of concept mapping in the measurement development and evaluation process. This article focuses on a review of the current state of practice regarding the use of concept mapping as methodological tool in this process. We systematically reviewed 23 scale or measure development and evaluation studies, and detail the application of concept mapping in the context of traditional measurement development and psychometric testing processes. Although several limitations surfaced, we found several strengths in the contemporary application of the method. We determined concept mapping provides (a) a solid method for establishing content validity, (b) facilitates researcher decision-making, (c) insight into target population perspectives that are integrated a priori, and (d) a foundation for analytical and interpretative choices. Based on these results, we outline how concept mapping can be situated in the measurement development and evaluation processes for new instrumentation.     

Use of multiple cluster analysis methods to explore the validity of a community outcomes concept map

Concept mapping is now a commonly-used technique for articulating and evaluating programmatic outcomes. However, research regarding validity of knowledge and outcomes produced with concept mapping is sparse. The current study describes quantitative validity analyses using a concept mapping dataset. We sought to increase the validity of concept mapping evaluation results by running multiple cluster analysis methods and then using several metrics to choose from among solutions. We present four different clustering methods based on analyses using the R statistical software package: partitioning around medoids (PAM), fuzzy analysis (FANNY), agglomerative nesting (AGNES) and divisive analysis (DIANA). We then used the Dunn and Davies-Bouldin indices to assist in choosing a valid cluster solution for a concept mapping outcomes evaluation. We conclude that the validity of the outcomes map is high, based on the analyses described. Finally, we discuss areas for further concept mapping methods research.     

An application of theory-driven evaluation to a drop-in youth center

This article reports on the theory-driven evaluation of a drop-in center for youth that incorporated a literature search, concept mapping with staff, and focus groups with youth. Findings revealed strong agreement among the three sources of data around specific elements identified as critical components of a program theory of global prevention in after-school-hours initiatives, such as drop-in centers. These results are used to illustrate how a theory-driven approach was relevant for the context and objectives of this evaluation, as well as how it was used to develop knowledge useful for action, social intervention theory, and further research.     

Web-based nutrition education for college students: Is it feasible?

College students have poor nutrition habits and often exhibit at-risk weight control behaviors; yet, health promotion efforts on campuses often target other risk behaviors such as alcohol and tobacco use. The goal of this study was to determine program content and examine the feasibility of a web-based nutrition education program for college students using innovative applications of tailoring, targeting and personalization of information. Comprehensive program planning entailed three phases: focus groups and use of concept mapping methodology, prototype web program development, and feasibility testing of the prototype. Results of concept mapping suggested that students and experts agreed on the relative importance of the major content areas unique to the college audience. A prototype web program, informed by health promotion theories, was developed and evaluated. Students and college health professionals participated in the feasibility evaluation, which included acceptance and usability testing of the prototype. Mean scores were computed for the acceptance ratings that indicated that the prototype web program was not only feasible, but also compelling and relevant for the college audience. In conclusion, a tailored, web-based interactive nutrition program could potentially be used to provide individualized nutrition information to encourage college students to adopt healthy eating behaviors.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

The conceptualization and measurement of quality of life: implications for program planning and evaluation in the field of intellectual disabilities

The concept of quality of life (QOL) is increasingly being used in the field of intellectual disabilities as a conceptual and measurement framework for program planning and evaluation. This article describes the development of a QOL conceptual and measurement framework, and summarizes how this framework is currently being used both nationally and internationally to assess and report personal QOL-related outcomes, to guide quality improvement strategies, and to evaluate the effectiveness of those strategies. Implications of such use are discussed, including those related to understanding mental models, developing internal data systems, supporting organization change, and building on current public policies. The article concludes with reference to the evolving nature of the QOL concept and the impact of this on model development and transdisciplinary research.     

Quality and rigor of the concept mapping methodology: A pooled study analysis

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.     

Kind discipline: Developing a conceptual model of a promising school discipline approach

This formative evaluation develops a novel conceptual model for a discipline approach fostering intrinsic motivation and positive relationships in schools. We used concept mapping to elicit and integrate perspectives on kind discipline from teachers, administrators, and other school staff. Three core themes describing kind discipline emerged from 11 identified clusters: (1) proactively developing a positive school climate, (2) responding to conflict with empathy, accountability, and skill, and (3) supporting staff skills in understanding and sharing expectations. We mapped the identified components of kind discipline onto a social ecological model and found that kind discipline encompasses all levels of that model including the individual, relational, environmental/structural, and even community levels. This contrasts with the dominant individual-behavioral discipline approaches that focus on fewer levels and may not lead to sustained student and staff motivation. The findings illustrate the importance of setting and communicating clear expectations and the need for them to be collaboratively developed. Products of the analysis and synthesis reported here are operationalized materials for teachers grounded in a “be kind” culture code for classrooms.     

Quality and rigor of the concept mapping methodology: a pooled study analysis

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.     

An assessment scale of ECB: perspectives from the faculty and staff of technical colleges in Taiwan

Faculty and staff in Taiwanese technical colleges are required to perform school self-evaluations. Most members of technical colleges come from postsecondary schools and are thus unfamiliar with school self-evaluation procedures. An effective school self-evaluation involves a complete ECB,(1) making it necessary to build faculty and staff evaluation capacity. In this research, we analyzed previous studies, interviewed experts, conducted expert reviews, constructed an ECB assessment scale, and collected the opinions of 1054 faculty and staff, to confirm the validity of the proposed assessment scale. This scale included 21 items summarized in four dimensions: understanding the evaluation guidelines, familiarizing with evaluation requirements, participating in the evaluation activity, and implementing the evaluation connotation. We explain the four dimensions behind the ECB model, and propose three lessons learned from it.     

The development and validation of The Inquiry Science Observation Coding Sheet

Evaluation reports increasingly document the degree of program implementation, particularly the extent to which programs adhere to prescribed steps and procedures. Many reports are cursory, however, and few, if any, fully portray the long and winding path taken when developing evaluation instruments, particularly observation instruments. In this article, we describe the development of an observational method for evaluating the degree to which K-12 inquiry science programs are implemented, including the many steps and decisions that occurred during the development, and present evidence for the reliability and validity of the data that we collected with the instrument. The article introduces a method for measuring the adherence of inquiry science implementation and gives evaluators a full picture of what they might expect when developing observation instruments for assessing the degree of program implementation.     

Medicine and Social Work: An Exploration in Interprofessionalism,Edited by Helen Rehr. Neal Watson Academic Publications,New York, 1974. $3.95 Paper

This article presents the results of focus groups conducted with participants following the completion of a Child Welfare Practice Certificate Program. Concept mapping was used as the guiding research method for the data collection and analysis. Social work students and child welfare workers generated statements during the focus group session that were then sorted into conceptual piles. Participants then rated the statements as to their importance and practicality and their interest in learning about them. Results revealed that the concept Reporting, Assessment, and Case Planning in the Real World was rated highest in all three categories. Recommendations are provided regarding infusing a child welfare curriculum into social work programs, and the authors discuss how concept mapping can be used in evaluation research.     

An Empirically Based Framework for Content Selection and Management in Divorce Education Programs

Contemporary research has shown that divorce education programs are becoming mandated or recommended in numerous states throughout the nation. Cooccurring with the popularity of these programs is the variation in content of divorce education programs. The focus of this project was to systematically explore the research literature on factors related to divorce adjustment and identify findings to be used in content selection and development for divorce education programs. The divorce-stress-adjustment perspective was utilized to frame the divorce process affecting both children and adults. A systematic literature review resulted in identification of 103 published studies that fit the project criteria. Selected research findings identified in the literature review process were organized according to a conceptual framework designed to categorize content in a tiered approach useful for selecting and managing content in divorce education programs. Implications of the project findings for content identification in divorce education, program development, and program evaluation are explored.     

Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts.The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts. The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

Perspectives on the strategic uses of concept mapping to address public health challenges

We examine the adaptation of approaches used to plan and implement the steps of concept mapping to meet specialized needs and requirements in several public health projects. Seven published concept mapping projects are detailed to document how each of the phases were modified to meet the specific aims of each project. Concept mapping was found to be a useful tool to complement public health roles such as assessment, program development, and priority setting. The phases of concept mapping allow for a blending of diverse perspectives, which is critical to public health efforts. The adaptability of concept mapping permits the use of multiple modalities such as the addition of face-to-face brainstorming; use of qualitative methods, including structured interviews; and review and use of published literature and guidelines. Another positive aspect of concept mapping for public health practice is its ability to identify program elements, provide a visual map of generated ideas and their relationships to one another, and assist in identifying priorities. Our reflections on the adaptability should help inform another generation in designing concept mapping projects and related products that may benefit from unique adaptations and the rapidly expanding social media technology and platforms.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

The Hassles Assessment Scale for Students in College: measuring the frequency and unpleasantness of and dwelling on stressful events.

Development of the Hassles Assessment Scale for Students in College, a new scale to measure students' stress, is described. In the scale, students rate each of 54 hassles for its frequency and unpleasantness in the past month and indicate the degree to which they dwelt or ruminated on it. Very high levels of internal consistency for the frequency, unpleasantness, and dwelling measures were found. Correlational analyses demonstrated the scale's criterion validity (scores were negatively correlated with the number of hours respondents reported engaging in physical exercise) and congruent validity (scores were positively correlated with scores on the Inventory of College Students' Recent Life Experience, an established scale for assessing student hassles). Exploratory factor analyses suggested the possibility that many items on the scale are independent, with each contributing some specific variance to the total variance of the item pool that is not shared with other items.     

The State of Our Art

Abstract

In the second part of this two-part paper, the author examines intergenerational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, which appeared in the previous issue, conceptual foundations for intergenerational programming were considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross-cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.