Quality and rigor of the concept mapping methodology: A pooled study analysis

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.     

An overview of concept mapping in Dutch mental health care

About 25 years ago, concept mapping was introduced in the Netherlands and applied in different fields. A collection of concept mapping projects conducted in the Netherlands was identified, in part in the archive of the Netherlands Institute of Mental Health and Addiction (Trimbos Institute). Some of the 90 identified projects are internationally published. The 90 concept mapping projects reflect the changes in mental health care and can be grouped into 5-year periods and into five typologies. The studies range from conceptualizing the problems of the homeless to the specification of quality indicators for treatment programs for patients with cystic fibrosis. The number of concept mapping projects has varied over time. Growth has been considerable in the last 5 years compared to the previous 5 years. Three case studies are described in detail with 12 characteristics and graphical representations. Concept mapping aligns well with the typical Dutch approach of the “Poldermodel.” A broad introduction of concept mapping in European countries in cooperation with other countries, such as the United States and Canada, would strengthen the empirical basis for applying this approach in health care policy, quality, and clinical work.     

Qualitative research in family therapy: a substantive and methodological review

Over the last decade, marriage and family therapy (MFT) researchers have developed a growing interest in qualitative research. In this article, we review substantive and methodological trends in the published qualitative studies within the MFT field. The research is compared and contrasted in the following areas: General topic, epistemological theory, methodological theory, sampling and sample, data collection, data analysis procedure, and approach to reliability and validity. We also provide recommendations for future research.     

Twenty Years of Social Studies Textbook Content Analysis: Still “Decidedly Disappointing”?

In an article published in 1993, Rahima Wade reviewed the quality of social studies textbook content analyses published in three major social studies journals between 1982 and 1992. She concluded that the quality of this research was disappointing. Borrowing and refining Wade's methods, this article presents the findings of a review of all 29 textbook content analyses published in the same journals between 1994 and 2016 to see whether textbook content analysis research has improved during the past two decades. The results indicate that the quality of this research has improved little: many studies were unclear in their sampling strategies, validity and reliability of coding variables, and data collection and analysis procedures. The methodological limitations rendered their substantive findings questionable. This article concludes with recommendations for textbook researchers to explicate their data collection and analysis processes and make their results more useful to social studies teaching and learning. It also has implications for social studies teachers on how to use textbooks critically to provide students a more meaningful and powerful learning experience.     

An Approach to Coordinated Teaching of Growth and Behavior in Class and Field

This study of the validity of admissions predictions made by the full-time faculty of a school of social work showed some capacity of the faculty to predict classroom performance, but no ability to predict field performance. Interjudge reliability for the group was low, with an overall reliability coefficient of .47. Nearly half the faculty could significantly predict classroom performance, however, there was a strong tendency toward negative correlation between admissions ratings and field performance, particularly by field instruction faculty. The pooled judgments of three faculty raters were shown not to have any significant advantage over average individual judgments in validity terms.     

A cross-national examination of the determinants of self-assessed health

Self-assessed health measures are often utilized in surveys of older adults, and these have been found to be powerful predictors of other health indicators and future health. However, there is still much that is puzzling about the measure. Few studies use comparative techniques to ascertain the determinants of self-assessed health across diverse settings. Using recent data from the Philippines, Taiwan, and Thailand, the current study tests whether determinants of self-assessed health are similar across samples and whether there are distinct country effects acting upon the association. Ordered probit models reveal that, on the one hand, the determinants of self-assessed health are relatively similar across the three countries, while, on the other, the overall probabilities of reporting favorable self-assessed health remain differentiated even when controlling for a broad series of determinants. The results suggest that cultural variation in the interpretation of health may be a factor and that more subtle measures should be included in future studies of this nature.     

A Microcomputer Management of Information System for Community-Based AIDS Prevention Ethnographic Team Research

Abstract

Ethnographers working on community-based intervention projects increasingly face the task of collecting and analyzing large amounts of qualitative data. In addition, these data sometimes must be simultaneously gathered across diverse populations and at multiple locations. A cost-effective approach that addresses the difficulties in conducting this type of qualitative research is the use of an ethnographic team. Yet, the team concept requires sophisticated strategies for managing large volumes of unstructured textbased data. This article documents how the Houston Nalional AlDS Demonstration Research (NADR) project configured specific basic microcomputer technologesto asslst and enhance the management of ethnographic team research. Specifically, this system was a valuable and affordable tool to increase the speed, efficiency and rigor of data collection and analysis.     

The impact of respondent attentiveness on reliability and validity

This study aims to assess whether respondent inattentiveness causes systematic and unsystematic measurement error that influences survey data quality. To determine the impact of (in)attentiveness on the reliability and validity of target measures, we compared respondents from a German online survey (N = 5205) who had passed two attention checks with those who had failed. Our results show that inattentiveness induces both random and systematic measurement error, which impacts estimates of the reliability and validity of multi-item scales. In addition, we conducted a sensitivity analysis, which revealed that the impact of inattentiveness on analyses can be substantial.     

The use of concept mapping in measurement development and evaluation: Application and future directions

The past decade has seen an increase of measurement development research in social and health sciences that featured the use of concept mapping as a core technique. The purpose, application, and utility of concept mapping have varied across this emerging literature. Despite the variety of uses and range of outputs, little has been done to critically review how researchers have approached the application of concept mapping in the measurement development and evaluation process. This article focuses on a review of the current state of practice regarding the use of concept mapping as methodological tool in this process. We systematically reviewed 23 scale or measure development and evaluation studies, and detail the application of concept mapping in the context of traditional measurement development and psychometric testing processes. Although several limitations surfaced, we found several strengths in the contemporary application of the method. We determined concept mapping provides (a) a solid method for establishing content validity, (b) facilitates researcher decision-making, (c) insight into target population perspectives that are integrated a priori, and (d) a foundation for analytical and interpretative choices. Based on these results, we outline how concept mapping can be situated in the measurement development and evaluation processes for new instrumentation.     

Use of multiple cluster analysis methods to explore the validity of a community outcomes concept map

Concept mapping is now a commonly-used technique for articulating and evaluating programmatic outcomes. However, research regarding validity of knowledge and outcomes produced with concept mapping is sparse. The current study describes quantitative validity analyses using a concept mapping dataset. We sought to increase the validity of concept mapping evaluation results by running multiple cluster analysis methods and then using several metrics to choose from among solutions. We present four different clustering methods based on analyses using the R statistical software package: partitioning around medoids (PAM), fuzzy analysis (FANNY), agglomerative nesting (AGNES) and divisive analysis (DIANA). We then used the Dunn and Davies-Bouldin indices to assist in choosing a valid cluster solution for a concept mapping outcomes evaluation. We conclude that the validity of the outcomes map is high, based on the analyses described. Finally, we discuss areas for further concept mapping methods research.     

Measuring and understanding trade in service tasks

The revolution in information and communications technologies has put service tasks with strong tradability characteristics at high risk of being offshored. This article reviews studies proposing indicators of service tradability, exploring the labour market implications of service offshoring and developing theories to rationalize the facts. It suggests that both skill intensity and tradability are determinants of wage and employment effects. Nonetheless, the lack of widely accepted definitions of tradability, the absence of high‐quality data on service trade flows and the difficulty of measuring import competition at higher disaggregation levels pose difficulties in achieving further progress, pointing to areas for future research.     

Interviewing undocumented immigrants: methodological reflections based on fieldwork in Mexico and the U.S

"This article discusses data collection methods and basic issues of research strategy in field studies of unapprehended illegal immigrants living in the United States. It suggests ways to increase the reliability and validity of interview responses. Necessary modifications in format and style of interviews are described. An annotated bibliography of recent field studies of undocumented immigrants is provided."     

The importance of using multiple outcome measures in infant research

Collecting data with infants is notoriously difficult. As a result, many of our studies consist of small samples, with only a single measure, in a single age group, at a single time point. With renewed calls for greater academic rigor in data collection practices, using multiple outcome measures in infant research is one way to increase rigor, and, at the same time, enable us to more accurately interpret our data. Here, we illustrate the importance of using multiple measures in psychological research with examples from our own work on rapid threat detection and from the broader infancy literature. First, we describe our initial studies using a single outcome measure, and how this strategy caused us to nearly miss a rich and complex story about attention biases for threat and their development. We demonstrate how using converging measures can help researchers make inferences about infant behavior, and how using additional measures allows us to more deeply examine the mechanisms that drive developmental change. Finally, we provide practical and statistical recommendations for how researchers can use multiple measures in future work.     

Noxious Chemical Exposure Trends as Measures of Green Victimization: Public Health,National Health and Nutrition Examination Survey Trends,and Green Criminology

Abstract

Green criminologists have argued that exposure to environmental pollution constitutes a serious form of green victimization of the general public and public health. This claim has not been widely assessed empirically. The present study employs data from the National Health and Nutrition Examination Survey (NHANES) over time to examine trends in exposure to environmental toxins among the U.S. population. NHANES data allow such an assessment through the collection of biological sampling used to determine exposure to environmental toxins. We examined trends for NHANES exposure estimates for biological indicators of pollutant exposure over time for 64 chemicals that have multiple measures. We divide the exposure trends into declining, stable, increasing, and unclear trajectory patterns and discuss the health consequences associated with exposure to measured chemical exposures. Discussion of the implications of this analysis and the need for future studies is provided.     

A Preliminary Study on the Effectiveness of the Chinese Career Key Online

The Chinese Career Key (CCK) Online was studied for its effectiveness and user satisfaction. Study participants were 92 students at a public high school in Hong Kong. The research findings provide preliminary evidence of reliability, validity, and user satisfaction for the CCK Online. Limitations and implications for future studies and counseling practice across cultures are discussed.     

Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach

Measurement errors can have profound effects on statisticalrelationships, and better information on the quality of measuresseems needed. This study uses a new technology—structuralmodeling of data from special supplements to regular surveys—togenerate estimates of construct validity, method effects (amajor source of correlated error), and residual error (mainlyrandom error) for a broad set of measures obtained from fivenational surveys and an organizational survey (total respondents= 7,706). Analysis of these estimates suggested that a typicalsurvey item, when administered by a respected survey organizationto a general population sample, can be expected to yield 50–83percent valid variance, 0–7 percent method effects variance,and 14–48 percent residual variance. Multivariate analysisshowed that over two-thirds of the variation in measurementquality could be explained by 13 survey design characteristics;characteristics of respondents explained a small additionalportion. Results provide: (a) information on design conditionsassociated with better (or worse) measurement quality, (b) empiricallybased suggestions for improving measurement quality in futuresurveys, and (c) a set of coefficients for predicting the qualityof measures not studied here.     

Evaluation of social network measurement instruments

This paper evaluates the reliability and validity of network measurement instruments for measuring social support. The authors present and discuss the results from eight experiments which were designed to analyze the quality of four measurement scales: (1) binary, (2) categorical, (3) categorical with labels, and (4) line production, as well as two measurement techniques for listing alters (free recall and recognition). Reliability and validity were estimated by the true score multitrait–multimethod (MTMM) approach. Meta-analysis of factors affecting the reliability and the validity of network measurement was done by multiple classification analysis (MCA). The results show that the binary scale and the first presentation of measurement instruments are the least reliable. Surprisingly, the two data collection techniques (free recall and recognition) yield equally reliable data.     

A reliability generalization meta-analysis of coefficient alpha and test–retest coefficient for the aging males’ symptoms (AMS) scale

Purpose: The aging males’ symptoms (AMS) scale is an instrument used to determine the health-related quality of life in adult and elderly men. The purpose of this study was to synthesize internal consistency (Cronbach’s alpha) and test–retest reliability for the AMS scale and its three subscales.

Methods: Of the 123 studies reviewed, 12 provided alpha coefficients which were then used in the meta-analyses of internal consistency. Seven of the 12 included studies provided test–retest coefficients, and these were used in the meta-analyses of test–retest reliability.

Results: The AMS scale had excellent internal consistency [α?=?0.89 (95% CI 0.88–0.90)]; the mean alpha estimates across the AMS subscales ranged from 0.79 to 0.82. The AMS scale also had good test–retest reliability [r?=?0.85 (95% CI 0.82–0.88]; the test–retest reliability coefficients of the AMS subscales ranged from 0.76 to 0.83. There was significant heterogeneity among the included studies.

Conclusions: The AMS scale and the three subscales had fairly good internal consistency and test–retest reliability. Future psychometric studies of the AMS scale should report important characteristics of the participants, details of item scores, and test–retest reliability.