Science,measurement, and validity: Is completion of Samuel Messick's synthesis possible?

Messick's (1989) theory of test validity is profoundly influential (Hubley and Zumbo, 1996; Angoff, 1988) in part because it brings together disparate contributions into a unified framework for building validity arguments. At the heart of Messick's theory lies a synthesis of realism and constructivism with respect to both scientific facts and measurement. Within this synthesis there remains a tension between the evidential basis and the consequential basis for test interpretation and use. This cannot be sidestepped simply by limiting the evidential basis to test interpretation and the consequential basis to test use: Interpretation and use are not so easily held separate. The roles of constructivism and context in Messick's theory underline the inherent link between facts and values, but the assumption that facts are objective and values are subjective goes unquestioned in Messick's theory. The inherent link between facts and values combines with this assumption to produce the unresolved tension in Messick's theory. This suggests that a unified theory of test validity requires a theory of value justification.     

An Indigenous Knowledges Perspective on Valid Meaning Making: A Commentary on Research with the EDI and Aboriginal Communities

Offering an Indigenous perspective, this commentary discusses collaborative research, shared meaning making, and knowledge building specific to child development, and reflects on social, cultural, and historical aspects that influence these processes. Drawing upon experiences of developing a collaborative research approach with which to engage Aboriginal communities to appreciate, understand, and potentially use the Early Development Instrument (EDI; Janus and Offord in Can J Behav Sci 39:1–22, 2007), a teacher-administered rating scale on kindergarten children’s development, the commentary focuses on five key questions relevant to research processes undertaken with Indigenous Peoples, and the importance of social, ethical, and cultural aspects of validity: How do Indigenous epistemologies and knowledges inform and influence research processes that utilize the EDI as a measurement tool? How can the EDI be used as a measurement tool within a research process that fosters the thriving of children and their families in Aboriginal communities while promoting Indigenous Peoples’ self-determination? In what ways do local, Indigenous cultural and ethical considerations inform aspects of validation research pertaining to the EDI? How can (Western mainstream) universities build research capacity that is informed by Indigenous knowledges and ways of being, doing, and knowing? What are the potential consequences of using normative research tools—such as the EDI—as a method to build knowledge on children’s development with Indigenous Peoples and Aboriginal communities? This commentary suggests that from an Indigenous perspective, research on child development is valid and meaningful knowledge if it is clearly linked to the children’s and families’ wellbeing according to local cultural norms and values.     

Validation Theory and Research for a Population-Level Measure of Children’s Development,Wellbeing, and School Readiness

This paper delineates general validity and research questions that are underlying an ongoing program of research pertaining to the Early Development Instrument (EDI, Janus and Offord 2007), a population-level measure, on which teachers rate kindergarten children’s developmental outcomes in the social, emotional, physical, cognitive, and communicative domains. It describes the large-scale research projects that are using the EDI to measure children’s developmental outcomes for entire populations (e.g., provinces, cities). Given the uniqueness of the EDI’s use as monitoring tool, or social indicator, for children’s developmental health at an early age, we spell out the opportunities and challenges that these projects provide with regard to validation research. The article sets the stage for the special issue, as it is dedicated to showcase the different aspects of the conceptual, theoretical, and empirical validation research as well as the collaborative community-based projects that are currently being undertaken with respect to the EDI.     

Bioecological Theory, Early Child Development and the Validation of the Population-Level Early Development Instrument

The Early Development Instrument (EDI; Janus and Offord in Canadian Journal of Behavioural Science 39:1–22, 2007) project is a Canadian population-level, longitudinal research project, in which teacher ratings of Kindergarten children’s early development and wellbeing are linked to health and academic achievement variables at the individual level, and to demographic, cultural, and socioeconomic variables at the community level. In this article, we draw from Bronfenbrenner’s bioecological theory of human development to create a coherent theoretical framework for guiding validation research within a population-level approach to child development research in general and for the EDI project in particular. The discussion draws from a range of social and health sciences as well as validity theory. The paper seeks to align complex conceptual, theoretical, methodological, and psychometric considerations, to provide specific design, methodology, and validation recommendations for a population-level approach to studying children’s development and wellbeing, and to discuss the strengths and challenges of this approach.     

Investigating the Structural Validity of Ryff’s Psychological Well-Being Scales Across Two Samples

Ryff’s (1989b) Psychological Well-Being (PWB) scales measure six related constructs of human functioning. The present paper examined the validity of Ryff’s 6-factor PWB model, using data from a life events study (N = 401) and an organisational climate study (N = 679). Previous validation studies, using Confirmatory Factor Analysis (CFA), have identified alternative PWB models, but limitations include the use of shorter scale versions with items relating to a number of life domains within the same PWB factor, and failure to examine the influence of participants’ socio-demographic characteristics on PWB. In this study, Exploratory Factor Analysis (EFA) mostly found consistency in the PWB items and structure between the two studies whereby a 3-factor model delineated between items relating to Autonomy, Positive Relations and a super-ordinate factor comprising the other PWB factors. Using CFA, Goodness of Fit indices reached acceptable levels for the adjusted PWB model identified by the EFA, whilst differences between adjusted models of PWB previously identified in the literature were hardly evident. Post-hoc analysis by gender demonstrated socio-demographic effects on the structure and items that comprise PWB. Further development of PWB measures is needed to reflect its hierarchical and multi-dimensional nature. In the scales’ current form, the construct validation of the PWB factors will continue to be problematic and will fail to adequately evaluate the nature and impact of PWB.     

Social Capital,Economic Development,and Homicide: A Cross-National Investigation

This article draws from an ongoing debate over explanations of homicide. Within this debate, we investigate the pro-social effects of civil society and social capital. Few cross-national studies explore whether elements of social capital either increase or decrease homicide. The cross-national work that does is often characterized by small, homogeneous samples and the use of inappropriate statistical techniques. Replicating elements of Lederman et al.’s (Econ Dev Cult Change 50:509–539, 2002) original study but with wave IV World Values Survey data and negative binomial regression, we find weak support for the beneficial consequences of social capital on homicide. One dimension of social capital, however, does exhibit a significant negative association with homicide rates, net of other influences: social activism. We also fail to support the Durkheimian hypothesis that the negative effect of social capital on homicide is conditional on modernization. We explore the implications of the findings along with avenues for future research.     

The Early Development Instrument: An Examination of Convergent and Discriminant Validity

The convergent and discriminant validity of the Early Development Instrument (EDI), a teacher-rated assessment of children’s “school readiness”, was investigated in a multicultural sample of 267 kindergarteners (53% male). Teachers evaluations on the EDI, both overall and in five domains (physical health/well-being, social competence, emotional maturity, language/cognition, communication/general knowledge), were related to direct, child-based assessments of performance on two standardized measures of school readiness, and measures of phonological awareness and early social competence. Regression analysis indicated that together the four comparison measures accounted for 36% of variance in overall EDI scores, each making a significant and unique contribution. Results supported the convergent validity of overall EDI scores but not the discriminant validity of EDI domain scores. Moreover, correlations between EDI scores and comparison measures varied widely across teachers, suggesting considerable individual differences in teacher’s ability to evaluate school readiness relative to direct, child-based assessments, and confirming that the EDI is more appropriate for deriving inferences at higher aggregated levels such as community or region. The validation of EDI domain scores remains an important challenge in future research.     

Comparison of Urban Housing Satisfaction in Modern and Traditional Neighborhoods in Edirne,Turkey

Numerous cross-cultural studies have focused on certain aspects of urban housing conditions and their social consequences. However, most data on housing satisfaction is restricted to Western countries. Relatively little comparison has been made between these findings and those in developing areas where rapid urbanization is occurring and where concomitant problems in urban housing are emerging. Moreover, since primary cities of developing countries in the initial stage of economic development have received extensive attention in urban housing research, it would be interesting to examine a non-primary city where relatively good standards of living have been achieved. Thus, this study investigates people’s housing satisfaction in modern and historical neighborhoods. This paper addresses some conceptual and measurement issues related to the study of housing satisfaction. We build a conceptual model, which links the multiple dimensions of housing satisfaction, measured by a modified version of Bardo and Dokmeci’s (1992, Genetic, Social and General Psychology Monographs 118(3)) housing satisfaction scale, in a causal sense. An empirical examination of the model in traditional and modern neighborhoods reveals that social and environmental living conditions positively influence overall housing satisfaction. The results also indicate that the drivers of the social and environmental living conditions constructs differ between traditional and modern neighborhoods.     

Exploratory Study on Drug Users’ Perspectives on Quality of Life: More than Health-Related Quality of Life?

In drug treatment outcome literature, a focus on objective and socially desirable indicators of change (e.g. no drug use) has predominated, while outcome indicators that are important for drug users themselves (e.g. quality of life, satisfaction with treatment) have largely been neglected. Nonetheless, Quality of Life (QoL) has become an important concept to evaluate effectiveness of treatment in mental health care research and disability studies. Given the almost exclusive focus on Health-related Quality of Life (HRQOL) in substance abuse research and the neglect of clients’ perspectives in this field, we explore in this study the concept of QoL as perceived by drug users. Focus group discussions (n = 9) were organised in various treatment settings and community services for drug users in the region of Ghent, Belgium to identify important dimensions of QoL and their interpretation by drug users. Data were clustered and analysed based on the theoretical framework of Robert Schalock (Quality of life. Volume 1: Conceptualization and measurement, 1996). The domains ‘personal relationships’, ‘social inclusion’ and ‘self-determination’ were discussed most frequently by the participants. They stressed the importance of a supportive social network in particular. It can be concluded that QoL is not primarily associated by drug users with health and it involves much more than the aspects typically represented in measures of HRQOL.     

Investigating the Substantive Aspect of Construct Validity for the Satisfaction with Life Scale Adapted for Children: A Focus on Cognitive Processes

The objective of this study was to investigate the substantive aspect of construct validity of the Satisfaction with Life Scale adapted for Children (SWLS-C; Gadermann et al. in Soc Indic Res 96:229–247, 2010). Specifically, the study examined the cognitive processes of children when responding to the items of the SWLS-C to find out how they interpret and respond to the items. Think-aloud protocol interviews were conducted with 55 students in grades 4–7 (58% girls, mean age of 11 years, ranging from 8.8 to 12.8 years) and content analysis was used to analyze the data. The findings indicate that most children had no difficulty in understanding the items, and used mainly two strategies for responding to the items: (1) an absolute strategy, in which children used absolute statements to indicate the presence or absence of something that is important for their judgment of their satisfaction with life, and (2) a relative strategy, in which children used relative or comparative statements. In the absolute statements, children primarily referred to social relationships, personal characteristics, time use, and possessions. In the relative statements, the children primarily compared what they have to (a) what they want (b) what they had in the past, (c) what other people have, and (d) what they feel they need. Furthermore, most children considered it important that information on their life satisfaction is obtained. The results are discussed with regard to multiple discrepancies theory (Michalos in Soc Indic Res 16:347–413, 1985) and previous empirical findings. The results provide insights into children’s cognitive processes when responding to items on life satisfaction and provide validity evidence that the SWLS-C is an appropriate measure to assess life satisfaction in children of this age.     

India’s Social Development in a Decade of Reforms: 1990–91/1999–2000

The economic reforms initiated in India in 1991 have brought about visible upliftment of economic conditions of the country. This paper examines if the economic process is associated with an enhancement of India’s social development in equal measure in the reform decade of nineties. Ray (1989) considered thirteen social indicators of India and constructed the country’s social development index (SDI) as a certain weighted average of the selected indicators, for the years between 1950–51 and 1975–76. The present work broadly follows Ray (1989) in tracking the social development in the decade of economic reforms from 1990–91 to 1999–2000. The movement of SDI in nineties has been compared with the movements of India’s per capita income and plan expenditure on the social sector. The results show that though there has been growth in the social sector, it is not by far quite encouraging and perhaps more needs to be done in the social sector. The views expressed in the article are those of the author and not of the institution he serves.     

Construct Validation and Application of a Common Measure of Social Cohesion in 33 European Countries

The aim of the paper is to assess the construct validation of a multidimensional measure of social cohesion which is well theoretically grounded and has an equivalent/comparable interpretation across all European countries. Up-to-now published research on social cohesion is deficient in either one or both of these important aspects. This paper attempts to cover this gap. The task is accomplished in two steps. In the first step, we conceptualize social cohesion, flowing mainly from Bernard and Chan’s definitions of social cohesion. Based on this theoretical framework we operationalize social cohesion and derive a set of intermediate indicators in the data. By return we verify whether these indicators empirically reflect/corroborate the multidimensional structure of the concept proposed by the theory. In the second step, we examine whether the obtained intermediate indicators of social cohesion form the same constructs across countries and whether they can yield a cross country equivalent measure of social cohesion. To test the validity of the theory we use multidimensional scaling and confirmatory factor analysis. Both models are able to verify the equivalence of the structural results between groups (i.e. countries). Confirmatory factor analysis produces further meaningful measures of these constructs. The analyses are based on the data from the 1999 European Values Study (EVS). The outcomes of the analyses reveal that, firstly, the existence of the multifaceted construct of social cohesion suggested by the theory has been corroborated by empirical analysis of the EVS data (i.e. social cohesion consists of components of formal and substantial relationships and political and socio-cultural domains). Secondly, the proposed constructs measuring social cohesion are equivalent across all analysed countries and thus allow the calculation of internationally comparable national scores of social cohesion. Application of the aggregate measures at the country level will illustrate the interest of the approach for further research.     

Investigating Validity Evidence of the Satisfaction with Life Scale Adapted for Children

This study introduces the Satisfaction with Life Scale adapted for Children (SWLS-C) and presents psychometric findings regarding its validation. The SWLS-C was adapted from the Satisfaction with Life Scale (SWLS; Diener et al. 1985), which is one of the most commonly used measures to assess satisfaction with life in adults. Three subject matter experts adapted the SWLS by changing the wording of the item stem and response format in order to make it more understandable for children. A stratified random sample of 1,233 students (48% girls) in grades 4–7 (mean age 11 years and 7 months) provided data on the SWLS-C and measures of optimism, self-concept, self-efficacy, depression, emphatic concern, and perspective taking. The SWLS-C demonstrated a unidimensional factor structure and high internal consistency. Furthermore, differential item functioning and differential scale functioning analyses indicated that the SWLS-C measures satisfaction with life in the same way for different groups of children (i.e., with regard to gender, first language learned at home—English vs. other language(s) than English—and across different grades) at the item and at the scale level. Associations between scores on the SWLS-C and demographic variables were statistically non-significant or of small effect size. In addition, the SWLS-C showed evidence of convergent and discriminant validity in relation to the other measures. Our results indicate that the SWLS-C is a psychometrically sound instrument that demonstrated evidence of construct validity for this age group. Limitations and future directions are discussed.     

Investigating Adolescent Health-Related Quality of Life: From a Self-Identity Perspective

The aim of the study was to investigate the relation between self-identity and health-related quality of life (HRQOL) in adolescence. This study assumed that four aspects (i.e. personal, social, ability, and academic identity) of identity firmness could predict adolescent’s HRQOL more than four aspects of identity importance. Meanwhile, this study also hypothesized that psychological and social relationships domains of HRQOL could be largely explained by the concept of identity firmness. One hundred and ninety-eight female adolescents participated in this study. Each participant completed a booklet containing the measures of the Questionnaire of revised edition of identity importance (QII-R), the Questionnaire of revised edition of identity firmness (QIF-R), the WHOQOL-BREF Taiwan version, and Visual-analogue HRQOL items. This study administered multiple regression analyses to test our hypotheses. The results demonstrated the concept of identity firmness could predict adolescent’s HRQOL more than the concept of identity importance. The results also supported our hypotheses that psychological and social relationships domains of HRQOL could be largely explained by the concept of identity firmness. In general, this study supported our hypotheses that the concept of identity firmness is a crucial predictor to adolescent’s HRQOL.     

The Volunteer Satisfaction Index: A Validation Study in the Chinese Cultural Context

Using a Hong Kong-sourced sample of 261 participants, this study set out to validate the Volunteer Satisfaction Index (VSI) in the Chinese cultural context and to evaluate its psychometric properties. The VSI was originally developed by Galindo-Kuhn and Guzley (2001) to measure the outcomes of volunteer experiences. In this study, exploratory factor analysis (EFA) yielded a different factor structure from that proposed by the scale developer. The three factors found were personal gain, relationship within organization and relationship with peers. Cronbach’s alpha values were high for all three subscales. Results from correlation and regression analysis also confirmed the construct and criterion-related validity of the scale. Thus, the reliability and validity of the scale were confirmed. Implications for the assessment of volunteer satisfaction and further directions for cross-cultural studies on related topics are discussed.     

Validation of Multilevel Constructs: Validation Methods and Empirical Findings for the EDI

The purposes of this paper are to highlight the foundations of multilevel construct validation, describe two methodological approaches and associated analytic techniques, and then apply these approaches and techniques to the multilevel construct validation of a widely-used school readiness measure called the Early Development Instrument (EDI; Janus and Offord 2007). Validation evidence is presented regarding the multilevel covariance structure of the EDI, the appropriateness of aggregation to classroom and neighbourhood levels, and the effects of teacher and classroom characteristics on these structural patterns. The results are then discussed in the context of the theoretical framework of the EDI, with suggestions for future validation work.     

The Relative Importance of Health

This article seeks to extend Michalos’ [Social indicators research and health-related quality of life (QoL) research. Social Indicators Research, 65, 27–72, 2004] discussion on bridging social indicators research and health-related QoL (HRQoL) research through an examination of (1) the relative importance of satisfaction with one’s own health to another common measure of QoL—Life satisfaction, and (2) the relative importance of health in relation to other major life domains. Using data from two surveys, this article found that individuals may perceive health as most important in relation to other major life domains but satisfaction with one’s own health may not necessarily be the most important determining factor (in relation to satisfaction with other major life domains) of QoL as measured by life satisfaction. These findings support Michalos’ (Social indicators research and HRQoL research. Social Indicators Research, 65, 27–72, 2004) call for caution regarding the interpretation of research results on HRQoL since many HRQoL measures are measures of satisfaction with one’s own health and should not be considered as measures of QoL.     

A Translation and Validation Study of the Life Orientation Test Revised in the Greek Speaking Population of Nurses among Three Hospitals in Athens and Ioannina

The life orientation test-revised (LOT-R) (Scheier et al. in Journal of Personality and Social Psychology 67:1063–1078, 1994) is a brief measure for assessing dispositional optimism. The aim of this study was to develop a Greek language version of the LOT-R and to assess the instrument’s psychometric properties. The LOT-R was translated and culturally adopted in Greek language, and the final version was administered, along with a questionnaire consisting socioeconomic characteristics and a single item measuring optimism, to 276 Greek speaking, hospital nurses (222 female, 54 male), aged 22–65 years (mean 37.8, SD 8.3). Results showed that the LOT-R has good internal consistency (Cronbach’s α = .71 and item total correlation coefficients from .27 to .73, a unitary structure, and stability over a 3-months period (r = .66). Moreover, the Greek version of the scale exhibited good convergent validity with single-item optimism scale (r = .73). Principal components analysis revealed a two-factor structure representing the constructs of optimism and pessimism. The Greek life orientation test revised appears to be a valid tool in assessing dispositional optimism in Greek speaking people and is expected to facilitate the examination of optimism in Greek speaking populations.     

Fifty Years After the Social Indicators Movement: Has the Promise Been Fulfilled?

In spite of its currency both in academic research and political rhetoric, there are numerous attempts to define and conceptualize the social cohesion concept but there has been paid little attention to provide a rigorous and empirically tested definition. There are even fewer studies that address social cohesion in a framework of cross-cultural validation of the indicators testing the equivalence of the factorial structure across countries. Finally, as far as we know there is no study that attempt to provide an empirically tested multilevel definition of social cohesion specifying a Multilevel Structural Equation Model. This study aims to cover this gap. First, we provide a theoretical construct of social cohesion taking into account not only its multidimensionality but also its multilevel structure. In the second step, to test the validity of this theoretical construct, we perform a multilevel confirmatory factor analysis in order to verify if the conceptual structure suggested in first step holds. In addition, we test the cross-level structural equivalence and the measurement invariance of the model in order to verify if the same multilevel model of social cohesion holds across the 29 countries analysed. In the final step, we specify a second-order multilevel CFA model in order to identify the existence of a general factor that can be called “social cohesion” operating in society that accounts for the surface phenomena that we observe.     

Exploring Time Perspective in Greek Young Adults: Validation of the Zimbardo Time Perspective Inventory and Relationships with Mental Health Indicators

In this article we examine the factorial structure of the Greek version of the Zimbardo Time Perspective Inventory (ZTPI; Zimbardo and Boyd in J Personal Soc Psychol 77:1271–1288, 1999), in a sample of 337 university students, using principal axis factoring (PAF) with oblique rotation, and its dimensionality using parallel analysis. Moreover, we evaluate the internal consistency reliability, the convergent validity (through associations with mental health indicators such as dispositional optimism, self-esteem, trait anxiety, depression, and proactive coping), as well as discriminant, and differential validity of this instrument. The results indicated that the ZTPI had a 5-factor structure (past negative, past positive, present fatalistic, present hedonistic, future). Correlational analyses indicated that an aversive view of the past, and a fatalistic attitude toward life were positively and significantly associated with trait anxiety and depression, while they were negatively correlated with self-esteem, proactive coping, and dispositional optimism. Future time perspective was positively associated with proactive coping, whereas a positive attitude toward the past was negatively associated with depression and trait anxiety. Psychometric properties of the five ZTPI scales were satisfactory (Cronbach’s alphas ranging from .710 to .845), thus facilitating the robust investigation of time perspective in Greek speaking populations. However confirmatory factor analyses revealed that the “positive attitude toward the past” dimension might not be a good indicator of time perspective. We discuss the theoretical implications of these findings for future studies of time perspective.