Comparing poverty and deprivation dynamics: Issues of reliability and validity

In this paper we seek to establish if earlier findings relating to the relationship between income poverty persistence and deprivation persistence could be due to a failure to take measurement error into account. To address this question, we apply a model of dynamics incorporating structural and error components. Our analysis shows a general similarity between latent poverty and deprivation dynamics. In both cases we substantially over-estimate the probability of exiting from poverty or deprivation. We observe a striking similarity across dimensions for both observed and latent outcomes. In both cases levels of poverty and deprivation persistence are higher for the latent case. However, there is no evidence that earlier results relating to the differences in the determinants of poverty and deprivation persistence are a consequence of differential patterns of reliability. Taking measurement error into account seems more likely to accentuate rather than diminish the contrasts highlighted by earlier research. Since longitudinal differences relating to poverty and deprivation cannot be accounted for by measurement error, it seems that we must accept that we are confronted with issues relating to validity rather than reliability. Even where we measure these dimensions over reasonable periods of time and allow for measurement error, they continue to tap relatively distinct phenomenon. Thus, if measures of persistent poverty are to constitute an important component of EU social indicators, a strong case can be made for including parallel measures of deprivation persistence and continuing to explore the relationship between them.     

Linked Ego Networks: Improving estimate reliability and validity with respondent-driven sampling

Respondent-driven sampling (RDS) is currently widely used for the study of HIV/AIDS-related high risk populations. However, recent studies have shown that traditional RDS methods are likely to generate large variances and may be severely biased since the assumptions behind RDS are seldom fully met in real life. To improve estimation in RDS studies, we propose a new method to generate estimates with ego network data, which is collected by asking respondents about the composition of their personal networks, such as “what proportion of your friends are married?”. By simulations on an extracted real-world social network of gay men as well as on artificial networks with varying structural properties, we show that the precision of estimates for population characteristics is greatly improved. The proposed estimator shows superior advantages over traditional RDS estimators, and most importantly, the method exhibits strong robustness to the recruitment preference of respondents and degree reporting error, which commonly happen in RDS practice and may generate large estimate biases and errors for traditional RDS estimators. The positive results henceforth encourage researchers to collect ego network data for variables of interests by RDS, for both hard-to-access populations and general populations when random sampling is not applicable.     

Assessing the validity of a single-item HIV risk stage-of-change measure

This study examined the validity of a single-item measure of HIV risk stage of change that HIV prevention contractors were required to collect by the California State Office of AIDS. The single-item measure was compared to the more conventional University of Rhode Island Change Assessment (URICA). Participants were members of Los Angeles County-defined behavioral risk groups recruited from a mobile HIV testing program (N = 123). The study found low correspondence between participants' stage of change on the single-item and the URICA, suggesting that they do not assess the same construct, and that the single item may not provide a valid measure. The current data suggests that the single-item measure should be revised or abandoned in favor of more conventional stage-of-change measures.     

Fear of intimacy with helping professionals scale: reliability and validity of english and mandarin versions

In this study the authors examine the reliability and validity of the Fear of Intimacy with Helping Professionals Scale (FIS-HP) with Chinese (N = 150) and American (N = 145) elderly persons. Factor analysis using principal component analysis with a varimax rotation was used to examine the FIS-HP factor structure for both samples. A three factor solution emerged for both samples. The FIS-HP has acceptable internal consistency reliability with both the United States and China samples. Correlation analysis supported five of the six hypotheses related to convergent validity. English and Mandarin versions of the scale are presented.     

Portfolio assessment in a collaborative program evaluation: the reliability and validity of a family literacy portfolio

Evaluations increasingly use open-ended assessments, such as portfolios, to gather information about program impact. Portfolios, however, present challenges in achieving acceptable levels of reliability and often lack evidence to support validity. This article describes the components of a portfolio assessment system developed in a collaborative evaluation of a family literacy program. Staff of a family literacy program and two evaluators collaborated to develop the assessment and the tools that frame the review of the portfolios. Rubrics, exemplar portfolios, portfolio logs, and lists of ‘look-fors’ were developed to guide raters in the scoring and to improve the reliability of scores and validity of decisions about the program. Inter-rater reliability for the portfolio assessment was 0.79 for holistic scores and 0.74 for analytic scores. With respect to validity, holistic and analytic scores correlated with criterion measures from 0.50 to 0.59, providing initial evidence of a common construct.     

The modified cognitive constructions coding system: reliability and validity assessments

The cognitive constructions coding system (CCCS) was designed for coding clients' expressed problem constructions on four dimensions: intrapersonal-interpersonal, internal-external, responsible-not responsible, and linear-circular. This study introduces, and examines the reliability and validity of a modified version of the CCCS-a version that involves coding clients' problem constructions at the level of speech turn. Results indicate that the modified CCCS generated reliable ratings of clients' problem constructions in 15 sessions of family therapy. Validity was established by comparing 71 subjects' own ratings of over 250 self-generated problem constructions to the ratings of trained CCCS coders. The modified CCCS allows for identifying moment-to-moment changes in clients' problem constructions and, ultimately, examining the impact of therapist behaviors on such changes.     

Effects on reliability and validity of egocentered network measurements

This paper examines the reliability and validity of egocentered networks. Reliability and validity are estimated by the multitrait-multimethod (MTMM) approach. A split ballot MTMM design [Saris, W.E., 1999. Forced choice or agree/disagree questions? An evaluation by the split ballot MTMM experiment. In: Proceeding of the Meeting of the IRMCS, pp. 122–146; Kogovšek, T., Ferligoj, A., Coenders, G. Saris, W. E., 2002. Estimating the reliability and validity of personal support measures: full information ml estimation with planned incomplete data. Social Networks 24, 1–20] is used, in which separate groups of respondents received different combinations of two methods. The effect of factors such as the methods used and the personal characteristics of respondents that can affect the quality of data was estimated by a meta analysis.Measurement method, type of question, network size, age, gender, extraversion and emotional stability all had statistically significant effects on the validity of measurement. After the list of alters is obtained with name generators, name interpreter questions can be asked in two ways. One way (“by alters”) is to take each alter individually and to ask all the questions about him/her, going alter by alter until the end of the list of alters. The other way (“by questions”) is to take the question and ask this question for all alters on the list, going question by question until the end of the list of name interpreter questions. Telephone interviewing (both by alters and by questions) gave more valid measurements than face-to-face interviews.Behavioral questions were more valid than questions with emotional content. The characteristics of ties were more validly measured in smaller networks. With reference to respondents’ personal characteristics younger respondents, men, extraverted and emotionally stable respondents all had more valid measurements. Reliability was significantly affected by the measurement method, the type of question and age. The telephone/by alters method was the most reliable measurement method. Behavioral questions were more reliable than questions with emotional content. Measurements among younger respondents were also more reliable.     

The impact of respondent attentiveness on reliability and validity

This study aims to assess whether respondent inattentiveness causes systematic and unsystematic measurement error that influences survey data quality. To determine the impact of (in)attentiveness on the reliability and validity of target measures, we compared respondents from a German online survey (N = 5205) who had passed two attention checks with those who had failed. Our results show that inattentiveness induces both random and systematic measurement error, which impacts estimates of the reliability and validity of multi-item scales. In addition, we conducted a sensitivity analysis, which revealed that the impact of inattentiveness on analyses can be substantial.     

An assessment of the validity and reliability of the SOGS-RA

The present is a validation study seeking to determine the degree of confidence that can be placed on inferences about problem gambling among adolescents in the Atlantic provinces, based on their South Oaks Gambling Screen-Revised for Adolescents (SOGS-RA) scores. The major source of data was a 1998 survey of 13,549 students in junior and high school of the public school systems of the 4 Atlantic provinces of Canada. The SOGS-RA was found to have adequate stability and internal consistency reliability. Statistically significant gender differences were demonstrated relative to endorsement and construct validity. Regarding the latter, this study shows that the existing cut-point of the SOGS-RA score for problem gambling identifies as problem gamblers, markedly different proportions of male than female daily gamblers. Regarding construct validity in relation to the Atlantic Alcohol and Drug Risk Continua, this study suggests that while statistically highly significant, the association between problem gambling and substance-related problems may be of low clinical significance. Regarding criterion validity, there is an urgent need to perform the types of enquiry that will allow clarification about how adolescent problem gambling is conceptualized, by adults versus adolescents, by males versus females, and from a clinical versus a public or population health perspective.     

Self-reported mobility and instrumental activities of daily living: test-retest reliability and criterion validity

To determine the test-retest reliability and criterion validity of self-reported function in mobility and instrumental activities of daily living (IADL) in older adults, a convenience sample of 70 subjects (72.9 ± 6.6 yr, 34 male) was split into able and disabled groups based on baseline assessment and into consistently able, consistently disabled, and inconsistent based on repeat assessments over 2 weeks. The criterion validities of the self-reported measures of mobility domain and IADL-physical subdomain were assessed with concurrent baseline measures of 4 mobility performances, and that of the self-reported measure of IADL-cognitive subdomain, with the Mini-Mental State Examination. Test-retest reliability was moderate for the mobility, IADL-physical, and IADL-cognitive subdomains (κ = .51-.66). Those who reported being able at baseline also performed better on physical- and cognitive-performance tests. Those with variable performance between test occasions tended to report inconsistently on repeat measures in mobility and IADL-cognitive, suggesting fluctuations in physical and cognitive performance.     

Estimating the reliability and validity of personal support measures: full information ML estimation with planned incomplete data

Egocentered networks are common in social science research. Here, the unit of analysis is a respondent (ego) together with his/her personal network (alters). Usually, several variables are used to describe the relationship between egos and alters.In this paper, the aim is to estimate the reliability and validity of the averages of these measures by the multitrait–multimethod (MTMM) approach. This approach usually requires at least three repeated measurements (methods) of the same variable (trait) for model identification. This places a considerable burden on the respondent and increases the cost of data collection.In this paper, we use a split ballot MTMM experimental design, proposed by Saris (1999), in which separate groups of respondents get different combinations of just two methods. The design can also be regarded as having a planned missing data structure. The maximum likelihood estimation is used in the manner suggested by Allison (1987) of a confirmatory factor analysis model for MTMM-designs specified in Saris and Andrews (1991). This procedure is applied to social support data collected in the city of Ljubljana (Slovenia) in the year 2000.     

Assessing nontraditional couples: validity of the marital satisfaction inventory--revised with gay,lesbian, and cohabiting heterosexual couples

Thirty-one gay male couples and 28 lesbian couples were compared with 36 cohabiting heterosexual couples using the Marital Satisfaction Inventory-Revised (MSI-R), a multidimensional measure of relationship functioning intended for use with both traditional and nontraditional couples. Analyses of scales' internal consistency and factor structure supported the construct validity of this measure with nontraditional couples. Analyses of mean profiles on the MSI-R indicated that cohabiting opposite-gender and same-gender couples were more alike than different, and were more similar to nondistressed samples of married heterosexual couples from the general community than to couples in therapy. Implications of current findings for clinical assessment and intervention are considered, and directions for future research are proposed.     

The reliability and validity of the physical activity survey in long-term care

The purpose of this study was to develop and test a measure of physical activity for residents in long-term-care facilities, the Physical Activity Survey in Long-Term Care (PAS-LTC). Sixty-six activities are included in the PAS-LTC: routine physical activity, personal-care activities, structured exercise, recreational activities, caretaking activities, and repetitive activities. The study included 13 residents in a long-term-care facility, most of whom were women (62%), with an average age of 84 years (+/- 6.0) and an average Mini Mental State Examination score of 6 (+/- 6.9). There was evidence of interrater reliability of the PAS-LTC with intraclass correlations of .83-.94. There was some evidence of validity of the measure with statistically significant correlations between PAS-LTC recorded during the evening and night shifts and the number of counts of activity per the ActiGraph (r = .60 and r = .57, respectively, p < .05) and the calories estimated (r = .58 and r = .60, respectively, p < .05). The PAS-LTC completed during the day shift and total activity based on the PAS-LTC showed nonsignificant correlations of .40 or greater with the ActiGraph activity counts and calories.     

The composite abuse scale: further development and assessment of reliability and validity of a multidimensional partner abuse measure in clinical settings

Absence of a well-validated comprehensive partner abuse questionnaire has been a major methodological issue in domestic violence research. A new multidimensional measure of partner abuse, the Composite Abuse Scale (CAS), has four dimensions: Severe Combined Abuse, Emotional Abuse, Physical Abuse, and Harassment. A general practice patient sample (N = 1,836) has been used in the development and testing of CAS. Factor analyses in this current study confirmed the four dimensions from a preliminary nurses sample study and resulted in a final scale of 30 items consisting of acts of physical, emotional, and sexual abuse. These four factors exhibited good internal reliability (Cronbach's alpha > 0.85) and the corrected item-total correlations were high (> 0.5). Evidence of criterion and construct validity is presented.     

College students coping with interpersonal stress: Examining a control-based model of coping

Objective: The ways that college students cope with stress, particularly interpersonal stress, may be a critical factor in determining which students are at risk for impairing mental health disorders. Using a control-based model of coping, the present study examined associations between interpersonal stress, coping strategies, and symptoms. Participants: A total of 135 undergraduate students from 2 universities. Methods: Interpersonal stress, coping strategies, depression, anxiety, and somatization were assessed via self-report. Results: Students reporting more interpersonal stress reported more depression, anxiety, and somatization, and they reported less use of engagement coping strategies and greater use of disengagement coping strategies. Engagement coping strategies accounted for a significant portion of the association between interpersonal stress and mental health symptoms. Unexpectedly, coping strategies did not moderate the association between stress and mental health symptoms. Conclusion: Interventions designed to improve students' coping strategies may be an effective way to reduce mental health problems on college campuses.     

Towards a parental muda for new Basque speakers: Assessing emotional factors and language ideologies

The concept of muda refers to how specific biographical moments can precipitate changes in the speaker's linguistic repertoire (Pujolar & Gonzàlez, 2012). In recent years, more inclusive or participatory approaches to intergenerational transmission in language revitalization contexts have been encouraging all parents, including those with low proficiency in the minority language, to participate in their child's language acquisition. This article examines a Basque‐ language campaign that instructs low‐proficiency parents to adopt child‐directed speech in Basque to mould affective orientations in the home environment. Drawing on a case study, I explore the complications of new speakerhood, especially the difficulties of bringing about a parental muda. I demonstrate how mudas are traversed by competing ideologies of language and language socialization. In disrupting monolingual ideologies, participatory approaches which aim at increasing the symbolic value of the language clash with the attitudes of speakers who view these mudas as inconsequential for achieving normalization.     

The NTEE classification system: tests of reliability/validity in the field of higher education

An analysis of the field of higher education reveals a surprisingly large number of classification errors within the (US) National Taxonomy of Exempt Entities database. Of the nearly 4,000 non-profit entities coded within the universe of institutions of higher education, we estimate that approximately 60 per cent were incorrectly included (Type II errors). Institutions incorrectly excluded from the higher education universe (Type I errors) represent roughly 10 per cent of the institutions coded correctly. These errors result primarily from assigning codes on the basis of the names of organisations (for example, College Park Towers is classified as a college, when it is in fact a housing complex for senior citizens) and from trying too hard to assign codes in ambiguous situations. The consequences can be significant for the users of these data, and we recommend raising bright warning flags while simultaneously enlisting the help of the entire non-profit sector in improving the classification coding process.The authors are with The Andrew W. Mellon Foundation. In addition to our colleagues on the staff of the Foundation, we wish to thank Helmut Anheier, Ted Bozovich, Harvey Dale, Virginia Hodgkinson, Stephen Noga and Christopher Toppe for helpful suggestions.