Sexual behaviour in aids‐related research: Reliability and validity of recall and diary measures

In order to assess the reliability and validity of two data collection instruments for measuring sexual practices in homosexual men, we administered a recall data collection instrument to 30 sexually active men, and a diary instrument to a subset of 19 participants. Each instrument covered a period of one month. For the recall instrument, the correlation coefficients between the test‐retest showed a good level of reliability for a number of infrequent sexual practices but was poor for frequent sexual practices. Correlation coefficients were calculated for sexual practices recorded in the first two and the second two weeks for both the recall and the diary. Generally, the level of agreement between the first and second two weeks for sexual practice data recorded in the diary was lower than the recall questionnaire. While the reliability of the recall method was found to be limited to certain sexual practices, it was concluded to be the most reliable and efficient method presently available for the collection of important high‐risk AIDS‐related sexual activities for a one‐month period.     

Using egocentered networks in survey research. A methodological preview on an application of social network analysis in the area of family research

Can questions about families be adequately answered with household survey data? This question leads us to the problem to defining the family. To help us answer this question we use network analysis as a tool to examine the family without a preconceived definition. To get information about the population of family configurations in Federal Republic of Germany, we choose a survey technique that measured ego-centered networks (10–12 network generators, 4–6 network interpreters). Based on two pretests (n₁ = 98, n₂ = 534) and the main survey (n = 10,000), we discuss some problems encountered during data collection. We also examine the validity and reliability of the data. We used variations of the measurement instrument and asked a group of 99 respondents twice to test reliability. Then we compare our results with census data and compatible US and German studies to test validity. The results show: (1) It is possible to get reliable network data by using survey technique. (2) Size of networks, named persons, named relations are reasonably stable. (3) The instrument to measure networks is constant against minor changes in formulation of questionnaire and against variation of collecting techniques. These findings hold both on the macro level (aggregate data), and also on the micro level (person related data).     

Measuring social networks cross-culturally

This paper explores the reliability and validity of a cross-cultural survey approach to network measurement using data from two ethnic groups in rural West Africa. Over a 2-week test-retest period, no significant differences in network size were detected. Network composition was less stable over the same period, particularly among peripheral versus core network members. Comparisons with support networks identified through lifecycle interviews and 24 h recall support the construct validity of the survey approach in both ethnic groups.     

Respondent mood and the instability of survey network measurements

The affect infusion model [Forgas, 1999, Handbook of Cognition and Emotion. Wiley, Chichester, p. 591] was used to hypothesize and explain relations between respondents’ moods and the instability of survey measurements of social support ties in social networks of high school students. Two relative measures of network measurement instability were defined. The sensitivity of these to the change in respondent mood was observed by repeated measurements. Four types of social support were measured twice, under different conditions using both recognition and free recall data collection techniques. Changes in respondent mood and the data collection technique were the best two predictors of measurement instability among several survey design variables included in the survey.     

Zu subtil?

Recent research on inter-ethnic relations in Germany discusses fundamental changes in patterns of attitudes and latent discrimination on the part of Germans towards ethnic minorities. It is argued that “modern”, rather subtle forms of attitudes and social distances emerged which cannot be measured with conventional indicators. Based on data from a Eurobarometer survey, Pettigrew and Meertens developed two scales — the Subtle Prejudice Scale and the Blatant Prejudice Scale — to close the gap. The aim of this article is to find out whether these scales are useful as supplement or substitution for commonly used measurement instruments in Germany. On the basis of data collected in 1999 the reliability and validity of the scales are tested. Moreover, the relations between the newly arranged indicators and the often used measurement instruments from the German General Social Survey (ALLBUS) are analysed for the first time. The empirical results show that especially the reliability and validity of the Subtle Prejudice Scale is questionable. It is concluded that the new scales in their current form provide no useful alternative to the commonly used measurement instruments.     

Determining the Appropriateness of the “What If” Situations Test (WIST) with Turkish Pre-Schoolers

Measurement instruments are needed to assess the child’s sexual abuse prevention program. The purpose of the study was to determine the reliability and validity of the WIST (What If Situations Test) for Turkish culture. Participants were children of the 3–6 age group attending pre-school education institutions and the sample size was identified by means of a power analysis. Seventy children were identified as the sample with 0.85 power and 0.05 type I error according to the power analysis. Language validity, content validity, internal validity coefficient (Cronbach alpha coefficient), and test–retest analyses were conducted in terms of validity and reliability in the scope of efforts for adaptation to Turkish culture. Firstly, Kendall W = 0.83 was the score for the expert opinions concerning the content validity of the language validity scale. It was found that the Cronbach alpha coefficients were between 0.68 and 0.90 for the scale sub-dimensions of appropriate and inappropriate recognition, saying, doing, telling, and reporting. The test–retest reliability of the scale was found to be r = 0.89 and the test–retest reliabilities for the sub-dimensions (appropriate recognition, inappropriate recognition, say skills, do skills, tell skills, and reporting skills) were between r = 0.48 and r = 0.92. The test–retest reliability for the Personal Safety Questionnaire (PSQ), as having complimentary items to the WIST, was found to be r = 0.82. The reliability and validity analysis of the ‘What If’ Situations Test (WIST), used to evaluate pre-schoolers’ skills regarding self-protection against sexual abuse, showed that the Test’s adaptation to Turkish culture was reliable and valid.     

Reliability of measures of centrality and prominence

This paper evaluates the reliability of measures of centrality and prominence of social networks among high school students. The authors present and discuss results from eight experiments. Four types of social support: (1) instrumental support, (2) informational support, (3) social companionship, and (4) emotional support—were measured three times within each class. Four measurement scales: (1) binary, (2) categorical, (3) categorical with labels and (4) line production—were applied. Reliability of in- and out-degree, in- and out-closeness, betweenness and flow betweenness was estimated by the Pearson correlation coefficient. Meta analysis of factors affecting the test-retest reliability of measures of centrality and prominence was done by multiple classification analysis. Results show that,

• -Global measures (considering direct and indirect choices) are more sensitive to measurement errors than local measures (considering only direct choices).
• -In-measures are more stable than out-measures.
• -Among types of social support, emotional support gives the least stable measures of centrality and prominence, whereas social companionship gives the most stable results.
• -The reliability of centrality and prominence measures is higher when the network is denser.

     

Construction of a survey to assess workload and fatigue among AMT operators in Mexico

Operators of machinery classified as Advanced Manufacturing Technology (AMT) are exposed to high levels of workload and fatigue. However, only few studies have been conducted on this topic in Hispanic-American countries workers. Several instruments be used to assess workload and fatigue; however, only few of them have been adapted to Spanish language. This paper reports on the development and validity testing of a survey instrument in Spanish, aiming to subjectively assess workload and fatigue among AMT operators in Mexico. Method: After an exhaustive literature review in search of already available measurement instruments, they were adapted for content and later translated into Spanish; a pilot test was conducted to evaluate validity and reliability; afterwards appropriate modifications were made to the testing instruments. Final version of the instrument was applied to a group of 121 operators of CNC lathes. Reliability was analyzed using KMO and Cronbach alpha indices. Results: For the assessment of workload, both NASA-TLX and ISTAS 21 methods were incorporated to the survey instrument. As for fatigue assessment tools, these were SOFI-S, FAS and OFER questionnaires. Results show KMO value and Cronbach alpha above 0.6.Conclusions. The survey instrument as designed, allows the collection of reliable and valid data regarding workload and fatigue among AMT operators in Mexico.     

Factorial Validity of the Alcohol Use Disorders Identification Test: Drunk Drivers versus College Students

ABSTRACT

Secondary analysis was undertaken on the data from two previous studies involving the Alcohol Use Disorders Identification Test (AUDIT). Test results for 257 multiple offender drunk drivers and 505 college drinkers were compared. Exploratory factor analysis were run to examine the factorial validity of the test's hypothesized constructs. Internal consistency reliability was 87 with drunk drivers and 76 with college drinkers. Factor analyses of the AUDIT failed to validate the existence of its three hypothesized constructs regardless of sample population; a review of Eigenvalues indicates that there is a two-factor structure to the test with these populations. These factors are labeled problem drinking and hazardous drinking. The test was generally reliable. It is suggested that when using rapid assessment instruments in social work practice settings like those in the original studies, that their conceptual structure be evaluated for context specific validity.     

Effects on reliability and validity of egocentered network measurements

This paper examines the reliability and validity of egocentered networks. Reliability and validity are estimated by the multitrait-multimethod (MTMM) approach. A split ballot MTMM design [Saris, W.E., 1999. Forced choice or agree/disagree questions? An evaluation by the split ballot MTMM experiment. In: Proceeding of the Meeting of the IRMCS, pp. 122–146; Kogovšek, T., Ferligoj, A., Coenders, G. Saris, W. E., 2002. Estimating the reliability and validity of personal support measures: full information ml estimation with planned incomplete data. Social Networks 24, 1–20] is used, in which separate groups of respondents received different combinations of two methods. The effect of factors such as the methods used and the personal characteristics of respondents that can affect the quality of data was estimated by a meta analysis.Measurement method, type of question, network size, age, gender, extraversion and emotional stability all had statistically significant effects on the validity of measurement. After the list of alters is obtained with name generators, name interpreter questions can be asked in two ways. One way (“by alters”) is to take each alter individually and to ask all the questions about him/her, going alter by alter until the end of the list of alters. The other way (“by questions”) is to take the question and ask this question for all alters on the list, going question by question until the end of the list of name interpreter questions. Telephone interviewing (both by alters and by questions) gave more valid measurements than face-to-face interviews.Behavioral questions were more valid than questions with emotional content. The characteristics of ties were more validly measured in smaller networks. With reference to respondents’ personal characteristics younger respondents, men, extraverted and emotionally stable respondents all had more valid measurements. Reliability was significantly affected by the measurement method, the type of question and age. The telephone/by alters method was the most reliable measurement method. Behavioral questions were more reliable than questions with emotional content. Measurements among younger respondents were also more reliable.     

A comparison of the reliability of questionnaire versus diary data

The crucial role that communication plays in organizations has led to the development of various methodologies to collect communications data. Two of the more common techniques are one-shot questionnaires and self-recording diaries completed at the time of the interaction. Questionnaires are often viewed as unreliable since they are based on the memory and perceptions of the participants. Diaries, on the other hand, are considered to be more reliable since they are completed immediately following an interaction when the event is still fresh in the individual's mind. This paper compares the cross-recorder reliability of these two data collection instruments. A reliability measure which indicates the degree to which an individual concurs as to the occurrence of interactions across all individuals is introduced. The results, based on studies in three organizations involving 49, 61 and 110 participants respectively, indicate that diary data are significantly more reliable than data obtained from questionnaires.     

THE DEVELOPMENT OF A RELATIONAL ETHICS SCALE*

In order for empirical testing to be performed systematically on family therapy theories, appropriate testing instruments must be developed and validated. The purpose of this study was to develep a reliable and valid scale for measuring the related constructs of relational ethics as described in contextual family therapy (Boszormenyi-Nagy & Krasner, 1986). Presented here is the five-stage procedure used in development in Relational Ethics Scale, data supporting the validity and reliability of the scale, and the final version of the instrument.     

Reliability and Validity of the IPAQ-L in a Sample of Hong Kong Urban Older Adults: Does Neighborhood of Residence Matter?

This study examined reliability and validity of the Chinese version of the International Physical Activity Questionnaire-Long Form (IPAQ-LC) in Chinese seniors, including moderating effects of neighborhood walkability and socioeconomic status (SES) on reliability and validity. The IPAQ-LC was interviewer-administered (n = 96), accelerometer and 7-day walk-diary data were collected (n = 94), and the IPAC-LC was readministered (N = 92). Acceptable reliability was found for all measures of physical activity (PA) overall and across different types of neighborhood. Participants from highly walkable neighborhoods were more reliable at estimating walking for transport. Participants from low-SES areas were less reliable at estimating leisure-time PA and sitting but more reliable at estimating transport-related walking. IPAQ-LC walking was significantly related to light- but not moderate-intensity accelerometry-based PA. It was moderately to strongly related to a 7-day diary of walking. The data imply slow-paced walking, probably due to age, climate, and terrain. The findings suggest that the IPAQ-LC's reliability and validity are acceptable in Chinese seniors.     

A New Test to Measure Emotion Recognition Ability: Matsumoto and Ekman's Japanese and Caucasian Brief Affect Recognition Test (JACBART)

In this article, we report the development of a new test designed to measure individual differences in emotion recognition ability (ERA), five studies examining the reliability and validity of the scores produced using this test, and the first evidence for a correlation between ERA measured by a standardized test and personality. Utilizing Matsumoto and Ekman's (1988) Japanese and Caucasian Facial Expressions of Emotion (JACFEE) and Neutral Faces (JACNeuF), we call this measure the Japanese and Caucasian Brief Affect Recognition Test (JACBART). The JACBART improves on previous measures of ERA by (1) using expressions that have substantial validity and reliability data associated with them, (2) including posers of two visibly different races (3) balanced across seven universal emotions (4) with equal distribution of poser race and sex across emotions (5) in a format that eliminates afterimages associated with fast exposures. Scores derived using the JACBART are reliable, and three studies demonstrated a correlation between ERA and the personality constructs of Openness and Conscientiousness, while one study reports a correlation with Extraversion and Neuroticism.     

News from the field

Two problems are always present in any research: reliability and validity, and this paper will attempt to assess reliability and validity in relation to dietary data. Reliability means stability of results over a period of time or in similar situations. (It is very difficult to find two identical situations, especially outside a laboratory, as some variation is always introduced.) Validity refers to the accuracy of techniques and results. It includes the idea of measuring methods and results against a standard or criterion.     

Readings in Evidence-Based Social Work

Abstract

This article describes the development, validity, and reliability of the Parent Empowerment and Efficacy Measure (PEEM). Development was guided by theory and by challenges faced by Pathways to Prevention family support staff who required a short, practical, and reliable measure of parent empowerment. The measure's psychometric properties were tested using data from 866 parents of children aged 5 to 12, living in high to low socioeconomic status areas. Principal factor analysis revealed a strong general dimension with high internal consistency (α = 0.92) that correlated at 0.60 or more with three validation measures, as well as the existence of two hypothesised subfactors (correlated at 0.78): efficacy to parent and efficacy to connect, each with internal consistencies of 0.85+. Test–retest reliability (n = 200) was 0.84. PEEM exhibits excellent convergent and concurrent validity and is a reliable tool for use in planning services, monitoring participant progress, and evaluating program effectiveness.     

The impact of respondent attentiveness on reliability and validity

This study aims to assess whether respondent inattentiveness causes systematic and unsystematic measurement error that influences survey data quality. To determine the impact of (in)attentiveness on the reliability and validity of target measures, we compared respondents from a German online survey (N = 5205) who had passed two attention checks with those who had failed. Our results show that inattentiveness induces both random and systematic measurement error, which impacts estimates of the reliability and validity of multi-item scales. In addition, we conducted a sensitivity analysis, which revealed that the impact of inattentiveness on analyses can be substantial.     

Comparison of two methods of measuring physical activity in South African older adults

The aim of this study was to assess the validity and reliability of the Yale Physical Activity Survey (YPAS) and the short version of the International Physical Activity Questionnaire (IPAQ) in older South African adults. The YPAS includes measures of weekly energy expenditure (EE) for housework, yard work, caregiving, exercise, and recreation. The IPAQ measures total time and EE during vigorous and moderate activity, walking, and sitting. The instruments were administered twice for test-retest reliability (men, n = 52, 68 +/- 5.4 years, and women, n = 70, 66 +/- 5.8 years). Data for criterion validity were obtained from accelerometers. YPAS reliability ranged from r = .44 to.80 for men and r = .59 to .99 for women (p < .0001). IPAQ reliability was lower for men (r = .29 to .76) than for women (r = .46 to .77). Criterion validity of the YPAS was .31 to .54 for men and .26 to .29 for women. The YPAS and short IPAQ had comparable results for reliability and criterion validity.     

A multitrait-multimethod investigation of interpersonal distance

Prior research has investigated the validity of interpersonal distance measures with respect to only one trait — comfort. This study expanded the previous research by measuring interpersonal distance along the trait dimensions of comfort, control and privacy. In addition, measures were taken along three different measurement techniques: stop distance, silhouette placement, and a paper and pencil measure (CID). The results were interpreted in terms of a multitrait-multimethod matrix. Results indicated that these measures were reliable and valid and the traits did have convergent validity. However, all of these measures were uncorrelated with a disguised distance measure of interpersonal distance.     

Experiments in the measurement of the size of social networks

This paper reports the results of a series of experiments conducted to measure the effects of alternative procedures for determining the size of social networks. The three methods tested are recall, recognition, and numerical estimation. Respondents were selected from work, church, and social groups. The major findings were that recall estimates are substantially below those obtained by recognition methods and become worse for larger groups. On average, numerical estimates are reasonable approximations of network size estimates based on recognition, but there is substantial variability in the accuracy of individual reports. Surprisingly, there is no evidence that recall is better for reports on number of close friends than of friends or acquaintances. There are order effects on numerical estimates of network size and better estimates are obtained by first asking about close friends. Finally, there is some evidence that respondent characteristics such as age affect recall of network size.