Understanding the factors that influence health promotion evaluation: The development and validation of the evaluation practice analysis survey

The demand for improved quality of health promotion evaluation and greater capacity to undertake evaluation is growing, yet evidence of the challenges and facilitators to evaluation practice within the health promotion field is lacking. A limited number of evaluation capacity measurement instruments have been validated in government or non-government organisations (NGO), however there is no instrument designed for health promotion organisations. This study aimed to develop and validate an Evaluation Practice Analysis Survey (EPAS) to examine evaluation practices in health promotion organisations. Qualitative interviews, existing frameworks and instruments informed the survey development. Health promotion practitioners from government agencies and NGOs completed the survey (n = 169). Principal components analysis was used to determine scale structure and Cronbach’s α used to estimate internal reliability. Logistic regression was conducted to assess predictive validity of selected EPAS scale. The final survey instrument included 25 scales (125 items). The EPAS demonstrated good internal reliability (α > 0.7) for 23 scales. Dedicated resources and time for evaluation, leadership, organisational culture and internal support for evaluation showed promising predictive validity. The EPAS can be used to describe elements of evaluation capacity at the individual, organisational and system levels and to guide initiatives to improve evaluation practice in health promotion organisations.     

Content validity and utility of the Assessment of Work Performance (AWP)

In the area of work rehabilitation, many decisions about future interventions for the client are based on the results of various kinds of assessments. Therefore, it is important that the assessment instruments used are adequate, useful, and reliable. The purpose of this study was to investigate the content validity and utility of the instrument Assessment of Work Performance (AWP) which is used to assess an individual's observable (working) skills during work performance, i.e. how efficient and appropriate a client performs a work activity. A questionnaire was answered by 67 respondents who used the AWP in various work rehabilitation settings in Sweden. The result indicates content validity and utility for the AWP that supports further testing of the instrument.     

An Assessment of the Psychometric Properties of Italian Version of CPGI

The aim of this study was to adapt to the Italian context a very commonly used international instrument to detect problem gambling, the canadian problem gambling index (CPGI), and assess its psychometric properties. Cross-cultural adaptation of CPGI was performed in several steps and the questionnaire was administered as a survey among Italian general population (n = 5,292). Cronbach’s alpha reliability coefficient was 0.87 and can be considered to be highly reliable. Construct validity was assessed first by means of a principal component analysis and then by means of confirmatory factor analysis, showing that only one factor, problem gambling, was extracted from the CPGI questionnaire (an eigenvalues of 4,684 with percentage of variance 52 %). As far as convergent validity is concerned, CPGI was compared with Lie/Bet questionnaire, a two-item screening tool for detecting problem gamblers, and with both depression and stress scales. A short form DSM-IV CIDI questionnaire was used for depression and VRS scale, a rating scale, was used for rapid stress evaluation. A strong convergent validity with these instruments was found and these findings are consistent with past research on problem gambling, where another way to confirm the validity is to determine the extent to which it correlates with other qualities or measures known to be directly related to problem gambling. In sum, despite the lack of a direct comparison with a classic gold-standard such as DSM-IV, the Italian version of CPGI exhibits good psychometric properties and can be used among the Italian general population to identify at-risk problem gamblers.     

Validity and Reliability of the Spanish Translation of the Attitudes Towards Divorce Scale

The set of studies described in this article analyzed the psychometric properties of the Attitudes Towards Divorce Scale (ATDS; Kinnaird & Gerrard, 1986) in a group of 964 Spanish emerging adults between 18 and 30 years old. Factorial validity, concurrent validity, internal reliability, and measurement invariance across males and females were examined. Two factors accounted for 50% of the total variance: positive and negative attitudes toward divorce. This 2-factor structure fit adequately in males and females. The internal consistency of the questionnaire was good (Cronbach’s α = .74), and its convergent validity was supported by a significant positive correlation with the Attitudes Towards Marriage Scale and gender differences in attitudes toward divorce. Overall, the results suggest that the AD, the Spanish version of the ATDS, is a reliable and valid instrument that could be used to measure Spanish emerging adults’ attitudes toward divorce.     

An empirical analysis of alleged misunderstandings of coefficient alpha

Cronbach’s alpha is the most frequently used measure to investigate the reliability of measurement instruments. Despite its frequent use, many warn for misinterpretations of alpha. These claims about regular misunderstandings, however, are not based on empirical data. To understand how common such beliefs are, we conducted a survey study to test researchers’ knowledge of and beliefs about alpha. For this survey, we selected authors from recent papers, in which alpha was used. The results provide empirical evidence for the claims that researchers have difficulty interpreting alpha in a proper way. At the same time, we expounded the claims, by showing that whereas some beliefs are fairly typical, others are not so often seen. This non-technical paper, aimed at both statisticians and substantive researchers, is concluded by providing a few suggestions that could be helpful to get us out of the current stalemate regarding the usability of alpha and its alternatives.     

Determining the Appropriateness of the “What If” Situations Test (WIST) with Turkish Pre-Schoolers

Measurement instruments are needed to assess the child’s sexual abuse prevention program. The purpose of the study was to determine the reliability and validity of the WIST (What If Situations Test) for Turkish culture. Participants were children of the 3–6 age group attending pre-school education institutions and the sample size was identified by means of a power analysis. Seventy children were identified as the sample with 0.85 power and 0.05 type I error according to the power analysis. Language validity, content validity, internal validity coefficient (Cronbach alpha coefficient), and test–retest analyses were conducted in terms of validity and reliability in the scope of efforts for adaptation to Turkish culture. Firstly, Kendall W = 0.83 was the score for the expert opinions concerning the content validity of the language validity scale. It was found that the Cronbach alpha coefficients were between 0.68 and 0.90 for the scale sub-dimensions of appropriate and inappropriate recognition, saying, doing, telling, and reporting. The test–retest reliability of the scale was found to be r = 0.89 and the test–retest reliabilities for the sub-dimensions (appropriate recognition, inappropriate recognition, say skills, do skills, tell skills, and reporting skills) were between r = 0.48 and r = 0.92. The test–retest reliability for the Personal Safety Questionnaire (PSQ), as having complimentary items to the WIST, was found to be r = 0.82. The reliability and validity analysis of the ‘What If’ Situations Test (WIST), used to evaluate pre-schoolers’ skills regarding self-protection against sexual abuse, showed that the Test’s adaptation to Turkish culture was reliable and valid.     

Assessment of Work Performance (AWP)--development of an instrument

Adequate work assessments are a matter of importance both for individuals and society [5,29,31,38,40,46,52]. However, there is a lack of adequate and reliable instruments for use in work rehabilitation [14,15,20,21,31,44]. The purpose of this study was to develop and evaluate an observation instrument for assessing work performance, the AWP (Assessment of Work Performance). The purpose of the 14-item instrument is to assess the individual's observable working skills in three different areas: motor skills, process skills, and communication and interaction skills. This article describes the development and results of preliminary testing of the AWP. The testing indicates a satisfactory face validity and utility for the AWP and supports further research and testing of the instrument.     

Development and validation of a measure of organizational capacity for implementing youth development programs

Accumulating evidence indicates that incorporating youth development (YD) principles, strategies, and supports into an organization promotes positive adult and youth outcomes. However, few validated measures assess this type of capacity. The YMCA commissioned a study to validate its Capacity Assessment for Youth Development Programming (Y-CAP), which examines the organizational infrastructure required to implement YD programs and processes in seven areas. Survey development was an iterative process informed by existing frameworks, instruments, and pilot testing of items. The Y-CAP was reviewed and revised three times prior to this study, with a final round of revisions made at the start of the validation phase as a result of thorough content, survey methodology, and psychometrics reviews. The revised Y-CAP was completed by 123 YMCA implementation teams. Rasch analyses were used to determine the extent to which validity evidence supports the use and interpretation of the Y-CAP scores. Convergent validity was assessed by comparing Y-CAP scales to the Algorhythm staff survey for youth-serving organizations, and focus groups informed the consequential validity of the Y-CAP. The results provide strong evidence for the reliability and validity of the Y-CAP, which can be used to guide continuous quality improvement initiatives that support capacity and functioning in youth-serving organizations and programs.     

Psychometric properties of the Spanish version of the activism and radicalism intention scale / Propiedades psicométricas de la versión española de la escala de intención de activismo y radicalismo

The purpose of this study is to adapt the Activism and Radicalism Intention Scale to Spanish. To do so, two studies were performed. In the first, which had a sample of 514 participants, the instrument was adapted to Spanish and a confirmatory factorial analysis was conducted. The psychometric results obtained were similar to those of the original version, and the scale’s two-dimensional structure was replicated. In the second study, the psychometric characteristics of the original instrument were also replicated, but in this case with a sample of 133 participants comprised of sympathizers and activists in the 15-M anti-austerity movement. Regarding the validity of the scale, both studies provide evidence on the coherence of the construct in terms of both its internal structure and its theoretical relationship with other psychosocial variables. The reliability of the instrument obtained in both samples studied was acceptable. In short, this study presents a tool in Spanish which can be useful for studying individuals’ and groups’ predisposition towards more or less violent political mobilization.     

Family-of-origin expressiveness: measurement, meaning, and relationship to alexithymia

The need for research instruments to assess the impact of affective expressiveness within the family is evident, yet few appear to be adequately designed for this purpose. In this article, we present two studies addressing this need. In the first study, the original 40-item Family-of-Origin Scale was administered to 416 students to determine those items that constitute the factor structure. This instrument was designed to assess perceived levels of health in the family of origin but has unsubstantiated construct validity. Results from a confirmatory factor analysis indicated that the instrument has one major factor, and results from five other studies provide evidence supporting the construct validity. Face validity of this 22-item construct indicates that it assesses an individual's perceived level of global expressive atmosphere within his or her family of origin. In the second study, the new Family-of-Origin Expressive Atmosphere Scale and the 20-item Toronto Alexithymia Scale were administered to 295 students. Students' self-reported expressive atmospheres in their family-of-origin scores were significantly correlated with the total scores of alexithymia and each of the three factors: impaired ability to identify feelings, impaired ability to describe feelings, and externally oriented thinking processes. No significant gender differences were found.     

Concept guided risk assessment: Promoting prediction and understanding

This article presents a concept-guided approach to the assessment of risk in the life situations of children alleged to be maltreatment victims. This concept-guided risk assessment is theoretically based and is the first to use a nomological hierarchy which can yield itself to systematic testing for construct validity. Data were collected on 1199 cases referred for investigations of child maltreatment. Reliability was examined by applying Cronbach's alpha coefficient to the 25 risk features and the 18 risk category constructs within the seven risk areas. Convergent validity coefficients for seven risk area constructs and eighteen risk category constructs were computed and structural validity was examined using an analysis technique adapted from the multitrait-multimethod technique of Campbell and Fiske (1959). Predictive validity was assessed using a multivariate measure assessing rated risk, seriousness, and concern. In general, the results support construct validity. Improvements are needed in assessments of child vulnerability and caregiver stress. The overall psychometric evidence, combined with the model's hierarchical and theoretical nature, suggests a promising direction for future research on understanding and preventing child maltreatment.     

Preenrollment Immunization Policies of American Colleges: An Assessment of the Need for Policy Implementation

Abstract

Alcohol abuse among college students is prevalent, yet few instruments with sound reliability and validity are available to assess these problems in this population. As part of a large, baseline assessment battery for a prospective study of offspring of alcoholics, the 27-item Young Adult Alcohol Problems Screening Test (YAAPST) was given to 490 freshmen at a large mid western university; approximately 9 months later, 482 subjects completed the scale again. In addition to asking about such traditional problems as experiencing blackouts and driving while intoxicated, the YAAPST included specific items relating to college experiences (eg, getting into sexual situations that were later regretted, missing classes, and receiving lower grades than usual). The YAAPST was designed to assess these drinking consequences over two different time frames, lifetime and past year, and also to indicate the frequency of occurrence during the past year. Results indicated that the YAAPST is a unidimensional scale with good psychometric properties (good internal consistency and test-retest reliability). Three different approaches were used to demonstrate the validity of the YAAPST. Findings supported criterion validity (with interview-based alcohol abuse/dependence diagnoses as the criterion), concurrent validity (comparing the YAAPST with other drinking measures), and construct validity (correlating the YAAPST with etiologically relevant personality, motivational, and peer influence variables). The YAAPST is a promising screening instrument for alcohol problems in college students. It has excellent psychometric properties and the potential to provide a range of useful information to the clinician or researcher.     

Examining the Reliability and Validity of the Fisher Divorce Adjustment Scale: The Persian Version

This study was an attempt to determine the factor structure, reliability, validity, and adaptation of the Fisher Divorce Adjustment Scale (FDAS) for Iranian divorced individuals. The primary rationale for this study was the lack of such an instrument in an Iranian context to help researchers and therapists determine postdivorce adjustment and distinguish those in need of receiving psychological help. Participants of this study were 486 individuals (49.5% men, 50.5% women) who were divorced and were selected from the available population. Participants were asked to complete a sociodemographic questionnaire, the FDAS (Fisher, 1978), the Satisfaction With Life Scale (SWLS; Diener, Emmons, Larsen, & Griffin, 1985), and the General Health Questionnaire (GHQ; Ghanbarnejad & Turki, 2013). The results showed that the internal consistency of FDAS using Cronbach’s alpha was .93 and the split-half coefficient was .89, indicating fine consistency. Also, the internal consistency of FDAS subscales measuring self-worth, disentanglement from the ex-partner, anger, grief, social trust, and social self-worth was .81, .86, .89, .88, .86, and .69, respectively, using Cronbach’s alpha. Data analysis shows an adequate convergent validity with the SWLS and GHQ. It can be concluded that the FDAS has an acceptable factor structure, reliability, and validity, and can be used in Iran and other Persian-speaking countries.     

A Psychometric Analysis of Field Evaluation Instruments Designed to Measure Students’ Generalist-Level Social Work Competencies

ABSTRACT

There is a dearth of research on the evaluation of the psychometric performance of instruments designed to measure students’ generalist-level social work competencies. There is also uncertainty on the performance of various response option formats used to measure students’ competencies in assessment instruments. Using a sample of 198 master of social work students and 198 field instructors, the current study employs confirmatory factor analysis to assess the psychometric performance of a field evaluation instrument designed to measure students’ generalist-level social work competencies. The results provide evidence to support the validity and reliability of the instrument. The results also highlight nuances associated with a 5-point response option format for items used to measure competency indicators. Implications, limitations, and directions for future research are discussed.     

Sexual Functioning in Spanish and Portuguese Young Adults: Initial Validation of the Arizona Sexual Experience Scale

Objectives: The authors conducted an initial validation of the Arizona Sexual Experience Scale (ASEX) and examined the sexual function of Spanish and Portuguese young adults. Methods: Participants were 523 Spanish and 595 Portuguese heterosexual young adults who completed a background questionnaire, the ASEX, and measures of sexual sensation seeking, excitation, and inhibition. Results: The ASEX showed good reliability and validity. Participants reported good sexual function. The most prevalent difficulties in men were related to desire and orgasm, and to orgasm and satisfaction in women. Conclusions: This is a psychometrically sound instrument to assess sexual functioning in Spanish and Portuguese young adults.     

The Structured Interview of Family Assessment Risk: Convergent Validity,Inter-rater Reliability and Structural Relations

The quality of parenting, composition and socio-economic status are family risk or protective factors on juvenile delinquency, however, there is not an instrument to help psychologists and social workers to specifically assess the adolescent offender’s family. This study set out to accomplish an additional validation study of the structured interview of family assessment risk (SIFAR), a structured professional judgment tool for the assessment of family risk and protective factors of juvenile delinquents. The statistical analyses included inter-rater reliability, convergent validity with YLS/CMI, the Partial Least Squares approach to structural equation modeling and receiving operator characteristics (ROC) analysis. A sample of 130 male adolescent delinquents detained in Portuguese forensic facilities and their parents, were paired analyzed. The YLS/CMI was used to analyze the convergent validity with SIFAR. SIFAR shows a strong correlation with the YLS/CMI family context, moderate to high values of inter-rater reliability; SIFAR factors show that they are predictive determinants of the Moderate Four risk factors. ROC analysis shows adequate accuracy power. The findings show that SIFAR it is useful as an additional assessment tool for structured risk assessment instruments since it allows understanding the vulnerabilities and strengths of the delinquent’s family.     

Adapting surveys for nonprofit research

Many useful survey instruments have been developed in the for‐profit management arena, but they often require varying levels of adaptation for relevant application in the nonprofit context. This research note first explains key steps in the process for adapting and testing a survey instrument. It then illustrates how each step should be applied and reported using a case study adaptation of Hodgkinson's (1992) strategic locus of control instrument for nonprofit use.     

Exploring the relation between bullying and homophobic verbal content: the homophobic content agent target (HCAT) scale

This investigation quantitatively examines the association among homophobic content, bullying, victimization, empathy, and several psychosocial outcomes of these constructs. The 2-factor Homophobic Content Agent Target (HCAT) scale was developed and validated among 191 middle school students to assess the extent to which students both use and are called various epithets in reference to sexual orientation. Cronbach reliability coefficients of alpha = .85 were obtained for both factors. Convergent validity was demonstrated with scales measuring bullying, fighting, victimization, relational aggression and victimization, anxiety and depression, and delinquency. Discriminant validity was demonstrated in comparison with school sense of belonging, empathy, and perspective-taking. Discriminative validity was demonstrated through sex differences on several scales. Results strongly suggest that homophobic content is prevalent in various forms of aggression and victimization, and that future research should examine the role of homophobia in bullying and victimization in schools.