Ecometrics: Toward a Science of Assessing Ecological Settings, With Application to the Systematic Social Observation of Neighborhoods

This paper considers the quantitative assessment of ecological settings such as neighborhoods and schools. Available administrative data typically provide useful but limited information on such settings. We demonstrate how more complete information can be reliably obtained from surveys and observational studies. Survey-based assessments are constructed by aggregating over multiple item responses of multiple informants within each setting. Item and rater inconsistency produce uncertainty about the setting being assessed, with definite implications for research design. Observation-based assessments also have a multilevel error structure. The paper describes measures constructed from interviews, direct observations, and videotapes of Chicago neighborhoods and illustrates an "ecometric" analysis—a study of bias and random error in neighborhood assessments. Using the observation data as an illustrative example, we present a three-level hierarchical statistical model that identifies sources of error in aggregating across items within face-blocks and in aggregating across face-blocks to larger geographic units such as census tracts. Convergent and divergent validity are evaluated by studying associations between the observational measures and theoretically related measures obtained from the U.S. Census, and a citywide survey of neighborhood residents.     

Effectiveness of the Coral Tree Family Service family inpatient unit: a longitudinal study exploring change in family functioning

Child mental health disorders represent a major burden to public health in Australia due to high prevalence rates, the widespread impact across domains, and the potential for difficulties to persist into adolescence and adulthood. Extensive evidence exists for the use of parent management training and cognitive behavioural therapy to treat difficulties experienced by children; however, a proportion of children do not benefit as expected from these treatments. The use of complementary therapeutic approaches and variations to the mode and intensity of existing intervention is warranted; family inpatient units represent a unique example of this. The aim of this study is to evaluate the effectiveness of a well-established Australian inpatient unit that admits the whole family, for children aged 12 years and younger referred with mental health, behavioural, or emotional difficulties. This study's longitudinal within-subject study design utilised routinely collected outcome data from the systemic clinical outcome and routine evaluation, 15 item (SCORE-15)—a valid and reliable measure of family functioning—from a sample of 980 participants attending the Family Residential Program. This study reported significant improvement in family functioning across all outcome variables between baseline and post-intervention, with no variables returning to pre-intervention levels at follow-up. Clinically significant changes in overall family functioning showed that 37% of participant scores moved from the clinical range at baseline to the nonclinical range post-intervention. This study represents the first empirical evaluation of the Family Residential Program, and the reported results provide compelling evidence for the program to effect improvement in family functioning for families with long-standing and severe difficulties. Poor rates of questionnaire completion following intervention frequent the literature base of family inpatient units, with low post-intervention (56%) and follow-up (12%) rates evident in this study. This study uniquely contributes to the growing evidence base of family inpatient units using rigorous evaluation methods.     

Base rates, multiple indicators, and comprehensive forensic evaluations: why sexualized behavior still counts in assessments of child sexual abuse allegations

Developmentally inappropriate sexual behavior has long been viewed as a possible indicator of child sexual abuse. In recent years, however, the utility of sexualized behavior in forensic assessments of alleged child sexual abuse has been seriously challenged. This article addresses a number of the concerns that have been raised about the diagnostic value of sexualized behavior, including the claim that when population base rates for abuse are properly taken into account, the diagnostic value of sexualized behavior is insignificant. This article also identifies a best practice comprehensive evaluation model with a methodology that is effective in mitigating such concerns.     

A Therapeutic Summer Weekend Camp for Grieving Children: Supporting Clinical Practice Through Empirical Evaluation

Through collaborative action-oriented community research, agency-based clinicians, and university-based researchers engaged in a two-year theory-driven evaluation of a therapeutic summer camp for grieving children. The evaluation examined the camp practice model and children’s engagement in therapeutic camp activities and psychosocial functioning based on the camp staff and parental caregivers’ assessments. The first year findings led to a synergistic decision-making process that strengthened one of camp therapeutic activities. The overall results reveal a high consistency among staff on assessments of engagement and psychosocial functioning, and parental caregivers’ satisfaction with the camp. Limitations of research evaluation and implications for practice are considered.     

Risk assessment of sex offenders: the current position in the UK

This paper examines the various risk factors that are considered in the current risk assessments used with sex offenders, and the strengths and weaknesses of each. In addition, particular attention is given to the assessments that are currently employed within the prison and probation services. Static instruments, dynamic instruments and combined approaches are evaluated, as are clinical assessments of risk. In particular, the now commonly used joint Prison‐Probation Offender Assessment System risk prediction instrument (OASys) is considered, with an evaluation of the relevance of the factors included in making an assessment of risk and whether this is supported by research into recidivism amongst sex offenders. Copyright © 2006 John Wiley & Sons, Ltd.     

Developing a new system to measure outcomes in a service coordination program for youth with severe emotional disturbance

This paper presents information on re-developing an outcome evaluation for a state-funded program providing service coordination utilizing wraparound to youth with severe emotional disturbance (SED) and their families. Originally funded by the Robert Wood Johnson Foundation, the Kentucky IMPACT program has existed statewide since 1990. Changing data needs and limitations of the original evaluation required revamping the program's data collection system. The new evaluation uses the extant knowledge base to improve: (1) design, (2) measures, and (3) utility. A pre-post design with multiple follow-ups provides the framework for data collection. An ecological framework provides a conceptual structure for selecting measures focusing on both the service recipients and their environment. Data collection via a personal digital assistant (PDA) ensures utility of the data for both consumers and researchers. Issues ranging from conceptualization to implementation of the project as well as lessons learned are discussed.     

Work functioning: a conceptual framework

Adequate and reliable methods for evaluation of clients with work disabilities are crucial for both the individuals who are assessed and for society as a whole. Sound and precise work assessments are needed to guide clients to suitable interventions using a minimum of rehabilitation resources. Occupational therapy literature contains evidence that work function assessments are complex and that there is confusion over work assessment concepts. Therefore, further development and evaluation of adequate concepts and theoretical models within the area of work assessment is needed. This article proposes a conceptual framework for different dimensions of work functioning and points out important factors for work assessment. The concepts proposed and defined in the article are: work functioning, work participation, work performance, and individual capacity.     

Middle school reform through data and dialogue. Collaborative evaluation with 17 leadership teams

This report describes a 2-year, longitudinal study of one school district's effort to link site-based, collaborative evaluation with formal, centralized program evaluation. Participants formed a research team in partnership with a local university. Team members assisted leadership teams in identifying issues for informal, site-based assessments and then used the information to monitor progress toward established goals. Participants collaborated in designing and conducting informal assessments of student achievement, school climate and safety, discipline, and parent involvement. Leadership teams used these data in developing their school improvement plans. Researchers and administrators used these data to revise the districtwide survey. At the end of the 2-year cycle, analysis of school improvement plans showed that collaborative evaluation is creating a connection between dialogue and data. These 17 middle schools are approaching school improvement in a more integrated fashion by actively involving key stakeholders (students, parents, and teachers) in the evaluation process.     

Validity of work-related assessments

Insufficient evidence of the validity of work-related assessments is frequently reported as a major concern in occupational rehabilitation. Despite this concern, and the continuing development of new and old assessments, no comprehensive evaluation of the evidence has been conducted. OBJECTIVES: The purpose of this study was to first determine the extent and quality of available evidence for the validity of work-related assessments, and then where sufficient evidence was available, determine the level of validity. STUDY DESIGN: This study examined available literature and sources in order to review the extent to which validity has been established for 28 work-related assessments. RESULTS: The levels of evidence and validity are presented for each assessment. Most work-related assessments have limited evidence of validity. Of those that had adequate evidence, validity ranged from poor to good. There was no instrument that demonstrated moderate to good validity in all areas. Very few work-related assessments were able to demonstrate adequate validity in more than one area, or with more than one study, even when contributory evidence was included. CONCLUSION: With this study clinicians will be able to examine their options with regard to the validity of the assessments they choose to use.     

Removing Barriers to the Use of Community Information Systems

Abstract

Community information systems (CINS) are emerging as important tools for community, government, and educational organizations. This paper considers the training, evaluation, and outreach efforts relating to the Philadelphia Neighborhood Information System (NIS), a collection of online applications that integrate and distribute housing and demographic data. It presents an overview of the types of NIS users and uses and some specific examples ofhow the NISisbeing used for individual property inquiries, community surveys, needs assessments, and research. Finally, the paper discusses the barriers to more analytical uses and offers recommendations for social work education aimed at preparing social workers to support community organizations in their efforts to harness the potential of CINS for social change.     

Piloting online WellnessRx learning modules: Demonstration of developmental evaluation

WellnessRx is a health initiative focusing on healthy living through education, knowledge translation, and community engagement. Stakeholders of WellnessRx identified web-based education learning modules on nutrition and physical education as a priority to be integrated into existing health sciences curricula, as well as adapted for use by health professionals. Five learning modules were created with essential knowledge, skills, attitudes and resources or tools for health professional students and practitioners. As part of the ‘developmental evaluation framework’ for WellnessRx, two of these modules were piloted within two health professional student programs. This paper describes the pilot-evaluation experience involving student surveys, focus groups and interviews, and faculty perspectives. For both modules, student pre-post knowledge assessments indicated some improvements in post-module knowledge. Post module evaluations by students indicated benefits with the online delivery being flexible for access, self-health, case-based assessments and useful nutrition and physical activity guides. Challenges for students included their time to do the modules and the activity expectations. Instructors felt each module could be better targeted to different years within an undergraduate program. Through developmental evaluation, the pilot results along with recommendations and lessons learned provided the direction needed to further develop the WellnessRx logic model and proposed learning modules.     

Participation in cross-national learning assessments and impact on capacity development: Programmes,practice, structures and teacher competency. Case study of Kenya,Tanzania and South Africa

This research is one of the few attempts to employ the conceptual framework of "technology transfer" to analyze the extent that participation in cross-national learning assessments has had on capacity development, particularly in the development of official public structures, by equipping educationists and influencing teachers’ competency in Kenya, Tanzania and South Africa. The researchers conducted semi-structured interviews with key informants drawn from the Ministries of Basic Education, national examinations councils, civil society organizations and curriculum development institutions in the three countries. The in-depth interviews were complemented by relevant literature on this topic. The study established improved technical capacities in the public education sector in South Africa and Kenya to design and conduct independent large-scale learning assessments. This important research demonstrates a certain level of commitment by African countries to establish official structures necessary to design/implement and sustain a culture of monitoring learning outcomes through public funded large-scale learning assessments. The research potentially contributes to the body of knowledge as far as ‘summative’ evaluation and analysis of the theory of change underpinning the participation in cross-national learning assessments espoused under the Education for All (EFA) Framework of Action     

Neuropsychological Correlates of Performance Based Functional Status in Elder Adult Protective Services Referrals for Capacity Assessments

We have previously described high rates of executive function impairment in clients referred by Adult Protective Services (APS) to geriatric psychiatry for decision-making capacity assessments. The purpose of this study was to determine the independent relationship between neuropsychological screening instruments, particularly instruments sensitive to executive function, and performance-based functional tasks in elder referrals. Our retrospective medical review (n = 75/157 referrals completed all neuropsychological and functional assessments) revealed that only the Executive Interview (EXIT25) contributed independent variance to money management performance (R² = 0.29, p < 0.001), telephone use ability (R² = 0.39, p < 0.001), and summed performance (R² = 0.45, p < 0.001). Executive instruments may specifically inform decision-making capacity assessments.     

Retaining clients in an outcome monitoring evaluation study: HIV prevention efforts in community settings

The Centers for Disease Control and Prevention (CDC), Division of HIV/AIDS Prevention (DHAP) conducted outcome monitoring studies on evidence-based interventions (EBIs) provided by CDC-funded community-based organizations (CBOs). Critical to the success of outcome monitoring was the ability of CBOs to recruit and retain clients in evaluation studies. Two EBIs, Video Opportunities for Innovative Condom Education and Safer Sex (VOICES/VOCES) and Healthy Relationships, were evaluated using repeated measure studies, which require robust follow-up retention rates to increase the validity and usefulness of the findings. The retention rates were high for both VOICES/VOCES CBOs (95.8% at 30 days and 91.1% at 120 days), and Healthy Relationships CBOs (89.5% at 90 days and 83.5% at 180 days). This paper presents an overview of the retention of clients, challenges to follow-up, and strategies developed by CBOs to achieve high retention rates. These strategies and rates are discussed within the context of the CBOs’ target populations and communities.     

The Effect of Parental Divorce on Discomfort and Cardiac Activity in Response to Public Displays of Affection in College Females

Two studies examined the effect of parental divorce on feelings of discomfort and cardiac activity in response to public displays of affection (PDAs). In the first study, females from divorced homes reported marginally significant higher discomfort in response to PDAs than females from intact homes. In the second study, 32 female participants (16 from divorced homes and 16 from intact homes) viewed a PDA video and a control video and subsequently were asked to complete the same measure of discomfort as in Study 1. Interbeat intervals (IBIs) were recorded, and mean IBI and respiratory sinus arrhythmia (RSA) were derived. IBI and RSA increased in both groups in response to the videos over baseline. No differences were found between the divorce and intact groups for responses to the PDA video. However, the divorce group had higher heart rates and lower RSA overall, which suggests a potential mechanism for the association between divorce and negative physical health outcomes in women. Future studies should examine mediators of the relationship between divorce and reactions to PDA, such as marital conflict and parenting quality.     

A Comparison of Survey Techniques on Sensitive Sexual Behavior in Italy

This article compares two national surveys carried out through the most commonly used procedures in Italy: CATI (computer-assisted telephone interviews) and SAQ-FI (self-answered questionnaires following interviews). Both surveys ask two identical questions concerning sensitive sexual behavior: early age at first intercourse and same-sex attraction. The SAQ-FI survey had both unit non-response and item non-response rates much lower than the CATI survey. Moreover, in the CATI survey, the groups with highest item non-response rates were also the groups with the lowest proportions of early intercourse and homosexual attraction. In addition, a differential analysis of the respondents produced diverse results for the two surveys. This is especially true of results by gender for same-sex attraction: Such behavior is more common among men (3.1%) than women (2.9%), according to the CATI survey, whereas the opposite is true of the SAQ-FI survey (6.1% of men vs. 7.7% women). In Italy at the beginning of the 21st century, CATI surveys reveal a lower level of early intercourse and same-sex attraction than SAQ-FI surveys. This article argues that the CATI survey underestimates the true level of these sensitive sexual behaviors in the Italian population.     

Examining the relationship between gender and drug-using behaviors in adolescents: the use of diagnostic assessments and biochemical analyses of urine samples

The present study examined the relationship between gender and drug-use among adolescents using diagnostic assessments and biochemical analyses of urine samples. The data were collected in the context of a referral and assessment program for adolescents suspected of using drugs, using the Adolescent Substance Battery [1]. A six-month random sample of 20 percent of adolescents assessed for drug use were targeted for biochemical assay. Compliance for urine delivery was relatively high at 91 percent. Urine samples were quantitatively screened for pharmaceuticals using a standard thin-layer chromatography (TLC) technique [2]. Statistical significance was found in the relationship between gender and marijuana use (p < .05). This study confirms that gender and drug-using behaviors among adolescents need additional research and evaluation.     

The categorical versus dimensional assessment controversy in the sociology of mental illness

This paper reviews the sociological controversy over using categorical versus dimensional assessments in the study of psychological distress. The preference of sociologists for dimensional assessments is traced to two assumptions: (1) that the associations of predictors with psychological distress syndromes are most accurately operationalized by using dimensional measures; and (2) that no true discrete mental illnesses can reasonably be inferred to exist that would justify the creation of dichotomous measures. Methods are described in this paper to test both assumptions. An argument is made that the first assumption is the critical one and that dimensional analysis is useful only when it can be demonstrated that the associations of predictors with dimensional scores are constant throughout the relevant dimensional severity range. The paper closes with an argument for the tandem use of categorical and dimensional assessments in future sociological research.     

Evaluability assessment of a community development leisure program in Spain

This paper presents the results of an evaluability assessment (EA) conducted in Spain. EAs are assessments that minimize the uncertainty and risk inherent in a subsequent evaluation (e.g. an impact or results assessment) that may require excessive resources. We designed our own EA model by combining three theoretical approaches, with a holistic and pedagogic vision of the process and a flexible methodology to enable adaptation to different programs and contexts, thus fostering transferability. The EA was carried using a qualitative methodology: documentary analysis, seven semi-structured interviews with stakeholders, eight direct observation sessions and a check-list for politicians and experts. The outcome report helped promote organizational learning, provided the program with a logical and assessable theoretical model, detected its strengths and helped overcome the barriers to efficient assessment. Based on these achievements, an evaluation of the results was recommended, with repeated pretest-posttest measurements and an equivalent control group.     

Following the bouncing ball of quality

Quality of health care is important to all individuals. As an area of research and evaluation, quality is of considerable interest to a wide variety of professionals. Increasingly, independent assessments of the quality care have become important components of purchasing for employers and consumers. This article offers a personal perspective on the emerging trends in quality and identifies several related implications, with behavioral health examples, as well as opportunities for interested evaluation researchers.