C330. The variability of evaluations of speculative ideas

This article reports results of an extensive simulation study which investigated the performances of some commonly used methods of estimating error rates in discriminant analysis. Earlier research papers limited their comparisons of these methods to independent training data. This study allows for a simple auto-regressive dependence among the training data. The results suggest that the estimation methods based on the normal distribution perform adequately well under conditions of negative or mild positive correlation in the data, and small dimensions (p) of the observation vectors. For large p or strong positive correlation structures the conclusion is that one of the better non-parametric methods should be used. Special circumstances and conditions which notably affect the relative performances of the methods are identified.     

The role of titles in enhancing data visualization

Many in the data visualization and evaluation communities recommend conveying the message or takeaway of the visualization in the visualization's title. This study tested that recommendation by examining how informative or generic titles impact a visualization’s visual efficiency, aesthetics, credibility, and the perceived effectiveness of the hypothetical program examined. Furthermore, this study tested how simple or complex graphs, and positive, negative, or mixed results (i.e., valence of the results) affected outcomes. Participants were randomly assigned to one of 12 conditions, representing a 2 (graph: simple or complex) x 2 (title: generic or informative) x 3 (valence: positive, negative, mixed) between-subjects study. The results indicated that informative titles required less mental effort and were viewed as more aesthetically pleasing, but otherwise did not lead to greater accuracy, credibility, or perceived effectiveness. Furthermore, titles did not interact with graph type or the valence of the findings. While the results suggest it is worthwhile to consider adding an informative title to data visualizations as they can reduce mental effort for the viewer, the intended goal of the visualization should be taken into consideration. Considering the goal of the visualization can be a deciding factor of the type of graph and title that will best serve its intended purposes. Overall, this suggests that data visualization recommendations that impact evaluation reporting practices should be scrutinized more closely through research.     

Dealing with context in logic model development: Reflections from a realist evaluation of a community health worker programme in Nigeria

Logic models (LMs) have been used in programme evaluation for over four decades. Current debate questions the ability of logic modelling techniques to incorporate contextual factors into logic models. We share experience of developing a logic model within an ongoing realist evaluation which assesses the extent to which, and under what circumstances a community health workers (CHW) programme promotes access to maternity services in Nigeria. The article contributes to logic modelling debate by: i) reflecting on how other scholars captured context during LM development in theory-driven evaluations; and ii) explaining how we explored context during logic model development for realist evaluation of the CHW programme in Nigeria. Data collection methods that informed our logic model development included documents review, email discussions and teleconferences with programme stakeholders and a technical workshop with researchers to clarify programme goals and untangle relationships among programme elements. One of the most important findings is that, rather than being an end in itself, logic model development is an essential step for identifying initial hypotheses for tentative relevant contexts, mechanisms and outcomes (CMOs) and CMO configurations of how programmes produce change. The logic model also informed development of a methodology handbook that is guiding verification and consolidation of underlying programme theories.     

Toward a model of school inspections in a polycentric system

Many education systems are developing towards more lateral structures where schools collaborate in networks to improve and provide (inclusive) education. These structures call for bottom-up models of network evaluation and accountability instead of the current hierarchical arrangements where single schools are evaluated by a central agency. This paper builds on available research about network effectiveness to present evolving models of network evaluation. Network effectiveness can be defined as the achievement of positive network level outcomes that cannot be attained by individual organizational participants acting alone. Models of network evaluation need to take into account the relations between network members, the structure of the network, its processes and its internal mechanism to enforce norms in order to understand the achievement and outcomes of the network and how these may evolve over time. A range of suitable evaluation models are presented in this paper, as well as a tentative school inspection framework which is inspired by these models. The final section will present examples from Inspectorates of Education in Northern Ireland and Scotland who have developed newer inspection models to evaluate the effectiveness of a range of different networks.     

A decade of changes: Within-group analysis of LGBTQ individuals' perceptions of their community context and the relevance for social service providers

The previous decade has seen a drastic shift in the social environment for lesbian, gay, bisexual, transgender, and queer (LGBTQ) people. It is not surprising then to find that the perceived “best” and “worst” aspects of life for nonmetropolitan LGBTQ individuals have also changed. We use data from two cohorts of the Rainbow Illinois survey to examine differences between responses in 2000 and 2011. Furthermore, we examine demographic differences in the perception of what is “best” and “worst” among the 2011 cohort. Results suggest that although the general social climate has improved for some, it remains problematic for others. Implications for social service providers are thoroughly discussed.     

Eliciting children’s recall regarding home life and relationships

Abstract

Child interviews form an important component of custody evaluations. Yet, research on children’s responses to questions about home life and relationships is lacking. In the present study, children (N?=?47) aged 6 to 10?years were interviewed about their daily routines and family relationships. Responses to four categories of questions were compared: open and specific questions about routines, and negative and positive aspects of family relationships. Responses were coded for amount of information, informativeness, topic pertinence, and refusals to answer. Results suggested that questions about everyday routines and relationships elicit relevant and informative responses from children. It is suggested that interviewers begin with open-ended questions regarding daily routines to structure family law interviews with children.