The theory-driven approach to validity

Current approaches for dealing with validity tend to focus on one type of validity and play down the importance of others. For example, internal validity is treated by Campbell and Stanley as “sine qua non,” whereas, Cronbach judges external validity to be prime importance in research. However, these single validity oriented approaches may achieve one type of validity at the expense of other types of validity.As an alternative, we propose a theory-driven approach to validity. Our central argument is that a model or theory should be formulated in a program evaluation and the modeling process should include the identification of potential threats to validity in research. In this sense, the purpose of applying a technique such as randomization is to strengthen rather than to replace the model or theory. That is, program evaluation should be model or theory-oriented instead of research design oriented. Randomization can be used in any research if it is applicable, but we need rely less on randomization as a safeguard to internal validity if the threats in a research can be adequately identified and controlled.Since the structural relationships among the relevant variables and intervening processes are carefully modeled, the theory-driven approach avoids the pitfalls of black-box evaluation and provides better understanding of the causal mechanisms underlying the relationships between treatment and effects. Unlike randomized experiments which aim only at internal validity, a theory-driven approach provides a general framework to deal with internal, external, construct and statistical conclusion validity.     

Enhancing program review for evaluation and management

Program review has not received the attention it warrants as a program evaluation tool despite its wide use for evaluation and management purposes. The use of the program review will probably endure on the strength of its face validity and irrespective of other developments in the field of program evaluation. Evaluators should realize this and, accordingly, attempt to improve its effectiveness. This paper presents one organization's approach to achieving this objective through the explication of development principles, implementation guidelines and review items. This paper also discusses benefits that can be expected from a systematic development of this tool and presents various research directions and potentials in this area.     

Stakeholders as partners in evaluation: A stakeholder-collaborative approach

An emphasis in recent years on the utilization of evaluation has led to such innovations as stakeholder-based evaluation, which typically allows concerned groups to participate at the planning and review phases of a study. This paper presents a case study of a “stakeholder-collaborative” approach, in which the roles of participants were extended, to incorporate the design, administration, and interpretation of surveys and interviews, extensive data-gathering and development of major portions of the final report. An internal evaluator assigned to the group assumed the roles of facilitator and task manager, to enhance the validity of results. Benefits and disadvantages of the approach for this case are discussed, as well as utilization of the report produced. Six months after submission of the final report, four group members and two major users were interviewed regarding this approach and the report's use; all rated the process positively. Interviewee's suggestions for decreasing problems and maximizing benefits are summarized.     

A New Approach to Evaluating the Quality of Measurement Instruments: The Split‐Ballot MTMM Design

Two distinctly different quantitative approaches are used to evaluate measurement instruments: the split‐ballot experiment and the multitrait‐multimethod (MTMM) approach. The first approach is typically used to indicate whether variation in the method causes differences in the response distribution; the second approach evaluates the reliability and validity of different methods. The new approach, suggested in this paper, combines the more attractive features of both methods. The strength of the split‐ballot experiment is its use of independent random samples from the same population to provide information about differences in response distributions. This is also possible with the new approach, but this approach provides more detailed information about the reasons for the differences. The MTMM approach provides information about reliability and validity on the basis of repeated observation of the same traits using different methods. This information is also provided by the new design. The difference is that the new approach reduces the need for repeated observations of the same trait. Each sample is provided with a different combination of only two methods and the complete model with all methods is estimated as a multiple‐group model. This reduces the burden for respondents and also reduces memory and order effects. Alternative designs and estimation methods are discussed, their efficiency is analyzed, and illustrations are provided .     

Using a community-created multisite evaluation to promote evaluation use across a sector

Multisite evaluation has been cited as a strategy for building knowledge about a program's success and enhancing evaluation capacity and evaluation use. Even so, the extent to which data from these efforts are made available to individual project participants varies. This article describes a community-created multisite evaluation that was designed to overcome this limitation. Based on the negotiated centralized evaluation model and implemented within the context of a community of practice, the EvalFest community-created multisite evaluation has used shared measures to collect data from 30,000 participants from partner sites across the community over three years. This article describes the design characteristics of the community-created multisite approach, and presents results related to partners’ evaluation use that demonstrate the early success of this method. We conclude by considering how this strategy might be applied to other sectors.     

Pre- and post-planned evaluation: Which is preferable?

This paper compares the advantages and disadvantages of the “pre-planned” and “post-planned” approaches to evaluating program effectiveness. These evaluative approaches are compared along a number of dimensions which include: (a) Reliability of data and the cost of collecting it; (b) Internal validity; (c) External validity; (d) Evaluation obtrusiveness and threat; and (e) Program goal displacement and program direction. A model designed to help program managers decide when and under what conditions either of these two evaluative approaches should be employed is presented. One major theme throughout this discussion is that despite the growing interest in and use of pre-planned evaluation, the post-planned method has many advantages which often go unnoticed. This paper will help program administrators, planners and evaluators in selecting and implementing these two methods in light of their respective strengths and limitations.     

The Viability of Readiness Assessment in Contributing to Improved Career Services: Response to Jepsen (2000)

This article responds to D. A. Jepsen's (2000) commentary on Sampson et al.'s theory‐based approach to using readiness assessment to improve career services (J. P. Sampson, G. W. Peterson, R. C. Reardon, & J. G. Lenz, 2000). Three topics are included in this response: the reliability, validity, and utility of readiness assessment measures; verbal ability and the use of cognitive information‐processing theory in practice; and the potential contribution of reflective evaluation in career decision making. This article concludes with recommended research questions to examine the impact of readiness assessment on the effectiveness of career service delivery.     

Defining,illustrating and reflecting on logic analysis with an example from a professional development program

Program designers and evaluators should make a point of testing the validity of a program's intervention theory before investing either in implementation or in any type of evaluation. In this context, logic analysis can be a particularly useful option, since it can be used to test the plausibility of a program's intervention theory using scientific knowledge. Professional development in public health is one field among several that would truly benefit from logic analysis, as it appears to be generally lacking in theorization and evaluation. This article presents the application of this analysis method to an innovative public health professional development program, the Health Promotion Laboratory. More specifically, this paper aims to (1) define the logic analysis approach and differentiate it from similar evaluative methods; (2) illustrate the application of this method by a concrete example (logic analysis of a professional development program); and (3) reflect on the requirements of each phase of logic analysis, as well as on the advantages and disadvantages of such an evaluation method. Using logic analysis to evaluate the Health Promotion Laboratory showed that, generally speaking, the program's intervention theory appeared to have been well designed. By testing and critically discussing logic analysis, this article also contributes to further improving and clarifying the method.     

The use of concept mapping in measurement development and evaluation: Application and future directions

The past decade has seen an increase of measurement development research in social and health sciences that featured the use of concept mapping as a core technique. The purpose, application, and utility of concept mapping have varied across this emerging literature. Despite the variety of uses and range of outputs, little has been done to critically review how researchers have approached the application of concept mapping in the measurement development and evaluation process. This article focuses on a review of the current state of practice regarding the use of concept mapping as methodological tool in this process. We systematically reviewed 23 scale or measure development and evaluation studies, and detail the application of concept mapping in the context of traditional measurement development and psychometric testing processes. Although several limitations surfaced, we found several strengths in the contemporary application of the method. We determined concept mapping provides (a) a solid method for establishing content validity, (b) facilitates researcher decision-making, (c) insight into target population perspectives that are integrated a priori, and (d) a foundation for analytical and interpretative choices. Based on these results, we outline how concept mapping can be situated in the measurement development and evaluation processes for new instrumentation.     

Validity of work-related assessments

Insufficient evidence of the validity of work-related assessments is frequently reported as a major concern in occupational rehabilitation. Despite this concern, and the continuing development of new and old assessments, no comprehensive evaluation of the evidence has been conducted. OBJECTIVES: The purpose of this study was to first determine the extent and quality of available evidence for the validity of work-related assessments, and then where sufficient evidence was available, determine the level of validity. STUDY DESIGN: This study examined available literature and sources in order to review the extent to which validity has been established for 28 work-related assessments. RESULTS: The levels of evidence and validity are presented for each assessment. Most work-related assessments have limited evidence of validity. Of those that had adequate evidence, validity ranged from poor to good. There was no instrument that demonstrated moderate to good validity in all areas. Very few work-related assessments were able to demonstrate adequate validity in more than one area, or with more than one study, even when contributory evidence was included. CONCLUSION: With this study clinicians will be able to examine their options with regard to the validity of the assessments they choose to use.     

The sequence analysis of hospitals that established social work department under fuzzy environment

This paper reviews the existing literature on hospital social work and discusses intervention strategies for improving social work practice in hospital. The objective of this study was to improve the quality of medical care. But few studies have compared social work services between different hospitals. This study describes qualitative analysis under fuzzy environment, extracts the main influencing factors and establishes a comprehensive evaluation index system. It provides comprehensive evaluation for alternative hospitals by the fuzzy clustering method. This paper proposes a new mixed fuzzy clustering algorithm on the basis of analysing the axiomatic fuzzy set (AFS) and K-means algorithm, which is not affected by some complicated parameter issues and has higher statistical validity. Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) is applied for selecting the best option for each cluster and a comparative analysis is done. Results from a case study in Shanghai, China, confirm that the proposed approach is effective by using information entropy to test. By comparing AFS, K-means and C-means algorithms, the hybrid algorithm can find the two closest attributes of evaluation index of hospital social work, and the proposed approach can be easily help raise the level of hospital social work service.     

The use of concept mapping for scale development and validation in evaluation

Evaluators often make key decisions about what content to include when designing new scales. However, without clear conceptual grounding, there is a risk these decisions may compromise the scale's validity. Techniques such as concept mapping are available to evaluators for the specification of conceptual frameworks, but have not been used as a fully integrated part of scale development. As part of a multi-site evaluation of family support programs, we integrated concept mapping with traditional scale-development processes to strengthen the creation of a scale for inclusion in an evaluation instrument. Using concept mapping, we engaged staff and managers in the development of a framework of intended benefits of program participation and used the information to systematically select the scale's content. The psychometric characteristics of the scale were then formally assessed using a sample of program participants. The implications of the approach for supporting construct validity, inclusion of staff and managers, and theory-driven evaluation are discussed.     

Visionary evaluation: Approaching Aboriginal ontological equity in water management evaluation

The 2017 Traditional Owner evaluation of the implementation of the Murray-Darling Basin Plan developed an approach to evaluation that tested the use of Standpoint Theory in the field of natural resource management. This methodological choice was intended to enable First Nation approaches to data generation and use in equal measure to non-indigenous approaches. The method is implemented as a nested, up-hierarchy of scale, enabling a pan-optican dimension of vision from "below" and "above". The paper does not present the evaluative results regarding the implementation of the Plan because that information is co-owned by the participating Nations for their uses. Instead, and in respect of that arrangement, the paper presents the evaluation practices funded by the Murray-Darling Basin Authority. The methodology was negotiated and implemented with the Nations in the pilot study as a co-production across cultural boundaries. The approach was then evaluated by the participants, and these results are reported. All those reviewing the methodology were directly involved in some aspect of the evaluation, 64 % of whom identified as Traditional Owners, 67 % of whom were involved in high level decision-making about the evaluation approach. Traditional Owners rated cultural competence of the tested approach at 68 %, the benefits of the approach at 75 %, satisfaction with the standard of the evaluation at 72 %, and satisfaction with complying with the Basin Plan’s requirements for evaluation at 78 %. Recommendations for broader engagement and better science communication are made.     

Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

Ties,sites and irregularities: pitfalls and benefits in using respondent-driven sampling for surveying a homeless population

Respondent-driven sampling (RDS) is a survey method for hidden populations and, as such, it offers a suitable approach for sampling the homeless. Surprisingly, the practical use of RDS in surveying homeless populations has only sporadically been described in the professional literature so far, and the specifics of using RDS for sampling this group have not been reflected in depth. The goal of the article is to investigate how three specific concerns, namely the transitional character of social ties among the homeless, problems with locating the interview site, and the irregular use of coupons, affect the sampling process. The findings of the article are based on experience using RDS for creating two independent samples of homeless persons in two Czech cities. Worries of poor referral effectiveness were disproved, but poor validity of the network size indicator, tendency towards irregular use of coupons, and negative effects arising from selecting service centres for the homeless as survey sites were confirmed.     

FAMILY GOAL RECORDING: AN ADAPTATION OF GOAL ATTAINMENT SCALING FOR ENHANCING FAMILY THERAPY AND ASSESSMENT*

Family Goal Recording is proposed as a systemic approach for measuring change in the presenting concerns of couples and families. This paper describes the procedures for using Family Goal Recording as an outcome measure for family therapy and a device to enhance goal attainment. In addition, the results of an initial study on the reliability and validity of this method are presented.     

Using multiple lines of evidence and independent teams to enhance the credibility of evaluations

The utilization of evaluation findings is an ongoing issue among evaluators. This article presents an approach to conducting evaluations that can be used to enhance the credibility of findings and to facilitate the production of adequate documentation in support of evaluation findings. The concepts of multiple lines of evidence and the use of independent teams are discussed. Use of this approach is expected to improve the utilization of evaluation findings.Two program evaluations that were undertaken at Canada's federal department of Consumer and Corporate Affairs are presented. To undertake the data collection and analysis, these studies were broken into separate “evaluation modules, ” each one undertaken by an independent team. The teams reported to an in-house study director, responsible for weighing the evidence and findings of each of the teams and writing the final evaluation report.     

Measuring corporate social performance: The problem of values

Debates over the social performance of corporations have carried with them implicit evaluations of that performance. The apparent subjectivity and bias of these informal evaluations has led to the development of more formalized corporate social performance assessment methodologies — social audits. In this paper it is argued that these social audits are also influenced by subjectivity and bias considerations; and thru the case study of a social audit of a corporate safety program, the underlying value issues at the core of these influences are explored. The value implications of the various social audit methodologies in use are discussed; and several value sensitive stages/elements of social audit research and evaluation research in general are identified. Recommendations set forth for minimizing the negative consequences of value-influenced evaluations are a value-explicit evaluation approach, greater standardization of evaluation procedures, a dialectic evaluation approach, and better informed evaluation consumers.