Exploring training needs of educational program evaluators in the Asia-Pacific region

Training is critical for the growth of the evaluation profession as more qualified and competent individuals will be required to meet the increased demand for evaluation. This paper deals with the training needs of professionals and is based on a part of a study on the status and challenges of educational program evaluation in the Asia-Pacific region. A web-based Delphi was used with a needs assessment framework for examining the current nature of training, what should be done, and discrepancies (gaps) between the two. The results should be of value for academic training programs and those in non-academic settings.     

Practices and challenges in educational program evaluation in the Asia-Pacific region: results of a Delphi study

While educational program evaluation has become more important in recent years because of increasing governmental demands for accountability, little is known about the development of and issues in regard to this topic in the Asia-Pacific region. The findings from a Delphi study conducted in a number of relevant countries are provided in this article. Thirty panelists participated in three rounds of iterations to identify current status, challenges, and concerns in the near future. The results should be of value to educational evaluators, researchers, and decision makers.     

Ethical problems in program evaluation: Advice for trapped evaluators

This paper argues that ethical problems in program evaluation are increased when conflicting or incompatible evaluation models are applied concurrently. Three models are illustrated: an Amelioration Model, generating better information for a program's own decision makers; an Accountability Model, focusing on public data disclosure and citizen participation in evaluation; and an Advocacy Model, in which the evaluation is designed to advance the program's interests in the competition for resources. Ethical problems inherent in each model, and arising from combining models are presented, along with case examples and discussion. The authors suggest several directions that program evaluators can consider to reduce their vulnerability to ethical problems.     

Essential competencies for program evaluators in a diverse cultural context

Essential evaluator competencies as identified by Stevahn, King, Ghere, and Minnema (2005) were studied in regard to how well they generalize to an Asian (Taiwan) context. A fuzzy Delphi survey with two iterations was used to collect data from 12 experts. While most competencies fit Taiwan, there were a number of unique ones. A complete set of results is provided along with the implications of the findings and what they might mean for evaluation in Taiwan particularly in relationship to the professionalization of evaluation.     

Essential competencies for program evaluators in a diverse cultural context

Essential evaluator competencies as identified by Stevahn, King, Ghere, and Minnema (2005) were studied in regard to how well they generalize to an Asian (Taiwan) context. A fuzzy Delphi survey with two iterations was used to collect data from 12 experts. While most competencies fit Taiwan, there were a number of unique ones. A complete set of results is provided along with the implications of the findings and what they might mean for evaluation in Taiwan particularly in relationship to the professionalization of evaluation.     

The inherently political nature of program evaluators and evaluation research

This paper discusses the importance of addressing the political dynamics inherent in any evaluation research process. The theoretical orientation contributing to the denial and avoidance of politics is articulated and contrasted with one which posits politics as normative and useful. A connection is drawn between stakeholder and case study approaches to evaluation. It is argued that both are effective routes to evaluations which facilitate decision making. A case example of a study of Community Development Block Grant agencies is used to detail problems accompanying avoidance of politics. Guidelines for effectively addressing and using politics are offered.     

Cost-utility analysis in educational needs assessment

Educational needs assessments (hereafter NAs) are inattentive to cost considerations and are frequently dominated by elite stakeholder groups. In this article I make a case for adopting a cost–utility approach, illustrating the argument with data generated in a NA of central library services in a Canadian school district. Using survey data from eight stakeholder groups, I found that (1) NAs based on the service preferences of a single stakeholder group can be misleading; (2) service preferences can be integrated into a single set of priorities, even when there are disagreements, by using the stakeholder group as the unit of analysis and assigning weights that privilege input from knowledgeable respondents; and (3) that the ranking of service operations produced by user preferences was not significantly correlated with the ranking produced by integrating preferences with costs. Cost–utility analysis would be more helpful if the utilities represented rigorously determined benefits of the services assessed, as well as stakeholder perceptions of the value of these benefits. Cost–utility analysis in NA will not reach its potential until cost considerations are routinely included in educational program evaluations.     

A children, teachers and designers as evaluators of usability of educational software

This article sets out to discuss the results of evaluations of usability made by children, teachers and designers through methods targeted on their profiles and contexts of use. Based on a survey of methods for evaluating usability, a field study was conducted in which such users--stakeholders--explored the educational software called "Mundo da Crian?a" ("Child's World"). The focus of this study was the analysis of this system and principally of the appropriateness of methods as to the avenues they suggest such users and contexts might explore.     

Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

Meeting the training needs of GP registrars in child abuse and neglect

A questionnaire survey undertaken among general practitioner registrars in North Thames in 1998 revealed significant deficiencies in their child protection training. A minority expressed confidence at the prospect of dealing with child protection cases in the future. A new, interactive training package was subsequently developed and delivered to a sample of GP registrars in Northeast Thames in 1999. Preliminary evaluation confirmed that the new training material was sensitive to the needs of recipients and levels of perceived confidence with participation in child protection work were increased. Further work is needed to develop the package and to oversee its successful incorporation into general practitioner vocational training. Copyright © 2001 John Wiley & Sons, Ltd.     

Organizing evaluation to serve the needs of program planners and managers

Evaluation is often both conceptually and functionally separate from program planning and operations management. Though program managers may not see themselves as served well by the work of evaluation staffs, there are a number of potentially beneficial side effects to be realized from close collaboration between program administrators and evaluators. These flow only when the evaluation task is broadly defined and when both administrators and evaluators recognize their interdependence. Particular functional benefits to management from evaluation are described. Characteristics of interorganizational relations which might increase the probability of their attainment are then outlined and discussed.     

Exploring mental distress among immigrant mothers participating in parent training

This study of ethnic minority mothers assessed the intervention effects of Parent Management Training — Oregon Model (PMTO) on maternal mental distress. Ninety-six mothers from Somalia and Pakistan and their 3-year-old to 9-year-old children were randomized with respect to enrollment in PMTO or a wait-list condition. Immigrants living in European countries report having significantly more mental distress than natives. Surprisingly, the results in this current study showed that there were low levels of mental distress at enrollment in PMTO among the sample. An analysis of covariance showed that PMTO was not effective in alleviating maternal mental distress in this sample. Ethnicity, family size and the child's age served as moderators on the relationship between enrollment in PMTO or the wait-list condition and maternal mental distress outcomes. None of the subgroup analyses were in favor of the intervention. The results emphasize the importance of research on parent training with immigrant families.     

Internship training in computer science: Exploring student satisfaction levels

The requirement of employability in the job market prompted universities to conduct internship training as part of their study plans. There is a need to train students on important academic and professional skills related to the workplace with an IT component. This article describes a statistical study that measures satisfaction levels among students in the faculty of Information Technology and Computer Science in Jordan. The objective of this study is to explore factors that influence student satisfaction with regards to enrolling in an internship training program. The study was conducted to gather student perceptions, opinions, preferences and satisfaction levels related to the program. Data were collected via a mixed method survey (surveys and interviews) from student-respondents. The survey collects demographic and background information from students, including their perception of faculty performance in the training poised to prepare them for the job market. Findings from this study show that students expect internship training to improve their professional and personal skills as well as to increase their workplace-related satisfaction. It is concluded that improving the internship training is crucial among the students as it is expected to enrich their experiences, knowledge and skills in the personal and professional life. It is also expected to increase their level of confidence when it comes to exploring their future job opportunities in the Jordanian market.     

Effectiveness of a training program for ECT nurses

Electroconvulsive therapy (ECT) is a highly technical procedure requiring a team that consists of an anesthetist, a psychiatrist, a clinical nurse specialist, and recovery nurses. Traditionally, nursing education and training in the context of providing a safe and high standard of care has not been addressed. Ninety-two nurses from 42 different health agencies participated in a training program focusing on defibrillation, electrocardiogram (ECG) and electroencephalogram (EEG) monitoring, intubation, stimulus dosing, setting up the ECT equipment, and caring for the patient. A non-experimental, one-group, pretest-posttest research design was used in this study to evaluate the effectiveness of the training program for nurses working with ECT. Effective training for nurses was hypothesized to make a difference in the standards of practice and clinical effectiveness for patients undergoing ECT. Findings from this study indicated a major knowledge deficit in key components of ECT among nurses who have responsibilities in this area of nursing care. With effective training, nurses' confidence levels increased related to setting up the equipment, administering a double dose, helping with intubation, and using a defibrillator. If nurses are to effectively function as team members in the ECT procedure, they must receive the training necessary to prepare them for this important role. The results of this study support the recommendation of the Royal Australian and New Zealand College of Psychiatrists that ECT nurses should be appropriately trained in anaesthetic and resuscitation techniques and modern ECT practice.     

Protein needs of older adults engaged in resistance training: a review

Protein recommendations by some professional organizations for young adults engaged in resistance training (RT) are higher than the recommended dietary allowance (RDA), but recommendations for resistance-training older adults (>50 years old) are not well characterized. Some argue that the current RDA is adequate, but others indicate increased protein needs. Although concerns have been raised about the consequences of high protein intake, protein intake above the RDA in older adults is associated with increased bone-mineral density when calcium intake is adequate and does not appear to compromise renal health in older individuals with normal renal function. Individual protein needs for older adults in RT are likely highly variable according to health and training regimen, but an intake of 1.0-1.3 g x kg(-1) x day(-1) should adequately and safely meet the needs of older adults engaged in RT, provided that their energy needs are met.