Dimensions and design criteria for developing training reaction evaluations

Although there is research on training program evaluation, there is little systematic research on the design, development and use of training reaction evaluations. In order to obtain meaningful information from a program evaluation, evaluation professionals must be rigorous in the design and development of all aspects of an evaluation study, including instrumentation. The purposes of the study were to classify the dimensions of information sought using reaction evaluations and to establish design criteria for developing training reaction instruments. There were three major phases of the study: (1) classification of the dimensions and questionnaire design criteria used in reaction evaluations; (2) validation of the classified dimensions and the questionnaire design criteria by subject matter experts; and (3) assessment of a sample of training reaction instruments currently used in training programs in US corporations. The research findings were: eleven dimensions for reaction evaluation were identified and classified by purpose. Five overall design criteria, each consisting of several sub-criteria, were judged important in the design of reactionnaires. These include: introduction and directions; question format; question construction; questionnaire layout; and data analysis. It was concluded that a well-designed training reaction instrument integrates the proper application of design criteria with appropriate reaction dimensions. Most training reaction instruments used by US corporations consisted of questions representing only a few dimensions. The instruments varied in form and length. Few of them properly utilized the established questionnaire design criteria.     

Using Conditioned Reinforcers to Improve Behavior-Change Skills: Clicker Training for Practitioners

We evaluated the use of a clicker sound as a conditioned reinforcer to teach behavior interventionists appropriate clinical skills during in-situ training. A multiple-baseline-across-behaviors design indicated that the intervention was effective in increasing all target behaviors for all participants. For two of the participants, we conducted maintenance probes one to five weeks after the final training session and performance continued to occur at a high level. Although the participants ranked the contingent clicker presentation as less disruptive and more fair than typical feedback methods, they ranked it as slightly less helpful, less worthwhile, and less pleasant than typical feedback methods. These findings suggest that clicker training can play a valuable role in training mental health professionals.     

A comparison of the practices used by human resource development professionals to evaluate web-based and classroom-based training programmes within seven Korean companies

The purpose of the study was to compare the practices used by human resource development (HRD) professionals to evaluate web-based and classroom-based training (CBT) programmes within seven Korean companies. This study used four components of evaluation and three factors of evaluation barriers to compare the differences between these two training approaches. This study also explored the key decision factors for determining how HRD professionals evaluated their web-based and CBT programmes. Two data sets were used for the study; one set of data was gathered from a survey questionnaire distributed to HRD professionals and the other was gathered from interviews with HR/HRD directors within the seven companies. The results showed that web-based and CBT programmes were not meaningfully different on the most components of evaluation and evaluation barriers. The results also found six key decision factors determining evaluation for web-based and CBT programmes.     

Improving the quality of evaluation participation: a meta-evaluation

This study presents a meta-evaluation of a beta-test undertaken of a customer service training programme. The initial evaluation showed a low level of participation in the beta-test evaluations. Therefore, the meta-evaluation (or an evaluation of the evaluation) focused on issues related to the conduct of the initial evaluation and reasons for non-response. The meta-evaluation identified solutions to the participation problem as related to authority, capacity and motivation.     

Diversity training pitfalls and possibilities: An exploration of small and mid-size US organizations

Abstract

This qualitative, exploratory study examines the development and implementation of diversity training in small and mid-size organizations in a Midwestern region of the United States. The results are contrasted with diversity best practices and found lacking in key elements identified as most likely to yield success. Specifically, the organizations in this study did not provide training beyond awareness level, failed to conduct front-end needs assessment, provided little if any in-depth evaluation of diversity training and showed varying support for diversity endeavors from the leadership and through connecting diversity training with other systemic initiatives. The article concludes with recommendations for additional exploration of adapting effective diversity training practices to minimize pitfalls and maximize possibilities in small and mid-size systems.     

The use of a personal development plan and the undertaking of learning activities,expertise-growth,flexibility and performance: the role of supporting assessment conditions

In the current search for tools that encourage and assess learning and development, personal development plans (PDPs) are being used ever more frequently by organizations. A PDP is an assessment tool used by the employee to reflect on, to document the competencies s/he has been working on and to present his/her plans for further development. This study conducted among employees working in a governmental organization in the Netherlands (N = 287) focused on the PDP practice as conceptualized by three supporting conditions: learning and reflection, information and feedback and the motivating supervisor. It examined which of these features enhance the undertaking of learning activities, expertise-growth, flexibility towards changing circumstances and performance. Results from the hierarchical regression analyses indicate that a motivating supervisor, information and feedback and reflection by the employee on the basis of his/her PDP affect whether or not a PDP contributes to the four measured output variables. Next, by conducting mediation analyses evidence is found for the mediating role of undertaking learning activities in the relation between the PDP practice and two output variables.     

Aspects of quality . . . and world domination!

Training, learning and development tend to be regarded as central to the success of every organization and the nation. However, not all organizations are able or willing to invest resources in such processes. The paper presents a view that training, learning and development occur as a consequence of successful persuasive actions in a chain of talk, which draw on the discursive and rhetorical resources of an emerging learning movement. Utilizing the findings from an examination of data from the UK Department of Education and Skills-sponsored National Training Award winners, the paper provides an employment of actor-network theory and the sociology of translation to show how some organizations make the case for training, learning and development as a virtuous endeavour. It concludes by suggesting that what is needed is a more detailed understanding of how the case for the adoption of training, learning and development, or not, within organizations is made meaningful.     

“How Much is it Worth to You? Subjective Evaluations of Help in Organizations”

Helping behavior is a fundamental aspect of life in organizations, but subjective evaluations of giving and receiving help often diverge. What one employee believes is a generous act might seem insignificant to another. Such disparity in evaluations of helpful acts can negatively affect interpersonal cooperation among peer employees. In this article, I review research on the topic of subjective evaluation in social exchange, present a preliminary model of how members of organizations construct evaluations of help, and explain how these evaluations may shape their work experiences. I conclude by discussing the implications of this framework for research on employee exchange and outline some directions for future research.     

A Comparison of Pyramidal Staff Training and Direct Staff Training in Community-Based Day Programs

This study evaluated two methods of training staff who were working with individuals with developmental disabilities: pyramidal training and consultant-led training. In the pyramidal training, supervisors were trained in the principles of applied behavior analysis (ABA) and in delivering feedback. The supervisors then trained their direct-care staff and, subsequently, provided ongoing performance feedback. In the consultant-led training, the direct-care staff was trained in the principles of ABA. Both groups learned similar amounts of knowledge of ABA principles, but pyramidal training was more effective in teaching staff to use correct teaching procedures with consumers. In addition, the pyramidal training group maintained the improvement in their teaching procedures at a 3-month follow up, while the consultant-led training group's performance declined.     

Managers' personal values as predictors of importance attached to training and development: a cross-country exploratory study

Few empirical studies have examined the effect of personal values on the importance attached by individuals to training and development in organizations. We argue that personal values play an important role in decision-making processes (i.e. commitment to training and development) and that such values are the product of socialization processes operating at an organizational and societal level. Questionnaire data were collected from 340 Irish and Canadian line managers to test the hypothesis that personal values affect the importance attached by respondents to training and development. Capability values were found to be a significant positive predictor of the perceived importance of training and development. The findings emphasize the need for simultaneously examining both personal values and organizational factors as predictors of training and development activity.     

Sources of differences between Chinese,Indian, and Caucasian scientists and engineers: Evaluations of managerial potential

This article proposes intrapersonal, interpersonal, and acculturation factors to help explain differences in evaluations of the managerial potential of Chinese, Indians, and U.S. Caucasians. Using data from 1658 scientists and engineers and their managers in 24 U.S. companies, we find that managers evaluate Indians (but not Chinese) to have less managerial potential and to be less effective at working with others than Caucasians. The absence of insecurity in Indians’ self-reported English fluency appears to be detrimental to the evaluation of their managerial potential. In contrast, Indians who report high managerial aspirations receive less harsh evaluations of their managerial potential. We discuss the implications of our findings for future research and practice.     

基于胜任素质模型的培训系统构建

在回顾将胜任素质模型应用于培训活动相关研究成果的基础上,构建了基于胜任素质模型的培训系统.然后,描述了基于胜任素质模型的培训需求分析、培训计划设计以及培训效果评估的过程.最后,从与人力资源管理体系其他模块互动作用的角度对该培训系统进行了应用研究的探讨.     

Evaluating leadership coaching: A review and integrated framework

Leadership coaching reflects an evolving dynamic between the client and coach that is qualitatively different from most approaches to leadership development and therefore holds particular challenges for evaluation. Based on reviews of academic and practitioner literatures, this paper presents an integrated framework of coaching evaluation that includes formative evaluations of the client, coach, client–coach relationship, and coaching process, as well as summative evaluations based on coaching outcomes. The paper also includes a quantitative synthesis examining evaluation methodologies in 49 leadership coaching studies. The results revealed that self-reported changes in clients' leadership behaviors are the most frequently assessed coaching outcome, followed by clients' perceptions of the effectiveness of coaching. Recommendations to advance coaching evaluation research include the creation of collaborative partnerships between the evaluation stakeholders (client, coach, client's organization, and coaching organization) to facilitate systematic formative evaluations, the collection of multi-source and multi-level data, and the inclusion of distal outcomes in evaluation plans.     

The Effects of Evaluating Video Examples of Staffs' Own Versus Others' Performance on Discrete-Trial Training Skills in a Human Service Setting

Many studies have been conducted evaluating the use of feedback in staff training in organizational settings. Central to this literature has been the use of a variety of forms of feedback, including videotaped feedback. A distinction is outlined between video modeling and a variety of possible video feedback procedures. Previous studies have indicated a possible positive training effect on staff skills from simply being exposed to relevant videotaped performances of those skills or as a result of evaluating video modeled performances. This study evaluated the effectiveness for training teaching skills of having staff evaluate videotaped teaching skills as performed by themselves versus others. The current study results replicate earlier research on the effectiveness of evaluating video models as a training method and do not suggest a difference between observing oneself as opposed to observing someone else. Future research suggestions for the general use of video in training are provided.     

An Evaluation of Train-to-Code to Teach Implementation of PECS©

We evaluated the impact of a customized training program developed using an observation and behavioral coding software system (i.e., Train-to-Code) to teach implementation of Phase 3A of the Picture Exchange Communication System to four undergraduate students. The training program coached participants on all relevant steps of the procedure. To accomplish this, participants viewed multiple video exemplars of correct and incorrect implementation of each step in the sequence of behaviors, and coded these behaviors in accordance with a taxonomy developed by the experimenters. The training program provided prompts and feedback in real time based on participants’ current level of performance and required mastery of seven levels of training, each with fewer prompts and feedback, until an expert level of unprompted coding performance was demonstrated. A multiple baseline design across participants was used to evaluate the effectiveness of the training program on levels of procedural integrity during performance test probes conducted with a confederate learner before and after the completion of the training program. Results showed improved performance relative to baseline following training, and maintenance of performance at 2–4 weeks follow-up. A conceptual analysis of these findings will be discussed, along with implications for staff training in applied settings.     

Changing Patterns of Residential Care

The use of soft-tie restraint, a common means of controlling the assaultive behavior of mentally retarded residents in institutions, was examined with a retrospective design. Naturally occurring changes over a 39-month period were divided into (a) changes in management personnel that were associated with changes in management support of behavioral programming, (b) changes in staff training and feedback, and (c) a change in the physical location of the Program. Use of restraint decreased when the use of behavioral programming was supported by management and increased when it was not. When a four-person Training Team provided staff with training and formal and informal fecdback, restraint was used less than it was when the Training Team was outside the Program and training and feedback were not provided. Changing the physical location of the Program had little effect on the use of restraint. Relative to baseline, use of restraint decreased by 80% when management supported the use of behavioral programming and the Training Team was in the Program providing training and feedback. Reductions in the use of restraint were correlated with a corresponding reduction in Special Incident Reports which were filed when an event such as injury to a resident or staff member occurred.     

Citizen Advisory Committees and Environmental Policy: What We Know, What's Left to Discover

Citizen Advisory Committees (CACs) are being used in increasing numbers to provide public input into environmental policy and management decisions. While there is a large body of literature consisting of guidelines for establishing and running CACs, the body of literature of empirical evaluations of CACs is markedly smaller. Fourteen empirical studies of CACs involved in environmental policy decisions (spanning the period from 1976 to 1994) are reviewed here, including case studies, large-scale surveys, and consultant reports. For each study consideration is given to the methods of study, the issues considered by the CACs, the organizations advised, the definitions of success used in the study, and the suggested factors contributing to the success, or lack of success, experienced by the CACs. The review shows that the influence of CACs on policy outcomes have varied from case to case, with some accomplishing little and others having significant policy impacts. The increased use of CACs by government and industry presents an excellent opportunity for academics and practitioners to strengthen their understanding of the internal processes and capabilities of CACs through comparative evaluation research.     

Evaluating training and development practices in Taiwan: challenges and opportunities

The purpose of this article is to explore the status and challenges of employee training and development (T&D) in Taiwan. Literature review suggested that government intervention, vocational and technical education, social network, and organizational structures are the key factors that influence T&D practices in Taiwan. Taiwanese government plays a major role in promoting value-added production and skilled workforce in the organizations via developing various national labour enhancement policies, strategies and regulations and providing a set of training subsidies for employee T&D. The promotion of Taiwan TrainQuali System in Taiwan is one of the major national employee T&D policies that motivates and forces the corporations to cultivate their manpower. The challenges are evident when corporations adapt and apply Western T&D models or programmes. This study investigates T&D applications in Taiwan and provides a guide to practitioners to overcome challenges in terms of implementing T&D practices in Taiwan. Implications and recommendations for practice and future research were also discussed.     

A Comparison of Immediate and Post-Session Feedback with Behavioral Skills Training to Improve Interview Skills in College Students

ABSTRACT

Successful interviewing skills help maximize the probability that a job candidate will make a positive impression upon a prospective employer. Previous research described a method to increase appropriate interview skills using Behavioral Skills Training (BST) with post-session feedback. Immediate feedback has been shown as an effective method that may improve the effectiveness and efficiency of the training package described by previous research. The purpose of this study was to replicate past studies using BST to improve interview skills of college students and extend the study by comparing post-session and immediate feedback. All participants demonstrated improvements in interview skills, thus replicating previous findings. More specifically, BST with immediate feedback showed greater acquisition, maintenance, and generalization, with fewer training minutes required to meet mastery criteria compared to BST with post-session feedback.     

A Parametric Analysis of Rehearsal Opportunities on Procedural Integrity

Behavioral skills training (BST) involves instruction, modeling, rehearsal, and feedback. Despite its empirical support, the literature provides little to no guidance regarding the best way to implement rehearsal to ensure accurate implementation of behavioral procedures. Thus, the purpose of this study was to conduct a parametric analysis of rehearsal opportunities within a BST package. We evaluated the effects of 1, 3, and 10 rehearsals with feedback on correct implementation of functional analysis (FA). Regardless of the number of rehearsals, 18 undergraduate participants effectively learned to implement FA conditions. The most efficient training involved the use of single rehearsals with feedback until meeting mastery criterion. Participants maintained integrity during a follow-up session and rated the use of rehearsal as highly acceptable.