Evaluating and Refining Expert Critiquing Systems: A Methodology*

Expert critiquing systems are a rapidly growing class of intelligent decision support systems that apply artificial intelligence techniques to criticize a user's proposed solution to a problem. Critic programs now exist in the medical, engineering, programming, knowledge acquisition, word processing, and other domains. Critic refinement is a nontrivial activity that, even when done well, consumes a sizable fraction of the complete effort to build and deploy a critiquing system. To ease that effort, it is important to adopt a rigorous approach that allows one to reproducibly measure the degree of success of the current critic version and to predict which refinements will improve the critic further. The current article presents one such approach with actual case studies that illustrate its usage and elaborate selected aspects of the refinement process.