Applicability of standards for evaluations of educational programs, projects and materials in an international setting: Qualitative research

Qualitative studies are much more the practice in Europe and increasingly elsewhere than in the United States. For an international audience, therefore, there needs to be much greater emphasis on qualitative techniques than there is in the standards report. The most difficult ethical problem arises from the conflict between the right to privacy of research subjects and the right to know of audiences. By its nature, qualitative research focuses on the particular, the detailed analysis of the specific incident. It is therefore often difficult to conceal the identity of the individual organisations and people concerned. Even a confidential report might work to the detriment of the subjects studied. Published reports pose the same problem. Have subjects the right to have material they believe to their disadvantage excised from a report even if that renders the report valueless? There are two technical points in the standards document that should be amplified. The first concerns the verification of data. We need to specify procedures to ensure credibility for qualitative studies. The second concern is the specification of quantitative techniques for the analysis of qualitative data. The statement in the standards document is too weak. We need to define much more clearly the steps that should be taken to ensure that qualitative data can be tested rigorously. Finally, there is the problem of collusion, the danger that the qualitative evaluators will select from their material the data which are in conformity with their own predilictions or presuppositions, the expectations of their audience, or the requirements of their sponsors. This problem may be acute in some parts of the world. To be of maximum international value, the standards report needs to be extended and to pay much greater attention to problems arising with qualitative evaluation.