NCJ Number
47742
Journal
EVALUATION QUARTERLY Volume: 2 Issue: 1 Dated: (FEBRUARY 1978) Pages: 5-51
Date Published
1978
Length
47 pages
Annotation
SYSTEMATIC ATTEMPTS TO STUDY EVALUATIONS ARE REVIEWED, AND WAYS IN WHICH SUCH 'METAEVALUATIONS' CAN BE USED TO IMPROVE THE TECHNICAL QUALITY OF EVALUATION RESEARCH ARE SUGGESTED.
Abstract
THREE RESEARCH TRADITIONS ARE RELEVANT TO METAEVALUATION: (1) THE INVESTIGATOR ACQUIRES ANOTHER INVESTIGATOR'S EVALUATION DATA AND REANALYZES THEM TO ANSWER EITHER THE SAME QUESTIONS OR NEW QUESTIONS; (2) EVALUATIONS FROM DIFFERENT SOURCES ARE RATED IN ORDER TO ASSESS TECHNICAL COMPETENCE IN GENERAL; AND (3) 'RESEARCH ON RESEARCH' IS CARRIED OUT, (E.G., ATTEMPTS TO DETERMINE WHICH EVALUATION MONITORING TECHNIQUES PRODUCE THE QUICKEST FEEDBACK ON UNEXPECTED DESIGN OR MEASUREMENT PROBLEMS THAT EMERGE ONLY AFTER THE EVALUATION HAS BEEN IMPLEMENTED). FOUR STUDIES WHOSE FINDINGS INDICATE THAT THE TECHNICAL QUALITY OF EVALUATION IS IN NEED OF IMPROVEMENT ARE DESCRIBED. STUDIES ILLUSTRATIVE OF EACH OF THE THREE RESEARCH TRADITIONS ARE USED TO DEVELOP SEVEN MODELS OF HOW METAEVALUATION RESEARCH CAN IMPROVE THE TECHNICAL QUALITY OF EMPIRICAL SUMMATIVE EVALUATIONS. THE SEVEN MODELS RELATE TO THE FOLLOWING: (1) ESSAY REVIEWS OF EVALUATION REPORTS; (2) REVIEWS OF THE LITERATURE ABOUT A SPECIFIC PROGRAM; (3) EMPIRICAL REEVALUATION OF A PROGRAM EVALUATION; (4) EMPIRICAL REEVALUATION OF MULTIPLE DATA SETS ABOUT THE SAME PROGRAM; (5) CONSULTANT METAEVALUATION; (6) SIMULTANEOUS SECONDARY ANALYSIS OF RAW DATA; AND (7) MULTIPLE INDEPENDENT REPLICATIONS. EACH MODEL IS APPROPRIATE TO A DIFFERENT EVALUATION PROBLEM. IT IS POINTED OUT THAT METAEVALUATION IS NOT THE ONLY WAY OF IMPROVING THE QUALITY OF EVALUATIONS AND SHOULD NOT BE CONSIDERED A SUBSTITUTE FOR LONGER-TERM APPROACHES TO IMPROVING EVALUATION (E.G., RESEARCH ON RESEARCH METHODOLOGY).