Dealing with unknown variables in policy/program evaluation

A leading objection to the feasibility of applying benefit-cost analysis in evaluation is that one often cannot know the benefits, the costs, or the probabilities of their occurring for a given proposal. Threshold analysis involves handling unknown variables by converting relevant evaluation problems into questions as to whether a given benefit, cost, probability, or combination of these elements is more or less than a threshold. Above the threshold, the proposed program would be relatively profitable; and below the threshold, it would be relatively unprofitable. Applying that analysis may require the drawing of simple graphs to stimulate the minds of knowledgeable persons as to the range of the actual benefits, costs or probabilities. The approach can be applied when the benefits or the costs are either monetary or nonmonetary, and when the problems involve either go/no-go decisions or conflicting-choice decisions.     

Part/whole percentaging as a useful method in policy/program evaluation

Part/whole percentaging analysis is a simple and meaningful way of evaluating alternative policies, decisions, or other options with multiple goals measured on different dimensions. That kind of analysis involves converting the raw scores of each alternative policy on each goal into a part/whole percentage by summing the raw scores for each goal and dividing each raw score by the total. One can then sum the part/whole percentages across the goals for each policy after assigning at least tentative relative weights to each goal. Part/whole percentages are added for goals that one would like to maximize and are subtracted for goals that one would like to minimize. Where the relative weight of a goal or score is too difficult to determine, one can determine the threshold value at which there is a tie between the tentative first choice and the second choice, and then just determine whether the actual value is above or below the threshold value. The aggregated part/whole percentages for each policy can be used to choose the best policy, the best subset of policies, or to allocate a budget across the policies in proportion to their aggregated part/whole percentages.     

Comparative influence of political, cost/benefit, and statistical information on administrative decision making.

The influence of three types of evaluation information was studied in simulated decisionmaking situations within a community mental health center setting. Twelve administrators reviewed one of two decision problems (direct services or indirect services funding) and were then presented political, cost/benefit and statistical information. After each presentation, the administrators rated the importance of each type of information for decision making. Only the cost/benefit data were rated as significantly influencing their decisions. The influence of each type was not dependent upon the decision problem, nor to length of time in administrative positions. The results were interpreted in terms of “who says what, how, and to whom. ” If evaluators desire a greater impact on the decision making behavior of community health administrators, they should consider employing a pay off-based evaluation strategy and providing cost/benefit information.     

Measures of occupational status in a special population

Previous research has shown that indirect measures of occupational status such as the Duncan Socioeconomic Index (SEI) are more valid than direct measures of occupational prestige, such as those developed by the National Opinion Research Center and by Treiman, for analyses of social mobility. The Professional, Technical, and Kindred (PTK) occupations available for the SEI are a biased sample of all PTK occupations. Do findings on the validity of the SEI pertaining to the general population hold for special populations, such as college graduates? Confirmatory factor analysis comparing the validities of four measures of occupational SES—the SEI, a 1960 counterpart of the SEI developed by Siegel and based on all Census occupations, the NORC prestige scale, and the Treiman international prestige scale—showed that the indirect measure developed by Siegel was clearly more valid than the other three measures. Since these findings indicate that the predictors in a regression equation are more valid than the criterion, there is some question as to what the criterion should really be.     

A dynamic macro social indicator model of changes in marriage, family, and population in the United States: 1947–1974

This paper presents an integrated 21-equation model of how marriage, family, and population conditions, as indexed by macro social indicators, affect each other and are affected by other social, demographic, and economic forces. An opportunity structures theoretical paradigm is applied to the specification of dynamic structural equations for determining changes (both trends and cyclical fluctuations) in marriage formation and dissolution, family and household composition, fertility, mortality, population growth, and population distribution. The equations are estimated on annual national data for the United States during the post-World War II years 1947 to 1972, and then they are used to make conditional forecasts of the values of some of the endogenous variables for 1973 and 1974. It is found that the equations fit the observed data well, lack demonstrable autocorrelation of disturbances, and forecast the 1973 and 1974 values usually with less than 2% error. Strategies are sketched for further refining some of the equations, and it is suggested that this model should be integrated into a larger societal model in order to estimate some of the effects of changes in demographic phenomena on other social conditions.     

Socioeconomic indexes and the new 1980 census occupational classification scheme

Because of their usefulness in a wide variety of research, socioeconomic scores have been revised several times to incorporate changes in the relevant components of the occupational structure and to conform with the changing census occupational classification schemes. The major purpose of this paper is to provide socioeconomic scores for the 1980 census occupational classification scheme. First, however, we review earlier versions of the socioeconomic index (SEI), summarize the relevant disparities between the 1970 and 1980 census occupational classificatory schemes, and then explain in detail the procedures used to reconcile the SEI with 1980 occupational codes. We also show that, although the 1980 occupational classificatory scheme differs radically from the 1970 occupational scheme, the socioconomic scores for 1980 occupational data replicate features found in 1970 data.     

Comparing measures of homicide trends: Methodological and substantive differences in the Vital Statistics and Uniform Crime Report time series (1933–1975)

Several Uniform Crime Reports (FBI) and Vital Statistics (NCHS) homicide time series for the United States, 1933–1975, are compared with respect to (a) definitions and collection procedures used; (b) differences in estimating procedures; and (c) substantive differences that result from use of either series. Among the findings from this investigation are that: (1) the sampling procedures employed by the NCHS appear to be superior to those of the FBI; (2) the FBI time series most commonly employed by researchers is discontinuous prior to 1958 due to revision procedures which were instituted at that time. Analyses over time periods which began prior to this year should employ a more accurate FBI time series which is published by the Office of Management and Budget (1973). (3) The specific time period under study may lead to different substantive conclusions with different indicators; (4) the results of statistical analyses of structural equation models contained herein indicate that while the difference is not generally statistically significant, the deterrent effect of execution is consistently estimated to be greater with the Vital Statistics series. This finding has important policy implications if estimates of the number of lives saved by executing prisoners are projected and utilized by policy makers. (5) Neither series indicates serious problems of autocorrelation of disturbances with the exogenous variables employed here; and (6) use of either series for the period after 1949 appears to produce the same substantive results.     

The development and field testing of a vocational education evaluation model

Reported is the development and field testing of an evaluation model used to evaluate the vocational education programs at the Metropolitan Community Colleges (MCC) of Kansas City, Missouri which can be implemented on vocational programs at other institutions. The model evaluated a vocational program on the following six criteria: (1) program's relationship to job market profile; (2) program's level of community support; (3) program's success in meeting vocational aspirations of clientele; (4) program's success in terms of student performance; (5) program 's cost-effectiveness; and (6) program 's success in reaching the handicapped and disadvantaged. The evaluation model provides a strategy and a set of procedures for assessing the quality of existing vocational education programs of study at any post-secondary institution. The model can be easily and inexpensively implemented, thus providing a data base which is useful for administrative planning and decision making.