A Deliberative Method for Ranking Risks (I): Overview and Test Bed Development

Risk ranking offers a potentially powerful means for gathering public input to help set risk-management priorities. In most rankings conducted to date, the categories and attributes used to describe the risks have varied widely, the materials and procedures have not been designed to facilitate comparisons among risks on all important attributes, and the validity and reproducibility of the resulting rankings have not been assessed. To address these needs, a risk-ranking method was developed in which risk experts define and categorize the risks to be ranked, identify the relevant risk attributes, and characterize the risks in a set of standardized risk summary sheets, which are then used by lay or other groups in structured ranking exercises. To evaluate this method, a test bed involving 22 health and safety risks in a fictitious middle school was created. This article provides an overview of the risk-ranking method and describes the challenges faced in designing the middle school test bed. A companion article in this issue reports on the validity of the ranking procedures and the level of agreement among risk managers regarding ranking of risks and attributes.     

Ecological Risk Ranking: Development and Evaluation of a Method for Improving Public Participation in Environmental Decision Making

This article reports an extension of the Carnegie Mellon risk-ranking method to incorporate ecological risks and their attributes. On the basis of earlier risk-perception studies, we identified a set of 20 relevant attributes for describing health, safety, and environmental hazards in standardized risk summary sheets. In a series of three ranking sessions, 23 laypeople ranked 10 such hazards in a fictional Midwestern U.S. county using both holistic and multiattribute ranking procedures. Results were consistent with those from previous studies involving only health and safety hazards, providing additional evidence for the validity of the method and the replicability of the resulting rankings. Holistic and multiattribute risk rankings were reasonably consistent both for individuals and for groups. Participants reported that they were satisfied with the procedures and results, and indicated their support for using the method to advise real-world risk-management decisions. Agreement among participants increased over the course of the exercise, perhaps because the materials and deliberations helped participants to correct their misconceptions and clarify their values. Overall, health and safety attributes were judged more important than environmental attributes. However, the overlap between the importance rankings of these two sets of attributes suggests that some information about environmental impacts is important to participants' judgments in comparative risk-assessment tasks.     

A Deliberative Method for Ranking Risks (II): Evaluation of Validity and Agreement among Risk Managers

A deliberative method for ranking risks was evaluated in a study involving 218 risk managers. Both holistic and multiattribute procedures were used to assess individual and group rankings of health and safety risks facing students at a fictitious middle school. Consistency between the rankings that emerged from these two procedures was reasonably high for individuals and for groups, suggesting that these procedures capture an underlying construct of riskiness. Participants reported high levels of satisfaction with their groups' decision-making processes and the resulting rankings, and these reports were corroborated by regression analyses. Risk rankings were similar across individuals and groups, even though individuals and groups did not always agree on the relative importance of risk attributes. Lower consistency between the risk rankings from the holistic and multiattribute procedures and lower agreement among individuals and groups regarding these rankings were observed for a set of high-variance risks. Nonetheless, the generally high levels of consistency, satisfaction, and agreement suggest that this deliberative method is capable of producing risk rankings that can serve as informative inputs to public risk-management decision making.     

A data envelopment analysis approach for ranking journals

We create a data envelopment analysis (DEA) model to rank business journals, using data from the Thomson Reuters Journal Citation Reports^® (JCR). As opposed to previous models that ranked journals in only one field and mostly relied on survey data, this model is used to rank 358 business journals from five different JCR categories according to such citation-based factors as the number of articles, the number of citations, impact factor, five-year impact factor, immediacy index, eigenfactor score, and article influence score. We compute relative efficiencies of the journals and thereby create plausible journal rankings that largely, but not completely, corroborate three widely used business publication journal ranking lists. In addition, we show how the different characteristics of the JCR data impact the DEA ranking model. Finally, we identify journals that are not on the business publication lists but consistently perform very well relative to those benchmark journals, and should possibly be included in the business publication ranking lists. We also identify journals whose inclusion in widely used business publication rankings cannot be justified by our methodology.     

Inductive Inference: An Axiomatic Approach

A predictor is asked to rank eventualities according to their plausibility, based on past cases. We assume that she can form a ranking given any memory that consists of finitely many past cases. Mild consistency requirements on these rankings imply that they have a numerical representation via a matrix assigning numbers to eventuality–case pairs, as follows. Given a memory, each eventuality is ranked according to the sum of the numbers in its row, over cases in memory. The number attached to an eventuality–case pair can be interpreted as the degree of support that the past case lends to the plausibility of the eventuality. Special instances of this result may be viewed as axiomatizing kernel methods for estimation of densities and for classification problems. Interpreting the same result for rankings of theories or hypotheses, rather than of specific eventualities, it is shown that one may ascribe to the predictor subjective conditional probabilities of cases given theories, such that her rankings of theories agree with rankings by the likelihood functions.     

COMPARISON OF CONSENSUS METHODS FOR PRIORITY RANKING PROBLEMS

Various consensus methods proposed for ranking problems yield controversial rankings and/or tied rankings which are vulnerable to considerable dispute. These include Borda-Kendall (BK) and minimum-variance (MV) methods. This paper compares three continuous (ratio-scale) consensus scoring methods with BK and MV ranking methods. One method, termed GM, is an eigenvector scaling of the geometric-mean consensus matrix. GM allows for (1) paired-comparison voting inputs (as opposed to all-at-once ranking), (2) pick-the-winner preference voting, and (3) ratio-scale preference voting. GM is relatively simple to calculate on small computers or calculators, and merging of “close” candidates into tied rankings can be achieved by using an e-threshold tie rule discussed in this paper. The GM method thus can be used for paired-comparison voting to calculate both a ratio-scaled consensus index (based on a consensus eigenvector) and a ranking of candidates that allows for ties between “close” candidates. Eigenvalue analysis is used as a means of evaluating voter inconsistencies.     

Stochastic Judgments in the AHP: The Measurement of Rank Reversal Probabilities

This paper presents a methodology for analyzing Analytic Hierarchy Process (AHP) rankings if the pairwise preference judgments are uncertain (stochastic). If the relative preference statements are represented by judgment intervals, rather than single values, then the rankings resulting from a traditional (deterministic) AHP analysis based on single judgment values may be reversed, and therefore incorrect. In the presence of stochastic judgments, the traditional AHP rankings may be stable or unstable, depending on the nature of the uncertainty. We develop multivariate statistical techniques to obtain both point estimates and confidence intervals of the rank reversal probabilities, and show how simulation experiments can be used as an effective and accurate tool for analyzing the stability of the preference rankings under uncertainty. If the rank reversal probability is low, then the rankings are stable and the decision maker can be confident that the AHP ranking is correct. However, if the likelihood of rank reversal is high, then the decision maker should interpret the AHP rankings cautiously, as there is a subtantial probability that these rankings are incorrect. High rank reversal probabilities indicate a need for exploring alternative problem formulations and methods of analysis. The information about the extent to which the ranking of the alternatives is sensitive to the stochastic nature of the pairwise judgments should be valuable information into the decision-making process, much like variability and confidence intervals are crucial tools for statistical inference. We provide simulation experiments and numerical examples to evaluate our method. Our analysis of rank reversal due to stochastic judgments is not related to previous research on rank reversal that focuses on mathematical properties inherent to the AHP methodology, for instance, the occurrence of rank reversal if a new alternative is added or an existing one is deleted.     

OPTIMAL ALLOCATIONS OF JOINT (COMMON) COSTS: A PARAMETRIC PROGRAMMING APPROACH

A parametric programming model for allocating joint costs is described and illustrated. Given M mutually exclusive missions and P alternative systems for accomplishing such missions, the model first determines the optimal choice of systems for all missions simultaneously. It then allocates both joint and separable costs such that no mission receives a greater cost allocation than any other mission alternative that might be suboptimal for the mission but nonoptimal when all missions are simultaneously considered. Although the model initially assumes a mission priority ranking, this assumption is relaxed later on as alternative rankings are evaluated and cost allocation ranges under all priority rankings are evaluated.     

On Academic Rankings,Unacceptable Methods,and the Social Obligations of Business Schools

Inspired by recent discussions of the systematic costs that external rankings impose on academic institutions, and the undeniable shifts in the landscape of institutional data, a concerted and pragmatic re‐evaluation of ranking efforts has begun. In this study, multiple administrators and researchers representing both public and private institutions across the United States weigh in on these issues. While reaffirming the social contract we hold with society, we argue that the fundamental methodological shortcomings of existing rankings, and ultimately any ordinal ranking system, limit the value of current rankings. These shortcomings emerge from the conceptualization and the architecture of comparisons, and are evident in survey designs, data collection methods, and data aggregation procedures. Our discussion continues by outlining the minimal requirements that a socially responsible, transparent, flexible, and highly representative rating (vs. ranking) approach should employ. Ultimately, we call on academic institutions and organizing bodies to take a collective stand against existing rankings and to embrace the strategic use of multidimensional alternatives that faithfully serve prospective students, parents, and other key stakeholders. We conclude with a number of suggestions and opportunities for practice‐oriented research in the decision sciences aimed to support this fundamental shift in evaluative framing.     

A multi-criteria ranking algorithm (MCRA) for determining breast cancer therapy

Breast cancer is the leading cause of cancer deaths among women. The selection of an effective, patient-specific treatment plan for breast cancer has been a challenge for physicians because the decision process involves a vast number of treatment alternatives as well as treatment decision criteria, such as the stage of the cancer (e.g., in situ, invasive, metastasis), tumor characteristics, biomarker-related risks, and patient-related risks. Furthermore, every patient's case is unique, requiring a patient-specific treatment plan, while there is no standard procedure even for a particular stage of the breast cancer. In this paper, we first determine a comprehensive set of criteria for selecting the best breast cancer therapy by interviewing medical oncologists and reviewing the literature. We then present two analytical hierarchy process (AHP) models for quantifying the weights of criteria for breast cancer treatment in two sequential steps: primary and secondary treatment therapy. Using the weights of criteria from the AHP model, we propose a new multi-criteria ranking algorithm (MCRA), which evaluates a large variety of patient scenarios and provides the best patient-tailored breast cancer treatment alternatives based on the input of nine medical oncologists. We then validate the predictions of the multi-criteria ranking algorithm by comparing treatment ranks of the algorithm with ranks of five different oncologists, and show that algorithm rankings match or are statistically significantly correlated with the overall expert ranking in most cases. Our multi-criteria ranking algorithm could be used as an accessible decision-support tool to aid oncologists and educate patients for determining appropriate and effective treatment alternatives for breast cancer. Our approach is also general in the sense that it could be adapted to solve other complex decision-making problems in medicine, healthcare, as well as other service and manufacturing industries.     

FAMCDM: A fusion approach of MCDM methods to rank multiclass classification algorithms

Various methods and algorithms have been developed for multiclass classification problems in recent years. How to select an effective algorithm for a multiclass classification task is an important yet difficult issue. Since the multiclass algorithm selection normally involves more than one criterion, such as accuracy and computation time, the selection process can be modeled as a multiple criteria decision making (MCDM) problem. While the evaluations of algorithms provided by different MCDM methods are in agreement sometimes, there are situations where MCDM methods generate very different results. To resolve this disagreement and help decision makers pick the most suitable classifier(s), this paper proposes a fusion approach to produce a weighted compatible MCDM ranking of multiclass classification algorithms. Several multiclass datasets from different domains are used in the experimental study to test the proposed fusion approach. The results prove that MCDM methods are useful tools for evaluating multiclass classification algorithms and the fusion approach is capable of identifying a compromised solution when different MCDM methods generate conflicting rankings.     

Some Limitations of Aggregate Exposure Metrics

Aggregate exposure metrics based on sums or weighted averages of component exposures are widely used in risk assessments of complex mixtures, such as asbestos-associated dusts and fibers. Allowed exposure levels based on total particle or fiber counts and estimated ambient concentrations of such mixtures may be used to make costly risk-management decisions intended to protect human health and to remediate hazardous environments. We show that, in general, aggregate exposure information alone may be inherently unable to guide rational risk-management decisions when the components of the mixture differ significantly in potency and when the percentage compositions of the mixture exposures differ significantly across locations. Under these conditions, which are not uncommon in practice, aggregate exposure metrics may be "worse than useless," in that risk-management decisions based on them are less effective than decisions that ignore the aggregate exposure information and select risk-management actions at random. The potential practical significance of these results is illustrated by a case study of 27 exposure scenarios in El Dorado Hills, California, where applying an aggregate unit risk factor (from EPA's IRIS database) to aggregate exposure metrics produces average risk estimates about 25 times greater - and of uncertain predictive validity - compared to risk estimates based on specific components of the mixture that have been hypothesized to pose risks of human lung cancer and mesothelioma.     

Accommodating Uncertainty in Comparative Risk

Comparative risk projects can provide broad policy guidance but they rarely have adequate scientific foundations to support precise risk rankings. Many extant projects report rankings anyway, with limited attention to uncertainty. Stochastic uncertainty, structural uncertainty, and ignorance are types of incertitude that afflict risk comparisons. The recently completed New Jersey Comparative Risk Project was innovative in trying to acknowledge and accommodate some historically ignored uncertainties in a substantive manner. This article examines the methods used and lessons learned from the New Jersey project. Monte Carlo techniques were used to characterize stochastic uncertainty, and sensitivity analysis helped to manage structural uncertainty. A deliberative process and a sorting technique helped manage ignorance. Key findings are that stochastic rankings can be calculated but they reveal such an alarming degree of imprecision that the rankings are no longer useful, whereas sorting techniques are helpful in spite of uncertainty. A deliberative process is helpful to counter analytical overreaching.     

Extreme ranking analysis in robust ordinal regression

We extend the principle of robust ordinal regression with an analysis of extreme ranking results. In our proposal, we consider the whole set of instances of a preference model that is compatible with preference information provided by the DM. We refer to both, the well-known UTA^GMS method, which builds the set of general additive value functions compatible with DM's preferences, and newly introduced in this paper PROMETHEE^GKS, which constructs the set of compatible outranking models via robust ordinal regression. Then, we consider all complete rankings that follow the use of the compatible preference models, and we determine the best and the worst attained ranks for each alternative. In this way, we are able to assess its position in an overall ranking, and not only in terms of pairwise comparisons, as it is the case in original robust ordinal regression methods. Additionally, we analyze the ranges of possible comprehensive scores (values or net outranking flows). We also discuss extensions of the presented approach on other multiple criteria problems than ranking. Finally, we show how the presented methodology can be applied in practical decision support, reporting results of three illustrative studies.     

Evaluation of Multicriteria Decision Analysis Algorithms in Food Safety: A Case Study on Emerging Zoonoses Prioritization

Decision making in food safety is a complex process that involves several criteria of different nature like the expected reduction in the number of illnesses, the potential economic or health-related cost, or even the environmental impact of a given policy or intervention. Several multicriteria decision analysis (MCDA) algorithms are currently used, mostly individually, in food safety to rank different options in a multifactorial environment. However, the selection of the MCDA algorithm is a decision problem on its own because different methods calculate different rankings. The aim of this study was to compare the impact of different uncertainty sources on the rankings of MCDA problems in the context of food safety. For that purpose, a previously published data set on emerging zoonoses in the Netherlands was used to compare different MCDA algorithms: MMOORA, TOPSIS, VIKOR, WASPAS, and ELECTRE III. The rankings were calculated with and without considering uncertainty (using fuzzy sets), to assess the importance of this factor. The rankings obtained differed between algorithms, emphasizing that the selection of the MCDA method had a relevant impact in the rankings. Furthermore, considering uncertainty in the ranking had a high influence on the results. Both factors were more relevant than the weights associated with each criterion in this case study. A hierarchical clustering method was suggested to aggregate results obtained by the different algorithms. This complementary step seems to be a promising way to decrease extreme difference among algorithms and could provide a strong added value in the decision-making process.     

Augmenting the Deliberative Method for Ranking Risks

The Department of Homeland Security (DHS) characterized and prioritized the physical cross‐border threats and hazards to the nation stemming from terrorism, market‐driven illicit flows of people and goods (illegal immigration, narcotics, funds, counterfeits, and weaponry), and other nonmarket concerns (movement of diseases, pests, and invasive species). These threats and hazards pose a wide diversity of consequences with very different combinations of magnitudes and likelihoods, making it very challenging to prioritize them. This article presents the approach that was used at DHS to arrive at a consensus regarding the threats and hazards that stand out from the rest based on the overall risk they pose. Due to time constraints for the decision analysis, it was not feasible to apply multiattribute methodologies like multiattribute utility theory or the analytic hierarchy process. Using a holistic approach was considered, such as the deliberative method for ranking risks first published in this journal. However, an ordinal ranking alone does not indicate relative or absolute magnitude differences among the risks. Therefore, the use of the deliberative method for ranking risks is not sufficient for deciding whether there is a material difference between the top‐ranked and bottom‐ranked risks, let alone deciding what the stand‐out risks are. To address this limitation of ordinal rankings, the deliberative method for ranking risks was augmented by adding an additional step to transform the ordinal ranking into a ratio scale ranking. This additional step enabled the selection of stand‐out risks to help prioritize further analysis.     

Common benchmarking and ranking of units with DEA

This paper develops a common framework for benchmarking and ranking units with DEA. In many DEA applications, decision making units (DMUs) experience similar circumstances, so benchmarking analyses in those situations should identify common best practices in their management plans. We propose a DEA-based approach for the benchmarking to be used when there is no need (nor wish) to allow for individual circumstances of the DMUs. This approach identifies a common best practice frontier as the facet of the DEA efficient frontier spanned by the technically efficient DMUs in a common reference group. The common reference group is selected as that which provides the closest targets. A model is developed which allows us to deal not only with the setting of targets but also with the measurement of efficiency, because we can define efficiency scores of the DMUs by using the common set of weights (CSW) it provides. Since these weights are common to all the DMUs, the resulting efficiency scores can be used to derive a ranking of units. We discuss the existence of alternative optimal solutions for the CSW and find the range of possible rankings for each DMU which would result from considering all these alternate optima. These ranking ranges allow us to gain insight into the robustness of the rankings.     

Ranking irregularities when evaluating alternatives by using some ELECTRE methods

The ELECTRE II and III methods enjoy a wide acceptance in solving multi-criteria decision-making (MCDM) problems. Research results in this paper reveal that there are some compelling reasons to doubt the correctness of the proposed rankings when the ELECTRE II and III methods are used. In a typical test we first used these methods to determine the best alternative for a given MCDM problem. Next, we randomly replaced a non-optimal alternative by a worse one and repeated the calculations without changing any of the other data. Our computational tests revealed that sometimes the ELECTRE II and III methods might change the indication of the best alternative. We treat such phenomena as rank reversals. Although such ranking irregularities are well known for the additive variants of the AHP method, it is the very first time that they are reported to occur when the ELECTRE methods are used. These two methods are also evaluated in terms of two other ranking tests and they failed them as well. Two real-life cases are described to demonstrate the occurrence of rank reversals with the ELECTRE II and III methods. Based on the three test criteria presented in this paper, some computational experiments on randomly generated decision problems were executed to test the performance of the ELECTRE II and III methods and an examination of some real-life case studies are also discussed. The results of these examinations show that the rates of the three types of ranking irregularities were rather significant in both the simulated decision problems and the real-life cases studied in this paper.     

基于方案占优和排序稳健性的多属性决策方法

在多属性决策问题中,不同的属性权重会产生不同的评价结果。由于实际问题的复杂性与不确定性,决策者对于属性权重的确定也存在不确定性。这些不确定既来自现实问题的复杂性和可变性,也来自决策者选择的模糊性与随机性。目前已有的研究主要是将不确定的权重信息转化为相对确定的信息（如转化为区间数等）,硬性地消除了不确定,从而给决策结果带来较大风险。本文从方案排序的视角出发,研究在权重空间下,方案的占优关系和排序的稳健性。首先,定义了占优矩阵用于刻画不确定权重信息下方案两两比较的占优关系;其次,分析了方案的排序区间,即在所有可能存在的权重组合下,方案的最好排序和最差排序。然后,定义了方案的全排序排序概率,并且给出了排序概率的计算方法。进而,我们给出了方法的决策步骤和实施过程。最后,本文将该方法应用到某远洋集团的港口评估当中。     

The lean improvement of the chemical emissions of motor vehicles based on preference ranking: A PROMETHEE uncertainty analysis

The motor vehicle has provided mobility and individual freedom for millions of people. However, vehicles embody the dilemma of contemporary industrialisation in that the environmental costs of automobility are equally large. This non-country specific study undertakes a PROMETHEE-based preference ranking of a small set of motor vehicles based on constituents of their exhaust emissions. As a model of an interested party's preference ranking of the motor vehicles, the subsequent uncertainty (sensitivity) analysis considered here, relates to what minimal (lean) changes would be necessary to a vehicle's emissions so that their preference ranking is improved. For a particular manufacturer, it can identify the necessary engineering performance modifications to be made to improve their perceived consumer based ranking. This is compounded by a further consideration of different levels of importance conferred on the criteria (vehicle emissions) and analogous analyses undertaken. The visual elucidation of the results rankings and changes to criteria values, offers a clear presentation of the findings to the interested parties.