Stochastic Judgments in the AHP: The Measurement of Rank Reversal Probabilities

This paper presents a methodology for analyzing Analytic Hierarchy Process (AHP) rankings if the pairwise preference judgments are uncertain (stochastic). If the relative preference statements are represented by judgment intervals, rather than single values, then the rankings resulting from a traditional (deterministic) AHP analysis based on single judgment values may be reversed, and therefore incorrect. In the presence of stochastic judgments, the traditional AHP rankings may be stable or unstable, depending on the nature of the uncertainty. We develop multivariate statistical techniques to obtain both point estimates and confidence intervals of the rank reversal probabilities, and show how simulation experiments can be used as an effective and accurate tool for analyzing the stability of the preference rankings under uncertainty. If the rank reversal probability is low, then the rankings are stable and the decision maker can be confident that the AHP ranking is correct. However, if the likelihood of rank reversal is high, then the decision maker should interpret the AHP rankings cautiously, as there is a subtantial probability that these rankings are incorrect. High rank reversal probabilities indicate a need for exploring alternative problem formulations and methods of analysis. The information about the extent to which the ranking of the alternatives is sensitive to the stochastic nature of the pairwise judgments should be valuable information into the decision-making process, much like variability and confidence intervals are crucial tools for statistical inference. We provide simulation experiments and numerical examples to evaluate our method. Our analysis of rank reversal due to stochastic judgments is not related to previous research on rank reversal that focuses on mathematical properties inherent to the AHP methodology, for instance, the occurrence of rank reversal if a new alternative is added or an existing one is deleted.     

Exploiting value system multiplicity and preference voting for robust ranking

Under a data envelopment analysis (DEA) framework, full ranking of a group of decision making units (DMUs) can be carried out through an adequate amalgamation of the cross-efficiency (CE henceforth) scores produced for each DMU. In this paper, we propose a ranking procedure that is based on amalgamating the weight profiles selected over the cross-evaluation rather than related CE scores. The new approach builds, for each DMU, a collective weight profile (CWP henceforth) by exploiting the preference voting system embedded within the matrix of weights, which views the assessing DMUs as voters and the input/output factors as candidates. The occurrence of zero votes is discussed as a special case and a two-level aggregation procedure is developed. The CWPs that are produced extend the concept of collective appreciation to the input/output factors of each DMU so that group dynamics is truly reflected, mainly in decision making circumstances where factor prioritization is necessary for making choices or allocating resources. The robustness of the proposed ranking approach is evaluated with three examples drawn from the literature.     

The lean improvement of the chemical emissions of motor vehicles based on preference ranking: A PROMETHEE uncertainty analysis

The motor vehicle has provided mobility and individual freedom for millions of people. However, vehicles embody the dilemma of contemporary industrialisation in that the environmental costs of automobility are equally large. This non-country specific study undertakes a PROMETHEE-based preference ranking of a small set of motor vehicles based on constituents of their exhaust emissions. As a model of an interested party's preference ranking of the motor vehicles, the subsequent uncertainty (sensitivity) analysis considered here, relates to what minimal (lean) changes would be necessary to a vehicle's emissions so that their preference ranking is improved. For a particular manufacturer, it can identify the necessary engineering performance modifications to be made to improve their perceived consumer based ranking. This is compounded by a further consideration of different levels of importance conferred on the criteria (vehicle emissions) and analogous analyses undertaken. The visual elucidation of the results rankings and changes to criteria values, offers a clear presentation of the findings to the interested parties.     

Temptation and Revealed Preference

Gul and Pesendorfer (2001) model the static behavior of an agent who ranks menus prior to the experience of temptation. This paper models the dynamic behavior of an agent whose ranking of menus itself is subject to temptation. The representation for the agent's dynamically inconsistent choice behavior views him as possessing a dynamically consistent view of what choices he “should” make (a normative preference) and being tempted by menus that contain tempting alternatives. Foundations for the model require a departure from Gul and Pesendorfer's idea that temptation creates a preference for commitment. Instead, it is hypothesized that distancing an agent from the consequences of his choices separates normative preference and temptation.     

A robust ranking method extending ELECTRE III to hierarchy of interacting criteria,imprecise weights and stochastic analysis

A great majority of methods designed for Multiple Criteria Decision Aiding (MCDA) assume that all assessment criteria are considered at the same level, however, decision problems encountered in practice often impose a hierarchical structure of criteria. The hierarchy helps to decompose complex decision problems into smaller and manageable subtasks, and thus, it is very attractive for computational efficiency and explanatory purposes. To handle the hierarchy of criteria in MCDA, a methodology called Multiple Criteria Hierarchy Process (MCHP), has been recently proposed. MCHP permits to consider preference relations with respect to a subset of criteria at any level of the hierarchy. Here, we propose to apply MCHP to the ELECTRE III ranking method adapted to handle three types of interaction effects between criteria: mutual-weakening, mutual-strengthening and antagonistic effect. We also involve in MCHP an imprecise elicitation of criteria weights, generalizing a technique called the SRF method. In order to explore the plurality of rankings obtained by the ELECTRE III method for possible sets of criteria weights, we apply the Stochastic Multiobjective Acceptability Analysis (SMAA) that permits to draw robust conclusions in terms of rankings and preference relations at each level of the hierarchy of criteria. The novelty of the whole methodology consists of a joint consideration of hierarchical assessments of alternatives performances on interacting criteria, imprecise criteria weights, and robust analysis of ranking recommendations resulting from ELECTRE III. An example regarding the multiple criteria ranking of some European universities will show how to apply the proposed methodology on a decision problem.     

Extreme ranking analysis in robust ordinal regression

We extend the principle of robust ordinal regression with an analysis of extreme ranking results. In our proposal, we consider the whole set of instances of a preference model that is compatible with preference information provided by the DM. We refer to both, the well-known UTA^GMS method, which builds the set of general additive value functions compatible with DM's preferences, and newly introduced in this paper PROMETHEE^GKS, which constructs the set of compatible outranking models via robust ordinal regression. Then, we consider all complete rankings that follow the use of the compatible preference models, and we determine the best and the worst attained ranks for each alternative. In this way, we are able to assess its position in an overall ranking, and not only in terms of pairwise comparisons, as it is the case in original robust ordinal regression methods. Additionally, we analyze the ranges of possible comprehensive scores (values or net outranking flows). We also discuss extensions of the presented approach on other multiple criteria problems than ranking. Finally, we show how the presented methodology can be applied in practical decision support, reporting results of three illustrative studies.     

基于方案占优和排序稳健性的多属性决策方法

在多属性决策问题中,不同的属性权重会产生不同的评价结果。由于实际问题的复杂性与不确定性,决策者对于属性权重的确定也存在不确定性。这些不确定既来自现实问题的复杂性和可变性,也来自决策者选择的模糊性与随机性。目前已有的研究主要是将不确定的权重信息转化为相对确定的信息（如转化为区间数等）,硬性地消除了不确定,从而给决策结果带来较大风险。本文从方案排序的视角出发,研究在权重空间下,方案的占优关系和排序的稳健性。首先,定义了占优矩阵用于刻画不确定权重信息下方案两两比较的占优关系;其次,分析了方案的排序区间,即在所有可能存在的权重组合下,方案的最好排序和最差排序。然后,定义了方案的全排序排序概率,并且给出了排序概率的计算方法。进而,我们给出了方法的决策步骤和实施过程。最后,本文将该方法应用到某远洋集团的港口评估当中。     

基于交叉效率DEA与群体共识的区间乘性语言偏好关系群决策

针对区间乘性语言偏好关系群决策问题，提出了一种基于交叉效率DEA和群体共识的群决策方法。首先，提出乘性语言偏好关系导出函数的定义，并构建产出导向的DEA模型，证明了一致性乘性语言偏好关系的DEA效率得分与排序向量之间存在比例关系。在此基础上，建立基于理想值的交叉效率DEA模型，提出乘性语言偏好关系的通用排序方法。同时，基于群体共识建立目标规划模型来计算各语言偏好关系的权重系数。最后，利用Monte Carlo随机模拟的方法对群体语言偏好空间进行统计分析，得到群决策期望排序向量及其可信度。算例分析表明本文方法能够有效的避免信息损失，具有较强的适用性和较高的可信度。     

On Academic Rankings,Unacceptable Methods,and the Social Obligations of Business Schools

Inspired by recent discussions of the systematic costs that external rankings impose on academic institutions, and the undeniable shifts in the landscape of institutional data, a concerted and pragmatic re‐evaluation of ranking efforts has begun. In this study, multiple administrators and researchers representing both public and private institutions across the United States weigh in on these issues. While reaffirming the social contract we hold with society, we argue that the fundamental methodological shortcomings of existing rankings, and ultimately any ordinal ranking system, limit the value of current rankings. These shortcomings emerge from the conceptualization and the architecture of comparisons, and are evident in survey designs, data collection methods, and data aggregation procedures. Our discussion continues by outlining the minimal requirements that a socially responsible, transparent, flexible, and highly representative rating (vs. ranking) approach should employ. Ultimately, we call on academic institutions and organizing bodies to take a collective stand against existing rankings and to embrace the strategic use of multidimensional alternatives that faithfully serve prospective students, parents, and other key stakeholders. We conclude with a number of suggestions and opportunities for practice‐oriented research in the decision sciences aimed to support this fundamental shift in evaluative framing.     

A New Approach to the Group Ranking Problem: Finding Consensus Ordered Segments from Users’ Preference Data

The group ranking problem involves constructing coherent aggregated results from users’ preference data. The goal of most group ranking problems is to generate an ordered list of all items that represents the user consensus. There are, however, two weaknesses to this approach. First, a complete list of ranked items is always output even when there is no consensus or only a slight consensus. Second, due to similarity of performance, in many practical situations, it is very difficult to differentiate whether one item is really better than another within a set. These weaknesses have motivated us to apply the clustering concept to the group ranking problem, to output an ordered list of segments containing a set of similarly preferred items, called consensus ordered segments. The advantages of our approach are that (i) the list of segments is based on the users’ consensuses, (ii) the items with similar preferences are grouped together in the same segment, and (iii) the relationships between items can be easily seen. An algorithm is developed to construct the consensus of the ordered segments from the users’ total ranking data. Finally, the experimental results indicate that the proposed method is computationally efficient, and can effectively identify consensus ordered segments.     

An integrated method for cognitive complex multiple experts multiple criteria decision making based on ELECTRE III with weighted Borda rule

The ELECTRE (ELimination Et Choix Traduisant la REalité, in French) is an effective multiple criteria decision making method based on comparative analysis. Among the family of the ELECTRE methods and their extensions, the ELECTRE III is widely used since it can tackle uncertain and imprecise information. The hesitant fuzzy linguistic term set can represent people's perceptions more comprehensively and flexibly than exact numbers especially in cognitive complex decision-making process. In this paper, we develop an integrated method based on the ELECTRE III to handle the cognitive complex multiple experts multiple criteria decision making problems in which the cognitive complex information is represented by hesitant fuzzy linguistic term sets and the outranking relations between alternatives are calculated by a novel score-function-based distance measure between hesitant fuzzy linguistic elements. A combinative weight-determining method involving both subjective and objective opinions of experts is introduced to derive the weights of criteria. After obtaining the ranking of alternatives from each experts’ decision matrix by the distillation algorithm, the weighted Borda rule is implemented to aggregate the rankings of alternatives regarding different experts. Some ordinal consensus measures are introduced to identify the reliability of the final ranking result. An application of hospital ranking in China is provided to validate the efficiency of the proposed method.     

Evaluation of Multicriteria Decision Analysis Algorithms in Food Safety: A Case Study on Emerging Zoonoses Prioritization

Decision making in food safety is a complex process that involves several criteria of different nature like the expected reduction in the number of illnesses, the potential economic or health-related cost, or even the environmental impact of a given policy or intervention. Several multicriteria decision analysis (MCDA) algorithms are currently used, mostly individually, in food safety to rank different options in a multifactorial environment. However, the selection of the MCDA algorithm is a decision problem on its own because different methods calculate different rankings. The aim of this study was to compare the impact of different uncertainty sources on the rankings of MCDA problems in the context of food safety. For that purpose, a previously published data set on emerging zoonoses in the Netherlands was used to compare different MCDA algorithms: MMOORA, TOPSIS, VIKOR, WASPAS, and ELECTRE III. The rankings were calculated with and without considering uncertainty (using fuzzy sets), to assess the importance of this factor. The rankings obtained differed between algorithms, emphasizing that the selection of the MCDA method had a relevant impact in the rankings. Furthermore, considering uncertainty in the ranking had a high influence on the results. Both factors were more relevant than the weights associated with each criterion in this case study. A hierarchical clustering method was suggested to aggregate results obtained by the different algorithms. This complementary step seems to be a promising way to decrease extreme difference among algorithms and could provide a strong added value in the decision-making process.     

THE TIMING OF LABOR TRANSFERS IN DUAL RESOURCE-CONSTRAINED SYSTEMS: “PUSH” VS. “PULL” RULES*

Altough the dual resource-constrained (DRC) system has been studied, the decision rule used to determine when workers are eligible for transfer largely has been ignored. Some earlier studies examined the impact of this rule [5] [12] [15] but did not include labor-transfer times in their models. Gunther [6] incorporated labour-transfer times into his model, but the model involved only one worker and two machines. No previous study has examined decision rules that initiate labor transfers based on labor needs (“pull” rules). Labor transfers always have been initiated based on lack of need (“push” rules). This study examines three “pull” variations of the “When” labor-assignment decision rule. It compares their performances to the performances of two “push” rules and a comparable machine-limited system. A nonparametric statistical test, Jonckheere's S statistic, is used to test for significance of the rankings of the rules: a robust parametric multiple-comparison statistical test, Tukey's B statistic, is used to test the differences. One “pull” and one “push” decision rule provide similar performances and top the rankings consistently. Decision rules for determining when labor should be transferred from one work area to another are valuable aids for managers. This especially is true for the ever-increasing number of managers operating in organizations that recognize the benefits of a cross-trained work force. Recently there has been much interest in cross-training workers, perhaps because one of the mechanisms used in just-in-time systems to handle unbalanced work loads is to have cross-trained workers who can be shifted as demand dictates [8]. If management is to take full advantage of a cross-trained work force, it need to know when to transfer workers.     

Preference Monotonicity and Information Aggregation in Elections

If voter preferences depend on a noisy state variable, under what conditions do large elections deliver outcomes “as if” the state were common knowledge? While the existing literature models elections using the jury metaphor where a change in information regarding the state induces all voters to switch in favor of only one alternative, we allow for more general preferences where a change in information can induce a switch in favor of either alternative. We show that information is aggregated for any voting rule if, for a randomly chosen voter, the probability of switching in favor of one alternative is strictly greater than the probability of switching away from that alternative for any given change in belief over states. If the preference distribution violates this condition, there exist equilibria that produce outcomes different from the full information outcome with high probability for large classes of voting rules. In other words, unless preferences closely conform to the jury metaphor, information aggregation is not guaranteed to obtain.     

One Person,Many Votes: Divided Majority and Information Aggregation

This paper shows that information imperfections and common values can solve coordination problems in multicandidate elections. We analyze an election in which (i) the majority is divided between two alternatives and (ii) the minority backs a third alternative, which the majority views as strictly inferior. Standard analyses assume voters have a fixed preference ordering over candidates. Coordination problems cannot be overcome in such a case, and it is possible that inferior candidates win. In our setup the majority is also divided as a result of information imperfections. The majority thus faces two problems: aggregating information and coordinating to defeat the minority candidate. We show that when the common value component is strong enough, approval voting produces full information and coordination equivalence: the equilibrium is unique and solves both problems. Thus, the need for information aggregation helps resolve the majority's coordination problem under approval voting. This is not the case under standard electoral systems.     

Failure Modes and Effects Analysis: An Evaluation of Group versus Individual Performance

Failure modes and effects analysis (FMEA) is one of the most frequently used tools in process and product design: it is used in quality and reliability planning, and event and failure mode analysis. It has a long history of use and is a formally prescribed procedure by a number of prominent standards organizations. In addition, it's popular use has evolved as a less formal and widely interpreted tool in the area of Lean/Six Sigma (LSS) process improvement. This paper investigates one of the most important issues related to FMEA practice—the quality of individual vs. group performance in ranking failure modes. In particular, we compare FMEA rankings generated by: (i) individuals, (ii) group consensus, and (iii) non‐collaborative aggregation of group input (a synthesized group ranking). We find that groups outperform individuals and that synthetic groups perform as well as group consensus. We explain the implications of this result on the coordination of the design of products and processes amongst distributed organizations. The increasing distribution of product design efforts, both in terms of geography and different organizations, presents an opportunity to improve coordination using distributed synthetic group‐based FMEA.     

Common benchmarking and ranking of units with DEA

This paper develops a common framework for benchmarking and ranking units with DEA. In many DEA applications, decision making units (DMUs) experience similar circumstances, so benchmarking analyses in those situations should identify common best practices in their management plans. We propose a DEA-based approach for the benchmarking to be used when there is no need (nor wish) to allow for individual circumstances of the DMUs. This approach identifies a common best practice frontier as the facet of the DEA efficient frontier spanned by the technically efficient DMUs in a common reference group. The common reference group is selected as that which provides the closest targets. A model is developed which allows us to deal not only with the setting of targets but also with the measurement of efficiency, because we can define efficiency scores of the DMUs by using the common set of weights (CSW) it provides. Since these weights are common to all the DMUs, the resulting efficiency scores can be used to derive a ranking of units. We discuss the existence of alternative optimal solutions for the CSW and find the range of possible rankings for each DMU which would result from considering all these alternate optima. These ranking ranges allow us to gain insight into the robustness of the rankings.     

Ensemble ranking: Aggregation of rankings produced by different multi-criteria decision-making methods

One of the essential problems in multi-criteria decision-making (MCDM) is ranking a set of alternatives based on a set of criteria. In this regard, there exist several MCDM methods which rank the alternatives in different ways. As such, it would be worthwhile to try and arrive at a consensus on this important subject. In this paper, a new approach is proposed based on the half-quadratic (HQ) theory. The proposed approach determines an optimal weight for each of the MCDM ranking methods, which are used to compute the aggregated final ranking. The weight of each ranking method is obtained via a minimizer function that is inspired by the HQ theory, which automatically fulfills the basic constraints of weights in MCDM. The proposed framework also provides a consensus index and a trust level for the aggregated ranking. To illustrate the proposed approach, the evaluation and comparison of ontology alignment systems are modeled as an MCDM problem and the proposed framework is applied to the ontology alignment evaluation initiative (OAEI) 2018, for which the ranking of participating systems is of the utmost importance.     

Augmenting the Deliberative Method for Ranking Risks

The Department of Homeland Security (DHS) characterized and prioritized the physical cross‐border threats and hazards to the nation stemming from terrorism, market‐driven illicit flows of people and goods (illegal immigration, narcotics, funds, counterfeits, and weaponry), and other nonmarket concerns (movement of diseases, pests, and invasive species). These threats and hazards pose a wide diversity of consequences with very different combinations of magnitudes and likelihoods, making it very challenging to prioritize them. This article presents the approach that was used at DHS to arrive at a consensus regarding the threats and hazards that stand out from the rest based on the overall risk they pose. Due to time constraints for the decision analysis, it was not feasible to apply multiattribute methodologies like multiattribute utility theory or the analytic hierarchy process. Using a holistic approach was considered, such as the deliberative method for ranking risks first published in this journal. However, an ordinal ranking alone does not indicate relative or absolute magnitude differences among the risks. Therefore, the use of the deliberative method for ranking risks is not sufficient for deciding whether there is a material difference between the top‐ranked and bottom‐ranked risks, let alone deciding what the stand‐out risks are. To address this limitation of ordinal rankings, the deliberative method for ranking risks was augmented by adding an additional step to transform the ordinal ranking into a ratio scale ranking. This additional step enabled the selection of stand‐out risks to help prioritize further analysis.     

Using Surveys in Public Participation Processes for Risk Decision Making: The Case of the 2003 British GM Nation? Public Debate

This article takes as its case study the “GM Nation?” public debate, a major participation process on the commercialization of agricultural biotechnology, which occurred in Britain during the summer of 2003. We investigate possible self‐selection biases in over 36,000 open questionnaire responses on the risks and benefits of genetically modified crops and food obtained during GM Nation? A comparison sample of equivalent responses from a statistically representative sample (n= 1,363) of the British general public obtained shortly after the conclusion of the debate is reported. This comparison shows that the GM Nation? open responses were indeed not fully representative of British “public opinion” regarding agricultural biotechnology. Rather, such opinion is not a unitary whole, but fragmented, with considerable ambivalence coexisting alongside outright opposition to GM agriculture. The methodological implications for multistage participation processes are discussed: in particular, the need to anticipate outcomes of complex design decisions, and to include representative public surveys as standard where measures of broader public attitudes to risk are an important objective.