RANKINGS OF ACADEMIC JOURNALS AND INSTITUTIONS IN ECONOMICS

We conducted a worldwide ranking of academic institutions that produce research in a list of thirty top research journals in economics. We also computed journal rankings for the same period and hence we do not rely on weights that were computed for research carried out in earlier periods. The United States is clearly the dominant force in the top‐fifty group, but European academic institutions are well represented in the group of the top 200 universities worldwide as are universities from Asia and the Far East in particular. (JEL: A14, A10)     

Knowledge dissemination in operations management: Published perceptions versus academic reality

The channels for knowledge generation and dissemination in the business disciplines are many: presenting research at conferences, writing books, distributing working papers, offering insights in society newsletters, giving invited talks, publishing studies in academic journals, and many other venues, including even blogs and perhaps Facebook^®. But the most important venue is probably published research in “top-level” academic journals. In the discipline of Operations Management, many studies and lists have been published that attempt to determine which of these journals are supposedly the “top” according to either citation analyses, the opinion of recognized experts, author affiliations, bibliometric studies, and other approaches. These lists may then, in turn, be used in different degrees to evaluate research. However, what really counts is what the academic institutions actually use for guidance in evaluating faculty research. Based on a new source of ranking data from AACSB-accredited schools, we compare published journal-ranking studies against that of academe to determine the degree to which the studies reflect academic “reality”. We present rankings of OM journals based on this new source of data and on an aggregate of the stream of published studies, and evaluate their consistency.     

A data envelopment analysis approach for ranking journals

We create a data envelopment analysis (DEA) model to rank business journals, using data from the Thomson Reuters Journal Citation Reports^® (JCR). As opposed to previous models that ranked journals in only one field and mostly relied on survey data, this model is used to rank 358 business journals from five different JCR categories according to such citation-based factors as the number of articles, the number of citations, impact factor, five-year impact factor, immediacy index, eigenfactor score, and article influence score. We compute relative efficiencies of the journals and thereby create plausible journal rankings that largely, but not completely, corroborate three widely used business publication journal ranking lists. In addition, we show how the different characteristics of the JCR data impact the DEA ranking model. Finally, we identify journals that are not on the business publication lists but consistently perform very well relative to those benchmark journals, and should possibly be included in the business publication ranking lists. We also identify journals whose inclusion in widely used business publication rankings cannot be justified by our methodology.     

COMPARISON OF CONSENSUS METHODS FOR PRIORITY RANKING PROBLEMS

Various consensus methods proposed for ranking problems yield controversial rankings and/or tied rankings which are vulnerable to considerable dispute. These include Borda-Kendall (BK) and minimum-variance (MV) methods. This paper compares three continuous (ratio-scale) consensus scoring methods with BK and MV ranking methods. One method, termed GM, is an eigenvector scaling of the geometric-mean consensus matrix. GM allows for (1) paired-comparison voting inputs (as opposed to all-at-once ranking), (2) pick-the-winner preference voting, and (3) ratio-scale preference voting. GM is relatively simple to calculate on small computers or calculators, and merging of “close” candidates into tied rankings can be achieved by using an e-threshold tie rule discussed in this paper. The GM method thus can be used for paired-comparison voting to calculate both a ratio-scaled consensus index (based on a consensus eigenvector) and a ranking of candidates that allows for ties between “close” candidates. Eigenvalue analysis is used as a means of evaluating voter inconsistencies.     

Augmenting the Deliberative Method for Ranking Risks

The Department of Homeland Security (DHS) characterized and prioritized the physical cross‐border threats and hazards to the nation stemming from terrorism, market‐driven illicit flows of people and goods (illegal immigration, narcotics, funds, counterfeits, and weaponry), and other nonmarket concerns (movement of diseases, pests, and invasive species). These threats and hazards pose a wide diversity of consequences with very different combinations of magnitudes and likelihoods, making it very challenging to prioritize them. This article presents the approach that was used at DHS to arrive at a consensus regarding the threats and hazards that stand out from the rest based on the overall risk they pose. Due to time constraints for the decision analysis, it was not feasible to apply multiattribute methodologies like multiattribute utility theory or the analytic hierarchy process. Using a holistic approach was considered, such as the deliberative method for ranking risks first published in this journal. However, an ordinal ranking alone does not indicate relative or absolute magnitude differences among the risks. Therefore, the use of the deliberative method for ranking risks is not sufficient for deciding whether there is a material difference between the top‐ranked and bottom‐ranked risks, let alone deciding what the stand‐out risks are. To address this limitation of ordinal rankings, the deliberative method for ranking risks was augmented by adding an additional step to transform the ordinal ranking into a ratio scale ranking. This additional step enabled the selection of stand‐out risks to help prioritize further analysis.     

Stochastic Judgments in the AHP: The Measurement of Rank Reversal Probabilities

This paper presents a methodology for analyzing Analytic Hierarchy Process (AHP) rankings if the pairwise preference judgments are uncertain (stochastic). If the relative preference statements are represented by judgment intervals, rather than single values, then the rankings resulting from a traditional (deterministic) AHP analysis based on single judgment values may be reversed, and therefore incorrect. In the presence of stochastic judgments, the traditional AHP rankings may be stable or unstable, depending on the nature of the uncertainty. We develop multivariate statistical techniques to obtain both point estimates and confidence intervals of the rank reversal probabilities, and show how simulation experiments can be used as an effective and accurate tool for analyzing the stability of the preference rankings under uncertainty. If the rank reversal probability is low, then the rankings are stable and the decision maker can be confident that the AHP ranking is correct. However, if the likelihood of rank reversal is high, then the decision maker should interpret the AHP rankings cautiously, as there is a subtantial probability that these rankings are incorrect. High rank reversal probabilities indicate a need for exploring alternative problem formulations and methods of analysis. The information about the extent to which the ranking of the alternatives is sensitive to the stochastic nature of the pairwise judgments should be valuable information into the decision-making process, much like variability and confidence intervals are crucial tools for statistical inference. We provide simulation experiments and numerical examples to evaluate our method. Our analysis of rank reversal due to stochastic judgments is not related to previous research on rank reversal that focuses on mathematical properties inherent to the AHP methodology, for instance, the occurrence of rank reversal if a new alternative is added or an existing one is deleted.     

A Deliberative Method for Ranking Risks (II): Evaluation of Validity and Agreement among Risk Managers

A deliberative method for ranking risks was evaluated in a study involving 218 risk managers. Both holistic and multiattribute procedures were used to assess individual and group rankings of health and safety risks facing students at a fictitious middle school. Consistency between the rankings that emerged from these two procedures was reasonably high for individuals and for groups, suggesting that these procedures capture an underlying construct of riskiness. Participants reported high levels of satisfaction with their groups' decision-making processes and the resulting rankings, and these reports were corroborated by regression analyses. Risk rankings were similar across individuals and groups, even though individuals and groups did not always agree on the relative importance of risk attributes. Lower consistency between the risk rankings from the holistic and multiattribute procedures and lower agreement among individuals and groups regarding these rankings were observed for a set of high-variance risks. Nonetheless, the generally high levels of consistency, satisfaction, and agreement suggest that this deliberative method is capable of producing risk rankings that can serve as informative inputs to public risk-management decision making.     

Evaluation of Multicriteria Decision Analysis Algorithms in Food Safety: A Case Study on Emerging Zoonoses Prioritization

Decision making in food safety is a complex process that involves several criteria of different nature like the expected reduction in the number of illnesses, the potential economic or health-related cost, or even the environmental impact of a given policy or intervention. Several multicriteria decision analysis (MCDA) algorithms are currently used, mostly individually, in food safety to rank different options in a multifactorial environment. However, the selection of the MCDA algorithm is a decision problem on its own because different methods calculate different rankings. The aim of this study was to compare the impact of different uncertainty sources on the rankings of MCDA problems in the context of food safety. For that purpose, a previously published data set on emerging zoonoses in the Netherlands was used to compare different MCDA algorithms: MMOORA, TOPSIS, VIKOR, WASPAS, and ELECTRE III. The rankings were calculated with and without considering uncertainty (using fuzzy sets), to assess the importance of this factor. The rankings obtained differed between algorithms, emphasizing that the selection of the MCDA method had a relevant impact in the rankings. Furthermore, considering uncertainty in the ranking had a high influence on the results. Both factors were more relevant than the weights associated with each criterion in this case study. A hierarchical clustering method was suggested to aggregate results obtained by the different algorithms. This complementary step seems to be a promising way to decrease extreme difference among algorithms and could provide a strong added value in the decision-making process.     

REVEALED PERFORMANCES: WORLDWIDE RANKINGS OF ECONOMISTS AND ECONOMICS DEPARTMENTS, 1990–2000

In this paper, I study the production of academic research by economics departments and economists. Worldwide rankings are provided based on both citations and publications. These rankings reveal a dominant position of the United States in the production of economics literature. Over time, however, the extent of this dominance is decreasing. (JEL: A10, A14)     

Failure Modes and Effects Analysis: An Evaluation of Group versus Individual Performance

Failure modes and effects analysis (FMEA) is one of the most frequently used tools in process and product design: it is used in quality and reliability planning, and event and failure mode analysis. It has a long history of use and is a formally prescribed procedure by a number of prominent standards organizations. In addition, it's popular use has evolved as a less formal and widely interpreted tool in the area of Lean/Six Sigma (LSS) process improvement. This paper investigates one of the most important issues related to FMEA practice—the quality of individual vs. group performance in ranking failure modes. In particular, we compare FMEA rankings generated by: (i) individuals, (ii) group consensus, and (iii) non‐collaborative aggregation of group input (a synthesized group ranking). We find that groups outperform individuals and that synthetic groups perform as well as group consensus. We explain the implications of this result on the coordination of the design of products and processes amongst distributed organizations. The increasing distribution of product design efforts, both in terms of geography and different organizations, presents an opportunity to improve coordination using distributed synthetic group‐based FMEA.     

A Deliberative Method for Ranking Risks (I): Overview and Test Bed Development

Risk ranking offers a potentially powerful means for gathering public input to help set risk-management priorities. In most rankings conducted to date, the categories and attributes used to describe the risks have varied widely, the materials and procedures have not been designed to facilitate comparisons among risks on all important attributes, and the validity and reproducibility of the resulting rankings have not been assessed. To address these needs, a risk-ranking method was developed in which risk experts define and categorize the risks to be ranked, identify the relevant risk attributes, and characterize the risks in a set of standardized risk summary sheets, which are then used by lay or other groups in structured ranking exercises. To evaluate this method, a test bed involving 22 health and safety risks in a fictitious middle school was created. This article provides an overview of the risk-ranking method and describes the challenges faced in designing the middle school test bed. A companion article in this issue reports on the validity of the ranking procedures and the level of agreement among risk managers regarding ranking of risks and attributes.     

What Influences Business Academics’ Use of the Association of Business Schools (ABS) List? Evidence From a Survey of UK Academics

The development of the Association of Business Schools (ABS) list in 2007 and its rapid adoption by UK business schools has had a profound effect on the nature of business and management academics’ ways of working. Using a large‐scale survey of UK business academics, we assess the extent to which individuals use the Academic Journal Guide (AJG/ABS) list in their day‐to‐day professional activities. In particular, we explore how their perceptions of the list, the academic influence of their research, academic rank and organizational context drive the varied use. Building on prior research on the importance of univalent attitudes in predicting behaviour, we find those who have either strong positive or negative views of the list are more extensive users than those who are ambivalent. We also find that the extent of use of the AJG/ABS list is greatest among those academics who have lower academic influence, in the middle or junior ranks within business schools and in middle and low‐status universities. We explore the implications of these findings for the value of journal rankings and for the management of business schools.     

Important Strategic and Tactical Manufacturing Issues in the 1990s*

This paper reports the identification and ranking of major manufacturing issues that American manufacturing managers and academics must focus on and resolve in the 1990s to be competitive on a global basis. Vice-presidents of manufacturing from over 75 companies across the United States were asked in a Delphi study to rank the key strategic and tactical issues facing American manufacturing in the next three to five years. Based on three rounds of a Delphi study, quality management, manufacturing strategy, and process technology emerged as the top ranked strategic issues. The top ranked tactical issues were quality control, manufacturing planning and control systems, and work force supervision. These issues were valid across diverse industry groups, since the industrial and educational background of the respondents was shown to have no impact on the final consensus rankings and opinions reported in this paper. Factor analysis of the responses by the panel revealed that certain issues tend to be consistently viewed together, and were interpreted accordingly to provide insights into the ranking of issues. Apart from building a consensus among experts on key manufacturing issues of present and future importance, this study can also be helpful for setting future academic research and pedagogical priorities for the field of production and operations management.     

A robust ranking method extending ELECTRE III to hierarchy of interacting criteria,imprecise weights and stochastic analysis

A great majority of methods designed for Multiple Criteria Decision Aiding (MCDA) assume that all assessment criteria are considered at the same level, however, decision problems encountered in practice often impose a hierarchical structure of criteria. The hierarchy helps to decompose complex decision problems into smaller and manageable subtasks, and thus, it is very attractive for computational efficiency and explanatory purposes. To handle the hierarchy of criteria in MCDA, a methodology called Multiple Criteria Hierarchy Process (MCHP), has been recently proposed. MCHP permits to consider preference relations with respect to a subset of criteria at any level of the hierarchy. Here, we propose to apply MCHP to the ELECTRE III ranking method adapted to handle three types of interaction effects between criteria: mutual-weakening, mutual-strengthening and antagonistic effect. We also involve in MCHP an imprecise elicitation of criteria weights, generalizing a technique called the SRF method. In order to explore the plurality of rankings obtained by the ELECTRE III method for possible sets of criteria weights, we apply the Stochastic Multiobjective Acceptability Analysis (SMAA) that permits to draw robust conclusions in terms of rankings and preference relations at each level of the hierarchy of criteria. The novelty of the whole methodology consists of a joint consideration of hierarchical assessments of alternatives performances on interacting criteria, imprecise criteria weights, and robust analysis of ranking recommendations resulting from ELECTRE III. An example regarding the multiple criteria ranking of some European universities will show how to apply the proposed methodology on a decision problem.     

Ranking irregularities when evaluating alternatives by using some ELECTRE methods

The ELECTRE II and III methods enjoy a wide acceptance in solving multi-criteria decision-making (MCDM) problems. Research results in this paper reveal that there are some compelling reasons to doubt the correctness of the proposed rankings when the ELECTRE II and III methods are used. In a typical test we first used these methods to determine the best alternative for a given MCDM problem. Next, we randomly replaced a non-optimal alternative by a worse one and repeated the calculations without changing any of the other data. Our computational tests revealed that sometimes the ELECTRE II and III methods might change the indication of the best alternative. We treat such phenomena as rank reversals. Although such ranking irregularities are well known for the additive variants of the AHP method, it is the very first time that they are reported to occur when the ELECTRE methods are used. These two methods are also evaluated in terms of two other ranking tests and they failed them as well. Two real-life cases are described to demonstrate the occurrence of rank reversals with the ELECTRE II and III methods. Based on the three test criteria presented in this paper, some computational experiments on randomly generated decision problems were executed to test the performance of the ELECTRE II and III methods and an examination of some real-life case studies are also discussed. The results of these examinations show that the rates of the three types of ranking irregularities were rather significant in both the simulated decision problems and the real-life cases studied in this paper.     

Extreme ranking analysis in robust ordinal regression

We extend the principle of robust ordinal regression with an analysis of extreme ranking results. In our proposal, we consider the whole set of instances of a preference model that is compatible with preference information provided by the DM. We refer to both, the well-known UTA^GMS method, which builds the set of general additive value functions compatible with DM's preferences, and newly introduced in this paper PROMETHEE^GKS, which constructs the set of compatible outranking models via robust ordinal regression. Then, we consider all complete rankings that follow the use of the compatible preference models, and we determine the best and the worst attained ranks for each alternative. In this way, we are able to assess its position in an overall ranking, and not only in terms of pairwise comparisons, as it is the case in original robust ordinal regression methods. Additionally, we analyze the ranges of possible comprehensive scores (values or net outranking flows). We also discuss extensions of the presented approach on other multiple criteria problems than ranking. Finally, we show how the presented methodology can be applied in practical decision support, reporting results of three illustrative studies.     

Bridging Practice and Theory: A Design Science Approach

Despite ambitious efforts in various fields of research over multiple decades, the goal of making academic research relevant to the practitioner remains elusive: theoretical and academic research interests do not seem to coincide with the interests of managerial practice. This challenge is more fundamental than knowledge transfer, it is one of diverging knowledge interests and means of knowledge production. In this article, we look at this fundamental challenge through the lens of design science, which is an approach aimed primarily at discovery and problem solving as opposed to accumulation of theoretical knowledge. We explore in particular the ways in which problem‐solving research and theory‐oriented academic research can complement one another. In operations management (OM) research, recognizing and building on this complementarity is especially crucial, because problem‐solving–oriented research produces the very artifacts (e.g., technologies) that empirical OM research subsequently evaluates in an attempt to build explanatory theory. It is indeed the practitioner—not the academic scientist—who engages in basic research in OM. This idiosyncrasy prompts the question: how can we enhance the cross‐fertilization between academic research and research practice to make novel theoretical insights and practical relevance complementary? This article proposes a design science approach to bridge practice to theory rather than theory to practice.     

Probabilistic Inversion in Priority Setting of Emerging Zoonoses

This article presents methodology of applying probabilistic inversion in combination with expert judgment in priority setting problem. Experts rank scenarios according to severity. A linear multi‐criteria analysis model underlying the expert preferences is posited. Using probabilistic inversion, a distribution over attribute weights is found that optimally reproduces the expert rankings. This model is validated in three ways. First, consistency of expert rankings is checked, second, a complete model fitted using all expert data is found to adequately reproduce observed expert rankings, and third, the model is fitted to subsets of the expert data and used to predict rankings in out‐of‐sample expert data.     

Inductive Inference: An Axiomatic Approach

A predictor is asked to rank eventualities according to their plausibility, based on past cases. We assume that she can form a ranking given any memory that consists of finitely many past cases. Mild consistency requirements on these rankings imply that they have a numerical representation via a matrix assigning numbers to eventuality–case pairs, as follows. Given a memory, each eventuality is ranked according to the sum of the numbers in its row, over cases in memory. The number attached to an eventuality–case pair can be interpreted as the degree of support that the past case lends to the plausibility of the eventuality. Special instances of this result may be viewed as axiomatizing kernel methods for estimation of densities and for classification problems. Interpreting the same result for rankings of theories or hypotheses, rather than of specific eventualities, it is shown that one may ascribe to the predictor subjective conditional probabilities of cases given theories, such that her rankings of theories agree with rankings by the likelihood functions.