对统计学领域数据挖掘研究的反思

数据挖掘是一门新兴交叉学科,近年来受到各界的广泛关注。统计学者从统计学视角开展了数据挖掘理论及应用研究,并取得了一定的研究成果。文章对相关著作和论文进行了分析研究,并在此基础上展望了数据挖掘的研究趋势。     

国外数据挖掘应用研究与发展分析

目前“数据挖掘(Data Mining,DM)”这一术语在学术界还没有一个公认的、权威的定义,但我们一般可以简单的认为数据挖掘是从海量数据中发现趋势或模式的过程。尽管有些人不愿承认数据挖掘与统计学的内在联系,但不可否认的是早期的数据挖掘的确脱胎于统计学,因此也可以说数据挖掘是利用统计学和机器学习技术创建预测行为的模型。需要强调的是数据挖掘的过程是一个“发现”的过程,而不是“发明”的过程。换句话说,数据挖掘所探寻的模式,是一种已有的、只是隐藏在数据中、暂时没有被发现的知识。世界上对数据挖掘的正式研究始于1989年8月举行…     

统计学与数据挖掘的比较分析

数据挖掘与统计学有着千丝万缕的联系,本文希望在等同的角度,从数据挖掘并非统计学分支的观点出发来阐释两门学科,既着眼于二者相重叠的地方,更重视二者的区别,并考虑一些其他与数据挖掘相关的问题。     

统计分析的新模式——数据挖掘技术

近年来,数据挖掘已经成为了统计学研究的一个新兴领域,引起了学科内部的极大关注,数据挖掘技术的发展极大地丰富了统计分析的手段。由于当今社会信息量、数据量急剧增大,因此人们迫切需要将这些数据转换成有用的信息和知识。面对     

基于统计数据的OLAP数据挖掘技术

一、概述数据挖掘是 90年代中后期兴起的一门跨学科的综合研究领域 ,它集计算机机器学习、统计学、数据库管理、数据仓库、可视化、并行计算、决策支持为一体 ,利用数据库、数据仓库技术存储和管理数据 ,利用机器学习和统计学方法分析数据 ,旨在发现大量复杂数据中蕴含的有价值的知识和信息。目前 ,随着数据挖掘应用的不断开展以及客观现实对数据分析需求的不断增长 ,人们越来越认识到数据挖掘的重要性和必要性。数据挖掘通过对数据的总结、分类、聚类、关联等分析 ,实现对数据内在结构特征的理解和对未知数据的预测。其中 ,数据总结是在数…     

数据挖掘的技术与商业定义及其研究对象

本文在对数据挖掘考察的基础上,从技术角度和商业角度对数据挖掘的概念予以界定,并探讨了数据挖掘在知识发现中的地位,明确了数据挖掘的对象和对该领域的研究方向.     

统计学与数据挖掘

文章以统计学最近 40年的发展走势作为论述的起点 ,逐步对统计方法在数据挖掘算法设计、开发过程中的应用情况 ,进行客观、系统的介绍和分析 ,进而提出统计学和数据挖掘协同发展的广阔前景。     

基于函数型数据分析的数据挖掘功能研究

数据挖掘功能是数据挖掘研究与应用的一个重要方面。数据挖掘功能用于指定数据挖掘任务中要找的模式类型。当前,数据挖掘的功能所处理的主要是传统的数据,对于函数型数据的研究还不是很多。文章探讨了数据挖掘中可以挖掘的几种函数型数据模式,包括数据描述、分类、聚类和回归。     

数据挖掘中抽样技术的应用

本文介绍了数据挖掘中抽样技术运用的历史与现状,总结了数据挖掘领域对抽样技术研究和使用中存在的问题,并指出了数据挖掘中抽样技术未来研究方向和发展前景。     

异常数据挖掘及在经济欺诈发现中的应用

数据挖掘是一个多学科的交叉领域,包括数据库系统、统计学、机器学习、可视化和信息科学,统计学中的聚类分析、相关分析、回归分析、主成分分析及神经网络在数据挖掘中都有广泛应用.     

数据挖掘及其对统计学的挑战

一、引言随着科学技术的发展 ,利用数据库技术来存储管理数据 ,利用机器学习的方法来分析数据 ,从而挖掘出大量的隐藏在数据背后的知识 ,这种思想的结合形成了现在深受人们关注的非常热门的研究领域 :数据库中的知识发现———ＫＤＤ(ＫｎｏｗｌｅｄｇｅＤｉｓｃｏｖ ｅｒｙｉｎＤａｔａｂａｓｅｓ) ,其中 ,数据挖掘技术便是ＫＤＤ中的一个最为关键的环节。1995年 ,在加拿大的蒙特利尔召开了第一届“知识发现和数据挖掘”国际学术会议 ,数据挖掘一词被很快流传开来。数据挖掘———ＤＭ(ＤａｔａＭｉｎｉｎｇ)就是从大量的、不完全的、有噪…     

大数据与数据工程学

大数据的冲击并非是对以样本数据为对象的统计学的颠覆,而是对现代统计学的扩展。本文结合大数据的相关特征,以数据经济价值的扩展为切入点,从数据价值挖掘的角度论证了数据挖掘与大数据分析的关系,探讨了大数据背景下数据衍生品的创造与数据工程学创建的必要性。在此基础上,参照“金融工程学”的概念及学科体系,对“数据工程学”的概念进行了界定,并对数据工程学学科体系构建的相关理论基础、主要研究内容与分析技术进行了归纳与说明。     

Data Mining and Knowledge Discovery in Databases – An Overview

Data mining seeks to extract useful, but previously unknown, information from typically massive collections of non-experimental, sometimes non-traditional data. From the perspective of statisticians, this paper surveys techniques used and contributions from fields such as data warehousing, machine learning from artificial intelligence, and visualization as well as statistics. It concludes that statistical thinking and design of analysis, as exemplified by achievements in clinical epidemiology, may fit well with the emerging activities of data mining and 'knowledge discovery in databases' (DM&KDD).     

Multivariate online regression analysis with heterogeneous streaming data

New data collection and storage technologies have given rise to a new field of streaming data analytics, called real-time statistical methodology for online data analyses. Most existing online learning methods are based on homogeneity assumptions, which require the samples in a sequence to be independent and identically distributed. However, inter-data batch correlation and dynamically evolving batch-specific effects are among the key defining features of real-world streaming data such as electronic health records and mobile health data. This article is built under a state-space mixed model framework in which the observed data stream is driven by a latent state process that follows a Markov process. In this setting, online maximum likelihood estimation is made challenging by high-dimensional integrals and complex covariance structures. In this article, we develop a real-time Kalman-filter-based regression analysis method that updates both point estimates and their standard errors for fixed population average effects while adjusting for dynamic hidden effects. Both theoretical justification and numerical experiments demonstrate that our proposed online method has statistical properties similar to those of its offline counterpart and enjoys great computational efficiency. We also apply this method to analyze an electronic health record dataset.     

《统计研究》的历史阶段性回顾与特征分析

本文基于各个时代背景下我国统计学科的重要事件,联系过去十四届中青年统计科学研讨会的会议主题,利用文本数据挖掘技术,对1984年至2014年第5期《统计研究》刊登的5192篇文章的篇名、关键词和摘要等做深入分析。探讨了我国30年来统计学科的发展历程,剖析了《统计研究》期刊的整体特征,进一步明确了《统计研究》杂志对中国统计学科建设发展的重要贡献。     

Bayesian nonparametric clustering for large data sets

We propose two nonparametric Bayesian methods to cluster big data and apply them to cluster genes by patterns of gene–gene interaction. Both approaches define model-based clustering with nonparametric Bayesian priors and include an implementation that remains feasible for big data. The first method is based on a predictive recursion which requires a single cycle (or few cycles) of simple deterministic calculations for each observation under study. The second scheme is an exact method that divides the data into smaller subsamples and involves local partitions that can be determined in parallel. In a second step, the method requires only the sufficient statistics of each of these local clusters to derive global clusters. Under simulated and benchmark data sets the proposed methods compare favorably with other clustering algorithms, including k-means, DP-means, DBSCAN, SUGS, streaming variational Bayes and an EM algorithm. We apply the proposed approaches to cluster a large data set of gene–gene interactions extracted from the online search tool “Zodiac.”

     

Multiscale Adaptive Regression Models for Neuroimaging Data

Neuroimaging studies aim to analyze imaging data with complex spatial patterns in a large number of locations (called voxels) on a two-dimensional (2D) surface or in a 3D volume. Conventional analyses of imaging data include two sequential steps: spatially smoothing imaging data and then independently fitting a statistical model at each voxel. However, conventional analyses suffer from the same amount of smoothing throughout the whole image, the arbitrary choice of smoothing extent, and low statistical power in detecting spatial patterns. We propose a multiscale adaptive regression model (MARM) to integrate the propagation-separation (PS) approach (Polzehl and Spokoiny, 2000, 2006) with statistical modeling at each voxel for spatial and adaptive analysis of neuroimaging data from multiple subjects. MARM has three features: being spatial, being hierarchical, and being adaptive. We use a multiscale adaptive estimation and testing procedure (MAET) to utilize imaging observations from the neighboring voxels of the current voxel to adaptively calculate parameter estimates and test statistics. Theoretically, we establish consistency and asymptotic normality of the adaptive parameter estimates and the asymptotic distribution of the adaptive test statistics. Our simulation studies and real data analysis confirm that MARM significantly outperforms conventional analyses of imaging data.     

Prediction of mechanical property of steel strips using multivariate adaptive regression splines

In recent times, the problem of prediction of properties of a steel strip has attracted enormous attention from different communities such as statistics, data mining, soft computing, and engineering. This is due to the prospective benefits of reduction in testing and inventory cost, increase in yield, and improvement in delivery compliance. The complexity of the problem arises due to its dependency on the chemical composition of the steel, and a number of processing parameters. To predict the mechanical properties of the strip (yield strength, ultimate tensile strength, and Elongation), a model based on multivariate adaptive regression spline has been developed. It is found that the prediction agrees well with the actual measured data.