基于Hadoop的网络爬虫技术研究

网络爬虫一般从一个起始网页开始,读取网页的内容和网页中的链接,依次循环下去,直到找到此网页所有的链接网页为止;当要爬取的数据量比较大时,传统的技术存在一定弊端,而Hadoop开源云计算框架在数据采集方面会有一定的优势。在介绍Hadoop云计算框架的基础上,本文阐述网络爬虫的原理,并实现基于Hadoop的网络爬虫。

Research on Web Crawler Technology Based on Hadoop

The Web crawler usually starts from a starting Webpage, reads the content of webpage and Webpage links, successively circles until it finds all the webpage links; when you want to climb from the large amount of data, the traditional technology has some disadvantages, and the Hadoop open source cloud computing framework will have a certain advantages in data acquisition. On the basis of the introduction of Hadoop cloud computing framework, this paper describes the principle of the web crawler and realization of the web crawler based on Hadoop.