arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Computer Science · Information Systems

On Learning Strategies for Topic Specific Web Crawling. Next Generation Data Mining Applications

Charu C. Aggarwal

See on the map

Abstract

Crawling has been a topic of considerable interest in recent years because of the rapid growth of the world wide web. In many cases, it is possible to design more effective crawlers which can find web pages belonging to specific topics. In this paper, we will discuss some recent techniques for crawling web pages belonging to specific topics. We discuss the following classes of techniques: (1) Intelligent Crawling Methods: These methods learn the relationship between the hyper-link structure/web page content and the topic of the web page. This learned information is utilized in order to guide the direction of the crawl. (2) Collaborative Crawling Methods: These methods utilize the pattern of world wide web accesses by individual users in order to build the learning information. In many cases, user access patterns contain valuable statistical patterns which cannot be inferred from purely linkage information. We will also discusses some creative ways of combining different kinds of linkage- and user-centered methods in order to improve the effectiveness of the crawl. We will discuss some of the recent algorithms proposed in each topic along with some discussions on the directions of future research. 1

More like this