The behavior policies define which pages the crawler will bring down to the indexer, how often to go back to a Web site to check it again, and something called a politeness policy.
这种行为策略定义了爬虫会将哪些页面带入索引程序、以什么样的频率回到Web站点上再次对它进行检查，以及一种礼貌原则。
This information helps the Web crawler determine what the set of pages is and when to crawl them.
这一信息能帮助web爬虫程序决定要爬行哪些页面以及爬行的时间。
Thus the efficiency of site crawls provides relief to both sides: the hosting Web server and the crawler as well, by keeping the number of GET requests for pages at a minimum.
这样，站点爬行的效率为两边都减轻了负担：宿主Web服务器和爬虫程序，方法是把GET页面请求数量保持在最小值。
Web-Crawler is a important part of search engine, it is responsible for the network information gathering.
网络爬虫是搜索引擎的重要组成部分，它在搜索引擎中负责网络信息的采集。
Including full text search and Web crawler.
包括全文搜索和Web爬虫。
Web crawler is a system which can automatically get web pages from Internet. It helps searching engine download web pages, so it is an important part of searching engine.
网络爬虫是一个可以从因特网上自动提取网页的系统，它为搜索引擎从万维网上下载网页，是搜索引擎的重要组成。
Each search engine has its own automated program called a "web spider" or "web crawler" that crawls the web.
每个搜索引擎都有自己爬行网页的自动化程序，叫做“网络蜘蛛（web spider）”或“网络爬虫（web crawler）”。
Using it, you can setup a multi-threaded web crawler in few minutes.
提供了简单易用的接口，可以在几分钟内创建一个多线程网络爬虫。
Traditional focused crawler is targeting web pages that are relevant to some specific topics. But some applications, such as web directory, are providing users with relevant websites.
传统的聚焦爬虫抓取的目标是与某一特定主题内容相关的网页，而在有些应用中，如网络目录，更多的是给用户提供主题相关网站。
Then a topic crawler system was designed and implemented, employing topic sensitive Hyperlink-Induced Topic Search (HITS) to predict the priority of fetched Web pages.
在此基础上设计并实现了一个主题爬虫系统，该系统利用主题敏感HITS来计算网页优先级。
Focused crawler is a subject-oriented information retrieval system. It can meet the users' need and retrieve information that is relevant to some specific subjects from the web automatically.
聚焦爬虫是一种面向主题的信息搜集系统，可以根据用户需要从互联网上自动搜集到主题相关信息，在主题搜索引擎、站点结构分析等方面取得越来越广泛的应用。
First, the search depth is not ideal. Search engines obtain information on Internet via web crawler, so they cannot get the Shared information stored in users' personal computer.
第一是搜索深度不够，当前搜索引擎通过网络蜘蛛获取互联网上的资源，无法检索用户个人电脑上的共享资源。
The main goals of focused web crawler are to get more web pages which are correlative with a certain topic and prepare data for users querying.
聚焦网络爬虫并不追求大的覆盖，而将目标定为抓取与某一特定主题内容相关的网页，为面向主题的用户查询准备数据资源。
However the current two ways of web crawling used by focus crawler are low efficiency.
然而目前的主题爬虫所采用的两种基本抓取网页的方式效率比较低下。
To expand the Web crawler, consider collecting image references or searching for specific text strings.
要扩展该Web crawler，可以考虑收集图像引用或搜索特定的文本字符串。
Provides an entry point for the search engine crawler to easily follow the links within your Web pages.
为搜索引擎爬网程序提供入口点，以使爬网程序轻松地跟踪您的Web页面内的链接。
The CollectUrls Web crawler program takes advantage of a fixed-size thread pool.
CollectUrlsWeb crawler程序利用一个固定大小的线程池。
The high-level task for this article is a Web crawler: given a base URL for a Web site, you'll collect elements from the site that you can use for some purpose.
本文的高级任务是创建一个Web crawler：给定一个网站的基URL，从该网站收集可以用作某种用途的元素。
Note that the web site crawler cannot crawl secured pages.
请注意，该网站的抓取工具无法抓取安全网页。
Note that the web site crawler cannot crawl secured pages.
请注意，该网站的抓取工具无法抓取安全网页。
应用推荐