Use a site map to lead the crawler around your site.
使用站点地图引导爬行器遍历您的站点。
Define the crawler subdirectories, as illustrated in Figure 8.
定义爬虫子目录，如图8所示。
Notice also that only the base URI string is added to the crawler.
另外请注意，只有基URI字符串被添加到了 crawler。
The CollectUrls Web crawler program takes advantage of a fixed-size thread pool.
CollectUrlsWebcrawler程序利用一个固定大小的线程池。
The next step is to make a sample announcement document available to the crawler.
下一步是为爬行器提供一个样例通知文档。
This information helps the Web crawler determine what the set of pages is and when to crawl them.
这一信息能帮助web爬虫程序决定要爬行哪些页面以及爬行的时间。
Select UNIX file system as the crawler type, as shown in Figure 6, and then click on the Next button.
选择UNIXfilesystem作为爬虫类型，如图6 所示，然后单击Next 按钮。
What you need to do, in short, is to generate a list of page references (URLs) for a crawler to fetch.
简而言之，我们需要做的就是生成一个页面引用列表（URL），爬虫程序通过这个列表获取信息。
To expand the Web crawler, consider collecting image references or searching for specific text strings.
要扩展该Webcrawler，可以考虑收集图像引用或搜索特定的文本字符串。
It's one of the most important signals your page offers to a crawler, so why not include a local signal?
这是你的页面向搜索引擎爬虫提供的最为重要的信息之一，所以为什么不提供一些本地化的信息呢？
Instead of converting your entire site to static URLs, pick the pages you want to index by a crawler.
代替将整个站点转换为静态url，您可以挑选一些希望爬行器建立索引的页面。
Provides an entry point for the search engine crawler to easily follow the links within your Web pages.
为搜索引擎爬网程序提供入口点，以使爬网程序轻松地跟踪您的Web页面内的链接。
First, let's look at how crawler-based search engines work (both Google and Yahoo fall in this category).
首先，让我们看看基于爬虫(crawler - based)的搜索引擎是如何工作的(Google和Yahoo都是这种类型)。
Each search engine has its own automated program called a "web spider" or "web crawler" that crawls the web.
每个搜索引擎都有自己爬行网页的自动化程序，叫做“网络蜘蛛（web spider）”或“网络爬虫（web crawler）”。
The Sitemaps 0.90 protocol offers the option of a Sitemaps index file to be provided to the crawler as well.
Sitemaps 0.90提供了一个选项，可以把Sitemapsindex文件提供给爬虫程序。
It USES a scheduler to initiate periodic events, such as crawler executions and full-text index maintenance.
它使用一个调度程序来发起周期性的事件，比如爬网程序执行和全文索引维护。
Next, navigate to the crawler details page and click 'Start full recrawl', as shown at the bottom of Figure 3.
接下来，导航到爬行器的细节页面并单击“Startfull recrawl”，如图3底部所示。
Use the OmniFind database crawler wizard to configure a crawler to access the VSAM content through the nickname.
使用OmniFind数据库crawler向导来配置一个crawler，从而通过昵称访问VSAM内容。
Define the crawler name (UNIX file system crawler 1, for example), as shown in Figure 7, and then click on the Next button.
定义爬虫名称(例如，UNIXfilesystemcrawler1)，如图7所示，然后单击Next按钮。
Word on the street is that director Marc Webb is looking to create a new love interest for his planned reboot of the wall-crawler franchise.
有传言说，导演马克·韦伯会为重新启动的攀墙者系列寻找一个新的恋爱目标。
The high-level task for this article is a Web crawler: given a base URL for a Web site, you'll collect elements from the site that you can use for some purpose.
本文的高级任务是创建一个Webcrawler：给定一个网站的基URL，从该网站收集可以用作某种用途的元素。
As with any search project, our faceted navigation demo requires that we define content sources and configure a crawler or other device to ingest and index them.
对于任何搜索项目，分面导航演示都要求定义内容源，配置收集这些内容源并为它们建立索引的信息收集器或其他设备。
Assuming that SCA and MDB applications were already deployed and started, ensure that the ica crawler and indexer for a particular document collection are running.
假设sca和MDB应用程序已经部署并启动，确保针对特定文档集合的ICA爬行器和索引器均已运行。
An effective site map is built as a high level overview of the different pages in the site and presents a large list of links to the crawler in as few pages as possible.
高效站点地图将会构建为站点中不同页面的简要概述，并且会在尽可能少的页面中为爬网程序呈现大量的链接的列表。
Today, the approach that many search engines still recommend for some level of crawling efficiency is to make an HTML-based site map page available for a crawler to pick up.
如今，许多搜索引擎仍然推荐实现某一级别爬行效率的方法，那就是使爬虫程序可以爬行基于HTML 的站点地图页面。
You can see that the home page has already been fetched and indexed, and the crawler is working on fetching and indexing the pages that have been linked directly from it.
从图中可以看到已经获取并索引了主页，程序现在的工作是获取和索引直接从该主页链接出的页面。
You can see that the home page has already been fetched and indexed, and the crawler is working on fetching and indexing the pages that have been linked directly from it.
从图中可以看到已经获取并索引了主页，程序现在的工作是获取和索引直接从该主页链接出的页面。
应用推荐