Indexing - create an index of web crawl data.
索引——创建网页检索数据索引。
This information helps the Web crawler determine what the set of pages is and when to crawl them.
这一信息能帮助web爬虫程序决定要爬行哪些页面以及爬行的时间。
In this paper, through the sort of the emergency event case website and improvement of the crawl algorithm, we get our emergency focused web crawler.
论文通过对突发事件案例网站进行分类，改进爬行器算法，实现面向突发事件案例的爬行器。
Sitemaps 0.90 is a simple and intuitive way for webmasters to provide the right level of information to Web crawlers so that they can efficiently crawl a Web site.
Sitemaps 0.90为Web 站点管理员提供了简单而且直观的方法，使他们可以向web爬虫程序提供正确的信息，从而使爬虫程序能够高效地爬行某个 Web 站点。
He gave an example of a web crawl they compressed with the system. The crawl contained 2.1B pages and the rows were named in the following form: “com.cnn.www/index.html:http”.
这个例子的蜘蛛包含 2.1B 的页面，行按照以下的方式命名：“com.cnn.www/index.html:http”.在未压缩前的web page 页面大小是：45.1 TB ，压缩后的大小是：4.2 TB ， 只是原来的 9.2%。
Tools like WebScarab and Paros capture conversations between the browser and server as well as crawl Web sites to help you identify potential risks.
诸如WebScarab和Paros之类的工具捕获浏览器和服务器之间的会话，爬行web站点以帮助您识别潜在的风险。
Using Portal Search, administrators can define content source crawlers which use the HTTP protocol to crawl and index Web sites or content repositories.
使用PortalSearch，管理员可以定义内容源爬网程序，爬网程序使用HTTP协议来抓取Web站点或内容存储库并为其建立索引。
The crawling process is the most crucial part of the scan, so you should always make sure that the web vulnerability scanner is able to crawl all of the website’s objects and inputs.
这个抓取的过程在扫描过程中是至关重要的一步，因此你要确定web漏洞扫描器能够抓取关于站点的所有对象和输入点。
Usually, users are forced to crawl the web or mine social-media sites to build their own.
通常，用户被迫抓取web或者挖掘社会媒体网站来建立他们自己的大型数据集。
If you want to crawl a web site and index that, you can use htdig2omega, which accepts a URL and searches the entire web site.
如果想要对一个web站点进行爬行并索引，可以使用htdig2omega，它能接受一个URL并搜索整个web站点。
We did not crawl the data because there is no good sources on the web - they are either incomplete or incorrect sometimes.
我们不自己组织数据因为网络上没有好的来源——它们不是不完整就是有些有错误。
Most bots are really dumb, they crawl the web and put their spam into every form's field they can find.
大多数的机器人是愚蠢的，它们检索网络，并且把垃圾信息放到它们能找到的每一个表单域里。
You'll collect a series of links from a single Web page and then crawl a whole site.
您将从单个网页收集一系列链接，然后蔓延到整个网站。
You can use a shorter or longer timeout based on how long you want the program to run, your network connection's speed, and the depth of the Web site you want to crawl.
根据想要让程序运行多久、网络连接速度和想对网站进行处理的深度，您可以使用更长或更短的超时。
How can the meta robots tag impact how search engines crawl, index and display content on a web page?
metarobots标签是如何影响搜索引擎爬虫抓取、索引并显示网页的？
To sign up, send an email to cricket_crawl@yahoo.com, or visit the Cricket Crawl web page.
报名请发邮件到cricket_crawl@yahoo.com,或者访问我们的活动网址。
Hadoop's longer-term goal is to provide world-class distributed computing tools that will support next- generation web-scale services such as analyzing web search crawl data.
Hadoop的长期目标是提供世界级的分布式计算工具，也是对下一代业务（如搜索结果分析等）提供支持的Web扩展（web-scale）服务。
On top of that, your shopping cart has slowed to a crawl because thousands of packets of useless data are slamming into your Web site, creating annoyance and outright anger in your customer base.
此外，您的购物车也慢得跟蜗牛一样，因为数千个无用数据包正在冲击您的Web站点，这也使客户烦恼和愤怒。
We know is the role of site map search engine crawl to content, crawl through the site map submission, thus completed grab included, the technique is used in large and giant web site.
我们很清楚网站地图的作用是将搜索引擎没有抓取到的内容，通过网站地图提交进行抓取，从而完成抓取收录，该技巧多被用于大型和巨型网站。
Crawl web content specified in the regular content. Do the robot developed by see.
说明：抓取网页内容中指定正则内容。对做网页抓取开发可以借见。
Often a bony lanky black spider will crawl around your web site wiggle its little feet in excitement gather up your email address and store it near its pinchers.
通常，一只瘦长的“黑蜘蛛”会在网站周围徘徊，兴奋地挥舞着腿脚，窃取你的电子邮箱地址。
Firstly, the basic algorithm to analyze web analytics Summary: If breadth-first strategy based on priority strategies and best way to crawl web pages.
首先对基本的网页分析算法进行分析综述：如基于广度优先策略和最佳优先策略的网页抓取方法。
Spider - The spider feature of Burp Suite is used to crawl web applications looking for new links, content, etc.
BurpSuite的爬行器功能主要是用来抓取web应用的新链接，内容，等等。
We add and update new sites to our index each time we crawl the Web, and we ask you to submit your URL here.
每一次全面扫网时我们会增加并更新索引中的网站，同时我们要求你在这里提交自己的网址。
Note that the web site crawler cannot crawl secured pages.
请注意，该网站的抓取工具无法抓取安全网页。
And when we crawl the Web, we crawl it pretty quickly, which can really stress the system.
而当我们检索网页的时候，速度很快，这也给系统带来了很大压力。
And when we crawl the Web, we crawl it pretty quickly, which can really stress the system.
而当我们检索网页的时候，速度很快，这也给系统带来了很大压力。
应用推荐