从一个网站抓取一张图片或一段文字，然后发给另一个网站社群内的朋友、或者张贴到自己的博客上可能会让你感到不便。
It can be awkward to snag a photo or a snippet of text from one Web site and send it to a friend in a social network on another, or post it to your own blog.
GoogleNews是一个新闻聚合服务，它从全世界大量不同的新闻机构和在线网站抓取新闻，然后将这些新闻显示在一个便于搜索的统一站点中。
Google news is a news aggregation service, which draws from many different news agencies and online sites around the world to present the news in a single unified site which is easily searchable.
一个内部信息的例子可能是数据中心的日志文件，外部信息可能是一些抓取的网站或从数据目录下载的数据集。
An example of internal information might be log files from a data center, and external information might be several crawled websites or a dataset downloaded from a data catalog.
网站管理工具允许你检查抓取的内容，反向链接，出站链接，也可以是站点地图。
The webmasters tool allows you to check crawl issues, back links, outbound links and also sitemap.
一旦你确认了所有者身份，你可以依次访问”网站配置“ > ”抓取工具的权限“ > ”删除网址“ > ”已经删除的网址（或者其他人的请求）“，然后在人员你想取消的请求对应处点击取消。
Once you've verified ownership, you can go to Site configuration > Crawler access > Remove URL > Removed URLs (or > Made by others) and click "Cancel" next to any requests you wish to cancel.
网站有很多热区，搜索引擎蜘蛛会定期从中抓取关键字。
Your website is full of hot spots search engine spiders check regularly for keywords.
被蜘蛛抓取的内容越新，关联性越强，你的网站排名就越有可能靠前。
The more fresh, relevant content they find, the higher the search engine spiders are likely to rank your site.
对SEO的影响：功能差的网站只有很少的站外链接，未必看到爬虫抓取网站深处的内容。
SEO impact: Weaker sites with few inbound links are unlikely to see spiders crawling deep content.
点击网站配置-抓取工具权限。
通常，用户被迫抓取web或者挖掘社会媒体网站来建立他们自己的大型数据集。
Usually, users are forced to crawl the web or mine social-media sites to build their own.
在我的例子中，我只要搜索官方文件，而这些正是Google已经做的：返回的结果中从包含设置的网站中抓取的，没有垃圾网站和错误的数据。
In my case, I wanted to only search official documents, and that's exactly what Google has done - returning results that are crawled from those pages only. No garbage, spam sites or erroneous data.
一旦你的网站或者博客通过验证，你就可以看到你所提交的网站的完整的细节，如抓取的大量网站地址目录，与内容等等。
Once your website or blog is validated, you can see the complete details of the website submitted like number of URLs indexed, any issues with the crawling etc.
如果说不带任何商业偏见地抓取网站是一个搜索引擎的道德义务的话，那么，至少在竞争者眼中，百度不是一个有道德的公司。
If crawling the Web empirically and without commercial bias is the moral duty of a search engine, Baidu is, at least in the eyes of its competitors, not a moral company.
从你的网站管理员工具账户中删除抓取错误。
屏幕抓取涉及非法进入瑞安航空的网站，并且以过高的价格和隐瞒消费者的加价非法出售机票。
Screenscraping involves gaining unauthorised access to the Ryanair website and mis-selling of flights to consumers with exorbitant charges and mark-ups which are hidden from consumers.
汉堡法庭在2009年5月做出决定认为，用屏幕抓取技术倒卖瑞安航空公司的机票是非法的。 紧接着瑞安航空继续向爱尔兰法庭控告这些用屏幕信息抓取技术以倒卖机票的网站。
Following the May 2009 decision of the Hamburg courts that screenscraping to resell Ryanair’s flights is unlawful, Ryanair continues to pursue screenscraper websites in the courts in Ireland.
你可以使用网站管理员工具中的“像Googlebot一样抓取”或者“测试robots . txt”功能来检查目录是否被正确拦截。
You can test whether a directory has been blocked correctly using either the Fetch as Googlebot or test robots.txt features in Webmaster Tools.
Flipboard似乎使用了一种类似的技术将网站的内容抓取下来在应用程序中索引。
It appears that Flipboard USES a very similar technique to scrape content from the sites that are indexed within the app.
答：不同的搜索引擎使用不同的算法来抓取和索引的网站。
Answer: different search engines use different algorithms to crawl and index sites.
当Xenu抓取完一个网站，它会生成一个包含有效url的报表。
After Xenu finishes crawling a site, it generates a report that contains the list of valid urls.
简单起见，Put. io从网上抓取文件，并且允许你存放在该网站上，或者立刻播放。
Use Put.io. Put simply, Put.io fetches files from the Internet and allows you to either store them there or immediately stream them.
在通常的情况下，文档的数据来源可能是外部(比如数据库，文件系统，蜘蛛从网站上的抓取等)，这些通常都比较耗时，尽量优化获取它们的性能。
Speed up document construction. Often the process of retrieving a document from somewhere external (database, filesystem, crawled from a Web site, etc.) is very time consuming.
我们很清楚网站地图的作用是将搜索引擎没有抓取到的内容，通过网站地图提交进行抓取，从而完成抓取收录，该技巧多被用于大型和巨型网站。
We know is the role of site map search engine crawl to content, crawl through the site map submission, thus completed grab included, the technique is used in large and giant web site.
网站地图的三大因素：文本、连结、关键词，都极其有利于搜寻引擎抓取主要页面内容。
Site Map of the three major factors: the text, links, key words, are extremely beneficial to the search engines crawl the main page content.
我们知道很多网站上都是有网站地图的，其目的的为了方便搜索引擎抓取，从而增加网站的收录，以此来提高网站关键词排名。
We know that many web site is a site map, its purpose in order to facilitate the search engine grab, thereby increasing site included, in order to improve the site keywords ranking.
我们知道很多网站上都是有网站地图的，其目的的为了方便搜索引擎抓取，从而增加网站的收录，以此来提高网站关键词排名。
We know that many web site is a site map, its purpose in order to facilitate the search engine grab, thereby increasing site included, in order to improve the site keywords ranking.
应用推荐