Note that the web site crawler cannot crawl secured pages.

请注意，该网站的抓取工具无法抓取安全网页。

youdao

This paper provides a Maintaining WAP Site Crawler system. This system can automatically traverse the WAP site, parse every page in the site and check syntax and semantic faults.

本文提出了一种维护WAP网站的网络爬虫系统，该系统可以自动遍历WAP网站，并对网页进行分析，检查语法和语义的错误。

youdao

Today, the approach that many search engines still recommend for some level of crawling efficiency is to make an HTML-based site map page available for a crawler to pick up.

如今，许多搜索引擎仍然推荐实现某一级别爬行效率的方法，那就是使爬虫程序可以爬行基于 HTML 的站点地图页面。

youdao