Use a site map to lead the crawler around your site.

使用站点地图引导爬行器遍历您的站点。

youdao

Define the crawler subdirectories, as illustrated in Figure 8.

定义爬虫子目录，如图8所示。

youdao

Notice also that only the base URI string is added to the crawler.

另外请注意，只有基 URI 字符串被添加到了 crawler。

youdao

There is no simple way to know for sure if Atlantis has some bad wires like those that flunked the inventory inspection, because they cannot be reached until the shuttle stack is enshrouded in scaffolding inside a "High-Bay" three slow miles down the crawler road from the pad.

CNN: Challenger explosion

Even though the sensitive information was hidden in the HTML and not visible in plain view, the Google crawler observed it and recorded the information to put into its search index.

FORBES: Blippy Apologizes For Google-Searchable Credit Cards, Announces New Security Plan

It said this was largely due to its partnership with the news crawler Moreover and integrates Moreover's news headlines into its regular listings.

BBC: Search engines swamped