|
| Do-download.com >E-Commerce >Robots.txt> Downloading Robots.txt 2.0 |
Robots.txt is a visual editor for Robot Exclusion Files and a log analyzer software. It allows a user to quickly and easily create the Robots.txt files, required to instruct search engine spiders, which parts of a Web site are not to be indexed
|
|
 |
|
Robots.txt 2.0 download will automatically start in 5 sec...
|
|
Click here if it does not
|
|
|
|
|
Robots.txt Related Software | - Site Spider 1.0.0
Spiders a site and creates output files containing (1) a list of URLs in the site, (2) a Google sitemap, (3) an XML summary file with title, description, and keywords for each page, and (4) links to URLs external to the spidered domain
|
|
|
|