|
RoboGen ROBOTS.TXT Editor
by Rietta Solutions
RoboGen is a visual editor for Robot Exclusion Files, ROBOTS.TXT. RoboGen presents a friendly and familier FTP-like interface that makes creating exclusion rules as easy. RoboGen maintains a database of over 180 search engine user-agents.
robots.txt search engine robot exclusion robot meta tag martijn koster
|
