|
RoboGen ROBOTS.TXT Editor
by Rietta Solutions
RoboGen is a visual editor for Robot Exclusion Files, ROBOTS.TXT. RoboGen presents a friendly and familier FTP-like interface that makes creating exclusion rules as easy. RoboGen maintains a database of over 180 search engine user-agents.
search engine robot meta tag martijn koster robots.txt robot exclusion
|
