|
RoboGen ROBOTS.TXT Editor is developed or distributed by Rietta Solutions and is licensed as shareware. Create and Manage Robot Exclusion Files, ROBOTS.TXT for your website. RoboGen is a visual editor for Robot Exclusion Files. It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which parts of a web site are not to be indexed and made searchable by the general web public. RoboGen does this by providing the user a way to log onto his FTP server and then select the documents and directories which are not to be made searchable. RoboGen maintains a database of over 180 search engine user-agents, which are selectable from a drop down menu....
|
