User-Agent

PRODUCER.GE-SEO-Bot

This is the crawler of PRODUCER.GE SEO, a technical SEO audit service. It visits a site only when someone starts an audit of it here, or when the owner of a verified site has scheduled monitoring.

How it identifies itself

Every request carries this User-Agent:

How it crawls

• It reads robots.txt first and follows the group for PRODUCER.GE-SEO-Bot, or the * group.
• Crawl-delay is respected between 0.5 and 2 seconds; without one it waits at least 0.5 seconds between requests of each worker.
• At most four requests at a time, on the start host only; a site is never audited by two crawls at once.
• Page and time limits: 500 pages for an anonymous audit, larger audits only for sites whose owner has verified them.
• Only public addresses: no private networks, no passwords, only ports 80 and 443.
• It follows links and the sitemap, checks the status of linked files, images and external links with light HEAD requests.

How to limit or block it

Add a group to your robots.txt. For example, to block it completely:

User-agent: PRODUCER.GE-SEO-Bot
Disallow: /

or to slow it down:

User-agent: PRODUCER.GE-SEO-Bot
Crawl-delay: 2

Questions or a problem

Contact us through producer.ge and name the site and the time. producer.ge