AstraciaBot

1. What AstraciaBot does

When a user starts research, Astracia visits public pages of potential competitor websites. The crawler identifies itself as:

User-Agent: AstraciaBot/1.0 (+https://astracia.com)

• reads and respects robots.txt before the first request to a domain;
• reads public HTML pages, such as the homepage, pricing, about, changelog and careers pages, and sitemap.xml;
• extracts report information such as messaging, prices, technology, site structure and change dates;
• rechecks monitored pages at intervals of one day or longer to identify changes.

2. What it does not do

• It does not log in, create accounts or bypass paywalls, CAPTCHAs or technical protections.
• It reads a small predefined set of pages per domain rather than crawling every page.
• It does not submit forms or post comments.
• It does not download images, videos or downloadable files.
• It does not collect personal information about site visitors or users or create profiles of individuals.

3. robots.txt and request limits

• We follow rules for User-agent: AstraciaBot, falling back to User-agent: *. The longest, most specific matching prefix takes precedence.
• We send at most a few requests per minute per domain and do not revisit the same URL sooner than 24 hours.
• We respect HTTP 429 and 503 responses and stop requests for that day.
• Responses are cached so research by different users does not require duplicate crawling.

4. How to block AstraciaBot

Add this rule to your website’s robots.txt:

User-agent: AstraciaBot
Disallow: /

The crawler reads it on its next visit and stops. To exclude a domain immediately and permanently, or request removal of collected public website information from reports, contact office@astracia.com. We manually exclude domains within 30 days, usually on the same day.

5. Contact

Questions about the crawler, public data or removal: office@astracia.com. See the public competitor information section of our Privacy policy and our Terms of service.