# Excessive reqs User-agent: SemrushBot User-agent: SemrushBot-SA User-agent: Yandex User-agent: AhrefsBot User-agent: dotbot Disallow: / # General paths to not crawl User-agent: * Disallow: /actions/ Disallow: /contactUs Disallow: /dashboard Disallow: /home Disallow: /media/ Disallow: /resetPassword Disallow: /schedule Disallow: /schedule/