# ========================================================= # VersantPro - Robots.txt # AI English Fluency & Versant Practice Platform # Worldwide SEO + GEO Crawling Configuration # Updated: 2026-08-12 # ========================================================= # --------------------------------------------------------- # DEFAULT: Allow all search engines and eligible crawlers # --------------------------------------------------------- User-agent: * Allow: / # Private / administrative areas Disallow: /admin/ Disallow: /private/ Disallow: /api/ # Raw JSON files should not appear in search results Disallow: /*.json$ # --------------------------------------------------------- # GOOGLE # --------------------------------------------------------- User-agent: Googlebot Allow: / Disallow: /admin/ Disallow: /private/ Disallow: /api/ Disallow: /*.json$ # --------------------------------------------------------- # BING # --------------------------------------------------------- User-agent: Bingbot Allow: / Disallow: /admin/ Disallow: /private/ Disallow: /api/ Disallow: /*.json$ # --------------------------------------------------------- # OTHER SEARCH ENGINES # --------------------------------------------------------- User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: YandexBot Allow: / # --------------------------------------------------------- # SEO CRAWLERS # Keep these blocked as in the original configuration # --------------------------------------------------------- User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / # --------------------------------------------------------- # SITEMAP # --------------------------------------------------------- Sitemap: https://versantpro.in/sitemap.xml # ========================================================= # END # =========================================================