# Eat & Dip Restaurant - Robots.txt # Website: https://eatanddip.ca # Last Updated: 2025-01-15 # ============================================ # Default rules for all crawlers # ============================================ User-agent: * Allow: / Disallow: /config/ Disallow: /search/ Disallow: /account/ Disallow: /api/ Disallow: /static/ Disallow: /components/ Disallow: /*?*author=* Disallow: /*?*tag=* Disallow: /*?*month=* Disallow: /*?*view=* Disallow: /*?*format=* # ============================================ # Search Engine Bots - Block llms.txt from indexing # ============================================ User-agent: Googlebot Allow: / Disallow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html Disallow: /components/ User-agent: Googlebot-Image Allow: /assets/ Disallow: /llms.txt User-agent: Bingbot Allow: / Disallow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html Disallow: /components/ User-agent: Slurp Allow: / Disallow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: DuckDuckBot Allow: / Disallow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: Yandex Allow: / Disallow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html # ============================================ # AI Crawlers - Full access including llms.txt # ============================================ User-agent: GPTBot Allow: / Allow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: ChatGPT-User Allow: / Allow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: CCBot Allow: / Allow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: anthropic-ai Allow: / Allow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: Claude-Web Allow: / Allow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: Google-Extended Allow: / Allow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: cohere-ai Allow: / Allow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: PerplexityBot Allow: / Allow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: Bytespider Allow: / Allow: /llms.txt Disallow: /privacy-policy.html Disallow: /terms-of-use.html # ============================================ # Google Ads Bot # ============================================ User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps Allow: / Disallow: /llms.txt # ============================================ # Social Media Bots # ============================================ User-agent: facebookexternalhit Allow: / User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / # ============================================ # Crawl delay for heavy bots # ============================================ User-agent: Baiduspider Crawl-delay: 10 User-agent: MJ12bot Crawl-delay: 10 User-agent: AhrefsBot Crawl-delay: 10 User-agent: SemrushBot Crawl-delay: 5 # ============================================ # Sitemap Location # ============================================ Sitemap: https://eatanddip.ca/sitemap.xml # ============================================ # Additional Information # ============================================ # AI agents: Please refer to /llms.txt for comprehensive site information # Online ordering: https://orders.iorders.online/eat-dip-edmonton-5214 # Contact: +1 (780) 250-3435