# robots.txt for https://migadev.com/ # Allow all web crawlers access to all content User-agent: * Disallow: # Specify the location of the sitemap Sitemap: https://migadev.com/sitemap.xml # Block specific folders or files if needed (examples) # Disallow: /private/ # Disallow: /temp/ # Disallow: /secret-page.html # Allow specific bots or give special instructions # User-agent: Googlebot # Allow: /special-folder/ # Block a specific bot # User-agent: BadBot # Disallow: / # Additional instructions or comments can be placed here