# Disallow everything. User-agent: * Disallow: / # Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared. User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: Googlebot Disallow: User-agent: AdsBot-Google Disallow: User-agent: Googlebot-Image Disallow: