• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

David Muñoz

  • Experiments
    • Indexation
    • Crawling
  • About me
  • Contact
  • Cambiar a ESCambiar a ES
  • You are here: Home / Experiments / Crawling

    Crawling

    Does Google follow redirects to pages blocked by robots.txt?

    David Leave a Comment

    TL;DR: No, at least in this experiment Google has evaluated the final page of the redirects and when it was blocked in robots.txt it has not followed the redirect. The goal of this experiment is to verify how the Google crawler reacts when it finds a redirection that points to a URL blocked with robots.txt […]

    Filed under: Crawling

    Primary Sidebar

    CATEGORIES

    • Crawling (1)
    • Indexation (2)

    LATEST POSTS

    • Does Google follow redirects to pages blocked by robots.txt?
    • Does Google index the words present in a domain name?
    • How to find out when a website was published?
    linkedin/in/davidmunozsanchez/

    • Privacy Policy
    • Cookie Policy
    • Legal Notice
    You accept cookies from this site? more info