• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

DAVID.MU

  • Experiments
    • Indexing
    • Crawling
  • About me
  • Contact
  • Cambiar a ES
  • You are here: Home / Archives for Experiments / Crawling

    Crawling

    Does Google follow redirects to pages blocked by robots.txt?

    David

    TL;DR: No, at least in this experiment Google has evaluated the final page of the redirects and when it was blocked in robots.txt it has not followed the redirect. The goal of this experiment is to verify how the Google crawler reacts when it finds a redirection that points to a URL blocked with robots.txt […]

    Filed under: Crawling

    Primary Sidebar

    CATEGORIES

    • Crawling (1)
    • Indexing (2)

    LATEST POSTS

    • Does Google follow redirects to pages blocked by robots.txt?
    • Does Google index the words present in a domain name?
    • How to find out when a website was published?
    linkedin/in/davidmunozsanchez/

    © 2025 DAVIDMU SL

    • Cookie Policy
    • Privacy Policy
    • Terms of service
    You accept cookies from this site? more info