• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

DAVID.MU

  • Experiments
    • Indexing
    • Crawling
  • About me
  • Contact
  • Cambiar a ES
  • You are here: Home / Archives for Experiments

    Experiments

    Does Google follow redirects to pages blocked by robots.txt?

    David

    TL;DR: No, at least in this experiment Google has evaluated the final page of the redirects and when it was blocked in robots.txt it has not followed the redirect. The goal of this experiment is to verify how the Google crawler reacts when it finds a redirection that points to a URL blocked with robots.txt […]

    Filed under: Crawling

    Does Google index the words present in a domain name?

    David

    thompson

    TL;DR: Yes, Google has indexed the words present in the domains used for this experiment and includes them in the information retrieval process, showing the domains as results to a search containing the word. The goal of this experiment is to check if Google takes into account the words present in a domain when selecting […]

    Filed under: Indexing

    How to find out when a website was published?

    David

    TL;DR: In order to find out the approximate date when a website was published you have to: 1. Search the URL in Google. 2. Add &as_qdr=y15 at the end of the URL showing up in your browser bar after searching and then hit enter. 3. The date will appear to the left of the search […]

    Filed under: Indexing

    Primary Sidebar

    CATEGORIES

    • Crawling (1)
    • Indexing (2)

    LATEST POSTS

    • Does Google follow redirects to pages blocked by robots.txt?
    • Does Google index the words present in a domain name?
    • How to find out when a website was published?
    linkedin/in/davidmunozsanchez/

    © 2025 DAVIDMU SL

    • Cookie Policy
    • Privacy Policy
    • Terms of service
    You accept cookies from this site? more info