Welcome to HBH! If you have tried to register and didn't get a verification email, please using the following link to resend the verification email.

robots.txt

  1. Home
  2. Forum
  3. Web hacking
  4. robots.txt

ghost's Avatar

ghost

0 0

okay before everyone trie to explain to me what the purpose of …/robots.txt is, I KNO

so my question is what do you guys think this means if you were to find it on a …/robots.txt " User-agent: * Disallow: / "

would i have to change my useragent to "*" and what does the dissallow part mean?:D:D -ZTB


ghost's Avatar

ghost

0 0

well apparently u dont kno what robots.txt does. It says which search engines (aka user agents, such as googlebot) can't display search results based on what come after "disallow:". Also, and this is a general note, in robots.txt and in most places "*" is a wildcard meaning anything. Also, "/" specifies the root directory of the web site, so what u posted means that the website with that robots.txt file dosen't allow any search engine to search any of the site. Hope this helps :)'