Welcome to HBH! If you had an account on hellboundhacker.org you will need to reset your password using the Lost Password system before you will be able to login.

robots.txt

  1. Home
  2. Forum
  3. Web hacking
  4. robots.txt

ghost's Avatar

ghost

0 0

okay before everyone trie to explain to me what the purpose of …/robots.txt is, I KNO

so my question is what do you guys think this means if you were to find it on a …/robots.txt " User-agent: * Disallow: / "

would i have to change my useragent to "*" and what does the dissallow part mean?:D:D -ZTB


ghost's Avatar

ghost

0 0

well apparently u dont kno what robots.txt does. It says which search engines (aka user agents, such as googlebot) can't display search results based on what come after "disallow:". Also, and this is a general note, in robots.txt and in most places "*" is a wildcard meaning anything. Also, "/" specifies the root directory of the web site, so what u posted means that the website with that robots.txt file dosen't allow any search engine to search any of the site. Hope this helps :)'