Welcome to HBH! If you have tried to register and didn't get a verification email, please using the following link to resend the verification email.

Mirroring and robots.txt

  1. Home
  2. Forum
  3. Web hacking
  4. Mirroring and robots.txt

ghost's Avatar

ghost

0 0

Is there a way to mirror a site to my HD without robots.txt making shit? I learned this the hard way.


Uber0n's Avatar

Uber0n

Member
0 0

Um.. Exactly what do you mean? Robots.txt is for making sure that web crawlers don't add any pages you want to keep secret and some other settings.


ghost's Avatar

ghost

0 0

But isn't that the file that stops mirroring bots as well? I think I made the wrong assumption here. :whoa: