Look, I'm the google bot.
Look, I'm the google bot.
ghost | 9795 Reads | What you'll need: Firefox, The Modify Headers extension for firefox.
What you can do: Many sites allow google to look into their forums, in order to get more traffic by getting more google search results. By pretending to be the google search indexer (aka spider, bot) we can access parts of these sites off limits to guests.
Example URL: http://www.tuts4you.com/forum/index.php?showtopic=9959 I found this address today doing a google search, and noticed that although I didn't have access, google had a cached version. Head over to this url, and you should get the same error as me.
First, head over to google and type in 'browser headers'. The first link should take you to a page which gives you all the information your browser sends to web servers. This information, specifically the User-Agent field, is how the google bot tells web servers what it is. Go back to the google search, and click on the 'cached' link.
Notice in the 'User-Agent' field, where it once showed your browser info, now it says: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Download the Modify Headers extension. (http://modifyheaders.mozdev.org/) And open it up. Next you need to make a rule, modifying the 'User-Agent' header to say 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' Once you're done, make sure it's enabled, and browse over to the example url. Even though you're not logged in, you should be able to browse their forums.
Let me know what you think.
Digitalchameleon
ghost 16 years ago
There's also a way to do it with registry editing, HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings
korg 16 years ago
Been using this for quite awhile, Sites that block you from viewing anything unless you join or Fuckin' pay (IE:Experts-exchange) Is bullshit. Works great. Also works on download sites that limit you also. @DarkMessiah You need to change these values in IE for this to work: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\User Agent\Post Platform
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\Version Vector
Change these reg. keys to the value you want to apply. Just remember to save your original (as always) to change them back.
Great article for people who didn't know this.
ghost 16 years ago
Awesome :D I had no clue about this, tested it a minute ago and works great :D Thanks for this article:happy:
ghost 16 years ago
Didn't know about the registry hack. Have to remember that, for situations in which I am away from my computer.:D
ghost 16 years ago
@korg, it's real easy to view Experts Exchange using the NoScript extension… just scroll to the bottom of the page, and all the answers are there after the b.s. "you must join to see the answer" messages. Javascript for the loss! :p @DC… cute trick. Might have to give it a whirl to see what I can make of it. Article actually covered material that wasn't already covered on here, so I have to give it an 'Awesome' for originality.
ghost 16 years ago
It is true that the worst content here receives the most attention. I have many better articles.
alldatizholy 16 years ago
i tried it on alot of sites and it dont work….http://www.gigposters.com/forums/premium-opportunities/61653-radio-publicity.html
alldatizholy 16 years ago
i tried it on alot of sites and it dont work….http://www.gigposters.com/forums/premium-opportunities/61653-radio-publicity.html
ghost 16 years ago
Most sites have noticed the googlebot uses only a limited amount of ip addresses. They filter on that now. Maybe yahoo or altavista or some other search engine spider still works. That would be interesting to find out.