Thanks for visiting! If you're new here, you may want to receive FREE daily updates to our Posts. So you won't miss a single article we write about technology!
Recently when we opened our new Teckitech Forums, we have looked constantly for a resource or anything that would help us block these bad bots. Not only do these bad bots steal your precious bandwidth and speed of your sever it also crawls your website and forums for your users personal information like names, email addresses, msn id. These bots and spam agents are getting smarter day by day. I have read recently somewhere that it now possible for these bots to read an email address even if you type it on an image. You can see the post on Complete Wordpress SEO Guide which has a plugin called KB Robot.txt that can also help you in robot.txt file.
So i have found this great file online and examined it so that it wouldn’t be spam itself allowing bad bots and blocking all good bots. I have found that it allows doesn’t block Google, MSN and other major search engine bots but certainly does block some nasty ones i have heard of, online! I have made a post about this news on our forums which you can read over here.
It is very easy to implement this robot.txt file onto your website. All you have to do is create a notepad file and copy and paste the content below. Now, save this file as robot.txt (or Download) and place it on the root of your domain. Say if your blog or website is located at sub.example.com, you place the robot.txt at the very root of the sub.example.com. On my server it would be /public_html/sub.example.com or /public_html/sub. You can also customize this robot.txt file for wordpress to restrict bots crawling your directories.
User-agent: BotRightHere
Disallow: /User-agent: WebZip
Disallow: /User-agent: larbin
Disallow: /User-agent: b2w/0.1
Disallow: /User-agent: Copernic
Disallow: /User-agent: psbot
Disallow: /User-agent: Python-urllib
Disallow: /User-agent: NetMechanic
Disallow: /User-agent: URL_Spider_Pro
Disallow: /User-agent: CherryPicker
Disallow: /User-agent: EmailCollector
Disallow: /User-agent: EmailSiphon
Disallow: /User-agent: WebBandit
Disallow: /User-agent: EmailWolf
Disallow: /User-agent: ExtractorPro
Disallow: /User-agent: CopyRightCheck
Disallow: /User-agent: Crescent
Disallow: /User-agent: SiteSnagger
Disallow: /User-agent: ProWebWalker
Disallow: /User-agent: CheeseBot
Disallow: /User-agent: LNSpiderguy
Disallow: /User-agent: Alexibot
Disallow: /User-agent: Teleport
Disallow: /User-agent: TeleportPro
Disallow: /User-agent: MIIxpc
Disallow: /User-agent: Telesoft
Disallow: /User-agent: Website Quester
Disallow: /User-agent: WebZip
Disallow: /User-agent: moget/2.1
Disallow: /User-agent: WebZip/4.0
Disallow: /User-agent: WebStripper
Disallow: /User-agent: WebSauger
Disallow: /User-agent: WebCopier
Disallow: /User-agent: NetAnts
Disallow: /User-agent: Mister PiX
Disallow: /User-agent: WebAuto
Disallow: /User-agent: TheNomad
Disallow: /User-agent: WWW-Collector-E
Disallow: /User-agent: RMA
Disallow: /User-agent: libWeb/clsHTTP
Disallow: /User-agent: asterias
Disallow: /User-agent: httplib
Disallow: /User-agent: turingos
Disallow: /User-agent: spanner
Disallow: /User-agent: InfoNaviRobot
Disallow: /User-agent: Harvest/1.5
Disallow: /User-agent: Bullseye/1.0
Disallow: /User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /User-agent: CherryPickerSE/1.0
Disallow: /User-agent: CherryPickerElite/1.0
Disallow: /User-agent: WebBandit/3.50
Disallow: /User-agent: NICErsPRO
Disallow: /User-agent: Microsoft URL Control - 5.01.4511
Disallow: /User-agent: DittoSpyder
Disallow: /User-agent: Foobot
Disallow: /User-agent: SpankBot
Disallow: /User-agent: BotALot
Disallow: /User-agent: lwp-trivial/1.34
Disallow: /User-agent: lwp-trivial
Disallow: /User-agent: BunnySlippers
Disallow: /User-agent: Microsoft URL Control - 6.00.8169
Disallow: /User-agent: URLy Warning
Disallow: /User-agent: Wget/1.6
Disallow: /User-agent: Wget/1.5.3
Disallow: /User-agent: Wget
Disallow: /User-agent: LinkWalker
Disallow: /User-agent: cosmos
Disallow: /User-agent: moget
Disallow: /User-agent: hloader
Disallow: /User-agent: humanlinks
Disallow: /User-agent: LinkextractorPro
Disallow: /User-agent: Offline Explorer
Disallow: /User-agent: Mata Hari
Disallow: /User-agent: LexiBot
Disallow: /User-agent: Web Image Collector
Disallow: /User-agent: The Intraformant
Disallow: /User-agent: True_Robot/1.0
Disallow: /User-agent: True_Robot
Disallow: /User-agent: BlowFish/1.0
Disallow: /User-agent: JennyBot
Disallow: /User-agent: MIIxpc/4.2
Disallow: /User-agent: BuiltBotTough
Disallow: /User-agent: ProPowerBot/2.14
Disallow: /User-agent: BackDoorBot/1.0
Disallow: /User-agent: toCrawl/UrlDispatcher
Disallow: /User-agent: suzuran
Disallow: /User-agent: TightTwatBot
Disallow: /User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /User-agent: VCI
Disallow: /User-agent: Szukacz/1.4
Disallow: /User-agent: Openfind data gatherer
Disallow: /User-agent: Openfind
Disallow: /User-agent: Xenu’s Link Sleuth 1.1c
Disallow: /User-agent: Xenu’s
Disallow: /User-agent: Zeus
Disallow: /User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /User-agent: RepoMonkey
Disallow: /User-agent: Microsoft URL Control
Disallow: /User-agent: Openbot
Disallow: /User-agent: URL Control
Disallow: /User-agent: Zeus Link Scout
Disallow: /User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /User-agent: Webster Pro
Disallow: /User-agent: EroCrawler
Disallow: /User-agent: LinkScan/8.1a Unix
Disallow: /User-agent: Keyword Density/0.9
Disallow: /User-agent: Kenjin Spider
Disallow: /User-agent: Iron33/1.0.2
Disallow: /User-agent: Bookmark search tool
Disallow: /User-agent: GetRight/4.2
Disallow: /User-agent: FairAd Client
Disallow: /User-agent: Gaisbot
Disallow: /User-agent: Aqua_Products
Disallow: /User-agent: Radiation Retriever 1.1
Disallow: /User-agent: Flaming AttackBot
Disallow: /
Hope this helps and please share your comments and thoughts on this!
Roshan Reply:
October 9th, 2008 at 1:04 pm
Hi Samantha
I just wanted to list some of these so that we can atleast block the ones which are in our control. Yeah! you are right, we can’t stop all of them but we can atleast do something which is in our control. Thank you very much for your comment and i very much appreciate it!
[Answer]