description,ua Sync2It bookmark management & clustering engine,!Susie (http://www.sync2it.com/susie) UnCHAOS search robot, UnChaos From Chaos To Order Hybrid Web Search Engine.(vadim_gonchar@unchaos.com) UnCHAOS search robot, UnChaos Bot Hybrid Web Search Engine. (vadim_gonchar@unchaos.com) UnCHAOS search robot, UnChaosBot From Chaos To Order UnChaos Hybrid Web Search Engine at www.unchaos.com (info@unchaos.com) Sygol Search (Italy) robot, http://www.sygol.com SearchEngineWorld's robots.txt validator,( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi ) DreamCast DreamPassport browser,(DreamPassport/3.0; isao/MyDiGiRabi) Privoxy web proxy,(Privoxy/1.0) Unknown Yahoo robot,*/Nutch-0.9-dev SitiDi.net search (Germany) robot,+SitiDi.net/SitiDiBot/1.0 (+Have Good Day) Die Kraehe Meta-Search-Engine (Germany) link checking,-DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de 123spider.de (Germany) web directory link checking,"123spider-Bot (Version: 1.02, powered by www.123spider.de" 192.com - UK web directory,192.comAgent 1st ZipCommander Net - IE based browser,1st ZipCommander (Net) - http://www.zipcommander.com/ 2Bone online link checker,2Bone_LinkChecker/1.0 libwww-perl/5.64 4Anything robot,4anything.com LinkChecker v2.0 Unknown guestbook spamming or harvesting tool from diff. IPs,8484 Boston Project v 1.0 neofonie search robot Germany,:robot/1.0 (linux) ( admin e-mail: undefined http://www.neofonie.de/loesungen/search/robot.html ) A-Online.at robot - now Jet2Web Search,A-Online Search A1 Keyword Research - search engine and keyword optimization software,A1 Keyword Research/1.0.2 (+http://www.micro-sys.dk/products/keyword-research/) miggibot/2007.03.27 MiggiBot website crawler engine - A1 Sitemap Generator,A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24 Aardvark web crawler for Sun's Blog recommendations,aardvark-crawler Abacho / Crawler.de robot,AbachoBOT Abacho / Crawler.de robot,AbachoBOT (Mozilla compatible) ABCdatos - Castilian program & tutorial directory,ABCdatos BotLink/5.xx.xxx#BBL Aberja Hybridsuchmaschine (Germany) link checking,Aberja Checkomat Nameprotect copyright search robot (24.177.134.x),abot/0.1 (abot; http://www.abot.com; abot@abot.com) About robot,About/0.1libwww-perl/5.47 Accela Technology RSS feed crawler,Accelatech RSSCrawler/0.4 Accoona Search robot,accoona Accoona Search robot,Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) Accoona Search robot,Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) Ace Explorer - IE based browser,Ace Explorer Ackerm search robot,Ack (http://www.ackerm.com/) Acoi picture finder robot,AcoiRobot Acoon.de search (Germany) robot,Acoon Robot v1.50.001 Acoon.de search (Germany) robot,Acoon Robot v1.52 (http://www.acoon.de) Acoon.de search (Germany) robot,Acoon-Robot 4.0.x.[xx] (http://www.acoon.de) Acoon.de search (Germany) robot,Acoon-Robot v3.xx (http://www.acoon.de and http://www.acoon.com) Acorn Search Project,Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org) LibMaster.com Active Bookmark HTML page creator,ActiveBookmark 1.x Activeworlds 3D homepage browser,Activeworlds Activeworlds 3D homepage browser,ActiveWorlds/3.xx (xxx) Ad Muncher - banner killer,Ad Muncher v4.xx.x Ad Muncher - banner killer,Ad Muncher v4x Build xxxxx website directory adaxas link checking,Adaxas Spider (http://www.adaxas.net/) Avant Browser - IE based browser,Advanced Browser (http://www.avantbrowser.com) Aesop robot,AESOP_com_SpiderMan Agada search (Germany) robot,agadine/1.x.x (+http://www.agada.de) SharewarePlaza link checking,Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) Linkomatic submission verifier,AgentName/0.1 libwww-perl/5.48 21seek.com (China) robot (218.17.90.xxx),AIBOT/2.1 By +(www.21seek.com A Real artificial intelligence search engine China) AideRss - Postrank RSS and Blog filtering,AideRSS/1.0 (aiderss.com) Nameprotect copyright search robot (24.177.134.x),aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) Nameprotect copyright search robot (24.177.134.x),aipbot/2-beta (aipbot dev; http://aipbot.com; aipbot@aipbot.com) Akregator news feed reader for KDE,Akregator/1.2.9; librss/remnants Aladin robot,Aladin/3.324 Phone.com UP.Browser for mobiles on Alcatel cellphone,Alcatel-BG3/1.0 UP.Browser/5.0.3.1.2 Aleksika Danmark - Search engine optimization spider,Aleksika Spider/1.0 (+http://www.aleksika.com/) Alertinfo - French version of Feedreader 3.xx ,AlertInfo 2.0 (Powered by Newsbrain) Vestris robot,AlkalineBOT/1.3 Vestris robot,AlkalineBOT/1.4 (1.4.0326.0 RTM) Allesklar.de robot,Allesklar/0.1 libwww-perl/5.46 Alligator download manager,Alligator 1.31 (www.nearsoftware.com) Unknown robot from Allrati.com,Allrati/1.1 (+) Altavista robot,AltaVista Intranet V2.0 AVS EVAL search@freeit.com Altavista robot,AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net Altavista robot,AltaVista Intranet V2.0 evreka.com crawler@evreka.com Altavista robot,AltaVista V2.0B crawler@evreka.com Amaya - W3C's Editor/Browser,amaya/x.xx libwww/x.x.x Amfibi Search robot,AmfibiBOT Amfibi Search robot,Amfibibot/0.06 (Amfibi Web Search; http://www.amfibi.com; agent@amfibi.com) Amfibi Search robot,Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) amibot - Amidalla search engine robot (62.241.33.xx),amibot AWeb Amiga browser,Amiga-AWeb/3.4.167SE Voyager - Amiga browser,AmigaVoyager/3.4.4 (MorphOS/PPC native) Amiga Miami TCP Stack,AmiTCP Miami (AmigaOS 2.04) NF embedded browser on Amois Skypephone,Amoi 8512/R21.0 NF-Browser/3.3 Amazon.com robot for checking their affiliate sites,amzn_assoc Annomille Italian historical oriented robot,AnnoMille spider 0.1 alpha - http://www.annomille.it annotate Google - Firefox extension for annotating Google search results,annotate_google; http://ponderer.org/download/annotate_google.user.js Megaproxy user,Anonymized by ProxyOS: http://www.megaproxy.com faked user agent,Anonymizer/1.1 AnswerBus natural language search using COLLATE technology,AnswerBus (http://www.answerbus.com/) AnswerChase search tool,AnswerChase PROve x.0 AnswerChase search tool,AnswerChase x.0 ANT Fresco Browser,ANTFresco/x.xx Antibot (discontinued) robot,antibot-V1.1.5/i586-linux-2.2 Anzwers (Yahoo) Australia robot,AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) Apexoo Search spider,Apexoo Spider 1.x JavaOS app. for SEGA Saturn Internet and Sanyo Internet-TV,Aplix HTTP/1.0.1 JavaOS app. for Sanyo Internet-TV,Aplix_SANYO_browser/1.x (Japanese) JavaOS app. for SEGA Saturn Internet,Aplix_SEGASATURN_browser/1.x (Japanese) Aport robot,Aport Walhello Internet Search robot,appie 1.1 (www.walhello.com) CoreMedia player on Apple iPhone,Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102 PubSub - Mac OS X utility for managing RSS/Atom subscriptions via the PubSub framework,Apple-PubSub/65.1.1 Araby search - Arabia,ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;) Covac Arachnid Web Crawler,ArachBot Euroseek spider,Arachnoidea (arachnoidea@euroseek.com) Amazon.com robot for checking their affiliate sites,aranhabot Excite spider,ArchitextSpider Heritrix - The Internet Archive's open-source crawler (207.241.225.2xx),archive.org_bot Simpy Bookmarklet crawler (69.55.233.xx),Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com) Arikus inContext search engine software,Arikus_Spider Tomba project: the Portuguese web archive,Arquivo-web-crawler (compatible; heritrix/1.12.1 +http://arquivo-web.fccn.pt) Asaha search robot (Turkey),ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) ASAHINA Antenna information detecting agent,Asahina-Antenna/1.x ASAHINA Antenna information detecting agent,Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x) Ask 24x Info robot,ask.24x.info Ask About Oil - Petroleum related search (24.227.212.xxx) using Nutch,AskAboutOil/0.06-rcp (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@askaboutoil.com) askEd! / Inferret search (Japan) robot using Nutch,asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) ASPSeek search engine software -Yahoo-Inc. / Telecom Canada robot ,ASPSeek/1.2.5 Swsoft.net robot using Aspseek,ASPseek/1.2.9d ASPSeek search engine software,ASPSeek/1.2.x ASPSeek search engine software,ASPSeek/1.2.xa ASPSeek search engine software,ASPseek/1.2.xx ASPSeek search engine software,ASPSeek/1.2.xxpre Associative Sort robot,ASSORT/0.10 Singingfish media spider (64.12.186.2xx) via AOL search,asterias/2.0 Atlocal local business search robot,AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) Atomic Email Hunter email extracing and harvesting,Atomic_Email_Hunter/4.0 Atomz robot,Atomz/1.0 atSpider (ceased) email harvester / spambot,atSpider/1.0 Attentio social media monitoring and analysing ,Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com) Samsung SPH-A660 phone with Sprint software,AU-MIC/2.0 MMP/2.0 Audiovox SMT5600 (AT&T) Smartphone mobile phone browser,AUDIOVOX-SMT5600 Augurnet Swiss (was www.augurnet.ch) search robot,augurfind Augurnet Swiss (was www.augurnet.ch) search robot,augurnfind V-1.x Auto Email Pro Email harvester,autoemailspider Linkscan tool from Elsop,autohttp Autowebdir - The Automatically Generated Web Directory,autowebdir 1.1 (www.autowebdir.com) Altavista robot ??,AV Fetch 1.0 Avant Browser - IE based browser,Avant Browser (http://www.avantbrowser.com) National Research Council Canada robot,AVSearch-1.0(peter.turney@nrc.ca) Unknown robot from 205.203.108.xx (telerate.com),AVSearch-2.0-fusionIdx-14-CompetitorWebSites Altavista robot,AVSearch-3.0(AltaVista/AVC) AWeb Amiga browser,AWeb Axada search Germany robot,axadine/ (Axadine Crawler; http://www.axada.de/; ) Axmo search robot,AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine. Azureus Java BitTorrent Client,Azureus 2.x.x.x Babaloo search robot (Slovenia),BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) BaBoom Web Portal (ODP) robot (66.98.254.xx),BaboomBot/1.x.x (+http://www.baboom.us) BackStreet Browser - Offline browser / website downloader,BackStreet Browser 3.x Baidu search (Japan) image crawler,BaiduImagespider+(+http://www.baidu.jp/search/s308.html) Baidu spidering engine - used by diff. IPs,BaiDuSpider Baidu spidering engine - used by diff. IPs,Baiduspider+(+http://help.baidu.jp/system/05.html) Baidu spidering engine - used by diff. IPs,Baiduspider+(+http://www.baidu.com/search/spider.htm) Baidu search (Japan) crawler,Baiduspider+(+http://www.baidu.com/search/spider_jp.html) Balihoo - Vertical search engine crawler (beta),Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) Project BanBots Perl script robot,BanBots/1.2 (spider@banbots.com) Barca Pro email & PIM software,Barca/2.0.xxxx Barca Pro email & PIM software,BarcaPro/1.4.xxxx Barrahome crawler,BarraHomeCrawler (albertof@barrahome.org) Unknown user agent from Microsoft,bCentral Billing Post-Process Business.com robot,bdcindexer_2.6.2 (research@bdc) Brandimensions Brand Protection robot,BDFetch Bdncentral Sitesearch robot,BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686) beammachine web directory (Germany) link checking,BeamMachine/0.5 (dead link remover of www.beammachine.net) Beauty robot for Cosmoty - German beauty and wellness search,beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) A Passion for Jazz music related search robot,BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) LinkcheckerBeepware (site is down) web directory link checking,BeebwareDirectory/v0.01 Big Brother link checking tool,Big Brother (http://pauillac.inria.fr/~fpottier/) GoonGee.com link popularity checking,Big Fish v1.0 BB4 network monitoring,BigBrother/1.6e BigClique Search robot,BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com) Biglotron search (France) robot,BIGLOTRON (Beta 2;GNU/Linux) Bigsearch.ca search robot,Bigsearch.ca/Nutch-x.x-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com) Bilbo - web frontend for the Nessus Security Scanner,Bilbo/2.3b-UNIX Bilgi.com (Beta) search robot - Turkey,BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) Bilgi.com (Beta) search robot - Turkey,BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) Carnegie Mellon School robot/link checking ?,billbot wjj@cs.cmu.edu Bitacle Blog Search Archive robot,Bitacle bot/1.1 Bitacle Blog Search Archive robot,Bitacle Robot (V:1.0;) (http://www.bitacle.com) Biyubi Navigator - Mexican browser for Fenix OS,Biyubi/x.x (Sistema Fenix; G11; Familia Toledo; es-mx) Blackberry Wireless Internet browser via Google WAP Proxy,BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0) FS Consulting (was www.fsconsult.net) Black Widow web crawler,BlackWidow BlackWidow web site scanner / downloading tool,BlackWidow Blaiz Enterprises RawGrunt search,Blaiz-Bee/1.0 (+http://www.blaiz.net) Blaiz Enterprises RawGrunt search,Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com) Blaiz Enterprises RawGrunt search,Blaiz-Bee/2.00.xxxx (+http://www.blaiz.net) Blitzsuche Germany robot,BlitzBOT@tricus.net Blitzsuche Germany robot,BlitzBOT@tricus.net (Mozilla compatible) BlockNote web page editor,BlockNote.Net blogdex robot from MIT.edu,BlogBot/1.x BlogBridge RSS reader,BlogBridge 2.13 (http://www.blogbridge.com/) Bloglines article search,Bloglines Title Fetch/1.0 (http://www.bloglines.com) Bloglines graphics crawler,Bloglines-Images/0.1 (http://www.bloglines.com) Bloglines news crawler,Bloglines/3.1 (http://www.bloglines.com) FeedMap / BlogMap geo coding service,BlogMap (http://www.feedmap.net) Intelliseek's BlogPulse blog search,Blogpulse (info@blogpulse.com) Intelliseek's BlogPulse blog search,BlogPulseLive (support@blogpulse.com) IceRocket Web search robot,BlogSearch/1.x +http://www.icerocket.com/ Art of Computing blog search project,blogsearchbot-pumpkin-3 BlogsNow realtime link tracker robot,"BlogsNowBot, V 2.01 (+http://www.blogsnow.com/)" BlogVipe news and Blog crawler (Netherlands),BlogVibeBot-v1.1 (spider@blogvibe.nl) blogWatcher robot from Okumura Group Tokyo (131.112.182.xxx),blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) IceRocket Web search robot,BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com) IceRocket Web search robot,BlogzIce/1.0 +http://www.icerocket.com/ Blooby search (beta) robot,BloobyBot Balihoo - Search Engine for Advertising Media,Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) Bluefish HTML-editor for Linux,bluefish 0.6 HTML editor Part of ButtMan remote access tool,BMCLIENT Bookmark Express bookmark manager,BMLAUNCHER Bobby web accessibility desktop testing tool,Bobby/4.0.x RPT-HTTPClient/0.3-3E Boitho search (Norway) robot via 80.202.212.xx / 80.80.111.xx,boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html) Boitho search (Norway) robot via 80.202.212.xx / 80.80.111.xx,boitho.com-robot/1.x Boitho search (Norway) robot via 80.202.212.xx / 80.80.111.xx,boitho.com-robot/1.x (http://www.boitho.com/bot.html) Bookdog - Mac bookmark manager,Bookdog/x.x Bookmark Buddy - favorite bookmark manager ,Bookmark Buddy bookmark checker (http://www.bookmarkbuddy.net/) Favourites managing program,Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/] Favourites managing program,Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/] (Version 2.0beta) Bookmark Base bookmark manager,BookmarkBase(2/;http://bookmarkbase.com) cybercity.fr user robot / faked user agent ?,Bot mailto:craftbot@yahoo.com BD-Brandprotect copyright infringement crawler,BPImageWalker/2.0 (www.bdbrandprotect.com) BravoBrian bSTOP parental control,BravoBrian bstop.bravobrian.it Robot for BravoBrian bSTOP,BravoBrian SpiderEngine MarcoPolo BrightCloud web filtering for classifying websites,BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) Webarchive Project Bruinbot crawler,BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) Inktomi (Hotbot-Lycos NBCi) robot,BSDSeek/1.0 BravoBrian bSTOP parental control,BStop.BravoBrian.it Agent Detector BitTorrent Search Engine btbot robot,BTbot/0.x (+http://www.btbot.com/btbot.html) µTorrent BitTorrent Client,BTWebClient/180B(9704) BuildCMS crawler - market monitoring project of BuildCMS,BuildCMS crawler (http://www.buildcms.com/crawler) Bulkfeeds: RSS directory link checking,Bulkfeeds/r1752 (http://bulkfeeds.net/) BullsEye/Intelliseek robot,BullsEye Relevare Portal software robot,bumblebee@relevare.com Microsoft server information robot (see link),BunnySlippers BurstFind search crawler (64.34.172.xx),BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com) Buscaplus (Spain) robot,Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) Logitech Desktop Managers (LDM) Backweb (BW) update check,BW-C-2.0 Basic Web Hacking 3 fake user-agent from Hellbound Hackers challenges,bwh3_user_agent Amfibi Search robot,Cabot/Nutch-0.9 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) Amfibi Search robot,Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) Evolution integrated mail solution Camel TCP stream class,CamelHttpStream/1.0 Some user agent,Cancer Information and Support International; Cosmix project crawler (204.14.48.x / 38.113.234.xxx),carleson/1.0 Carnegie Mellon University WebBOT,Carnegie_Mellon_University_Research_WebBOT-->PLEASE READ-->http://www.andrew.cmu.edu/~brgordon/webbot/index.html http://www.andrew.cmu.edu/~brgordon/webbot/index.html Carnegie Mellon University WebBOT,Carnegie_Mellon_University_WebCrawler http://www.andrew.cmu.edu/~brgordon/webbot/index.html Catall.de search & web directory (Germany),Catall Spider UIUCs Cazoodle search based on MetaQuerier,CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) CommonCrawl Foundation search crawler,CCBot/1.0 (+http://www.commoncrawl.org/bot.html) Empyreum Ccubee (Czech) search engine solution,ccubee/x.x Timewe mobile browser (WAP) simulator (Japan),CDR/1.7.1 Simulator/0.7(+http://timewe.net) Profile/MIDP-1.0 Configuration/CLDC-1.0 Cisco Content Engine,CE-Preload Nordic semantic search engine,CentiverseBot Nordic semantic search engine,CentiverseBot - investigator Nordic semantic search engine,CentiverseBot/3.0 (http://www.centiverse-project.net) Floortransformed.com robot (link ckecking ??),Ceramic Tile Installation Guide (http://www.floorstransformed.com) CERN Line Mode Browser,CERN-LineMode/2.15 Cosmix project crawler (204.14.48.x / 38.113.234.xxx),cfetch/1.0 MaxOS X CoreFoundation CFNetwork API,CFNetwork/x.x cg-eye CGI checker,cg-eye interactive Charon Amiga download manager,Charon/1.x (Amiga) Chat Catcher blog monitoring robot,Chat Catcher/1.0 Checkbot link validation,Checkbot/1.xx LWP/5.xx Checklinks - Perl link checker,CheckLinks/1.x.x NTL user agent,CheckUrl CheckWeb link validation,CheckWeb Chilkat HTTP component user-agent,Chilkat/1.0.0 (+http://www.chilkatsoft.com/ChilkatHttpUA.asp) Unknown spam bot from telekom.com.my (218.111.83.xxx),China Local Browse 2.6 Chitika Inc. Blog advertising,Chitika ContentHit 1.0 Christcentral.com Christcrawler (was www.christcrawler.com),ChristCRAWLER 2.0 CHttpClient - C++ class using WinInet,CHttpClient by Open Text Corporation CipinetBot -Cipinet Search Engine Web Crawler,CipinetBot (http://www.cipinet.com/bot.html) Cityreview regional search (Germany) link checking,Cityreview Robot (+http://www.cityreview.org/crawler/) Commision Junction link checking spider,CJ Spider/ CJB Net anonymous socks proxy service (216.194.70.x),CJB.NET Proxy Claria (ex Gator) SearchScout robot (64.152.73.xx),ClariaBot/1.0 Claymont Search robot,Claymont.com Seznam Search (Czech Republic) robot,CloakDetect/0.9 (+http://fulltext.seznam.cz/) Clush clustered search robot,Clushbot/2.x (+http://www.clush.com/bot.html) Clush clustered search robot,Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html) Clush clustered search robot,Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html) Clush clustered search robot,Clushbot/3.xx-Hector (+http://www.clush.com/bot.html) Clush clustered search robot,Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html) COAST Webmaster - Web management and maintenance software,COAST WebMaster Pro/4.x.x.xx (Windows NT) Proposed Content-Based Image Tracking System (CoBITS) P2P crawler,CoBITSProbe Cocoa del.icio.us (social bookmarks manager) client for Mac OS X,Cocoal.icio.us/1.0 (v36) (Mac OS X; http://www.scifihifi.com/cocoalicious) Cogent Search Bot from Cogent Software Solutions for unknown purposes,Cogentbot/1.X (+http://www.cogentsoftwaresolutions.com/bot.html) Cold Fusion server used by various IPs i.e.: - NetWORLD web catalogue link checking,ColdFusion Cold Fusion server used by Bookmark Tracker - online favourites managment,ColdFusion (BookmarkTracker.com) WebCollage Syndicator graphics crawler/collector,collage.cgi/1.xx Combine harvesting & indexing robot,combine/0.0 Combine harvesting & indexing robot,Combine/2.0 http://combine.it.lth.se/ Combine harvesting & indexing robot,Combine/3 http://combine.it.lth.se/ Combine harvesting & indexing robot,Combine/x.0 cometrics Web Content Mining solution - Germany,"cometrics-bot, http://www.cometrics.de" Wildsoft Germany (closed) Internet client system user-agent (was www.oskarweb.de),Commerce Browser Center UCLA Complex Networks Groups Complex Network Analysis,complex_network_group/Nutch-0.9-dev (discovering the structure of the world-wide-web; http://cantor.ee.ucla.edu/~networks/crawl; nimakhaj@gmail.com) Hungarian Academy of Sciences data mining search,Computer_and_Automation_Research_Institute_Crawler crawler@ilab.sztaki.hu ComRite Chinese Search Engine for Oversea Web Sites (69.248.26.xx),Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) unknown,Contact Probably E-Mail harvesting robot - same as LMQueueBot,ContactBot/0.2 ContentSmartz e-mail harvesting tools,ContentSmartz Internet Explorer versions 4.x and 5 plugin content,contype Converas RetrievalWare Internet Spider (63.241.61.x),Convera Internet Spider V6.x Converas RetrievalWare Internet Spider (63.241.61.x),ConveraCrawler/0.2 Converas RetrievalWare Internet Spider (63.241.61.x),ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl) Converas RetrievalWare Internet Spider (63.241.61.x),ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl) Suchmaschine21 (Germany) robot,CoolBot Cooliris photo and video browser,Cooliris/1.5 CFNetwork/459 Darwin/10.0.0d3 Coral Content Distribution Network,CoralWebPrx/0.1.1x (See http://coralcdn.org/) Xyleme SA France robot,cosmos/0.8_(robot@xyleme.com) Xyleme SA France robot,cosmos/0.9_(robot@xyleme.com) Coteo.com - local French directory link checking,CoteoNutchCrawler/Nutch-0.9 (info [at] coteo [dot] com) Cougarsearch.com robot,CougarSearch/0.x (+http://www.cougarsearch.com/faq.shtml) Covac Arachnid Web Crawler,Covac TexAs Arachbot CoverScout for iTunes - CD cover search tool,"CoverScout%203/3.0.1 CFNetwork/339.5 Darwin/9.5.0 (i386) (iMac5,1)" Naver Japan / Korea robot,Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Naver Japan / Korea robot,Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Converas RetrievalWare Internet Spider,CrawlConvera0.1 (CrawlConvera@yahoo.com) unknown robot via Level3.net,Crawler Cometsystems Comet Search robot via Findwhat (now Miva),Crawler (cometsearch@cometsystems.com) Crawler.de / Abacho robot,Crawler admin@crawler.de Crawler.de / Abacho robot,Crawler V 0.2.x admin@crawler.de Alexa crawler,crawler@alexa.com Pinpoint WAP search robot,CrawlerBoy Pinpoint.com Crawlly Beta search - Germany (72.232.194.2xx),Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com) Creatice Commons using Nutch open source robot,CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) Cricket A100 cell phone browser,Cricket-A100/1.0 UP.Browser/6.3.0.7 (GUI) MMP/2.0 Croccrawler robot,CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) Web mining project from CSCI 659 (computer science course) at Indiana Univerity,csci_b659/0.13 CSE HTML Validator for Windows,CSE HTML Validator Professional (http://www.htmlvalidator.com/) Cuam - IE based browser,Cuam Ver0.050bx Cuasar (Spain) music / ringtone search spider,Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ Curl file transferring tool,curl/7.10.x (i386-redhat-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4 Curl file transferring tool,curl/7.7.x (i386--freebsd4.3) libcurl 7.7.x (SSL 0.9.6) (ipv6 enabled) Curl file transferring tool,curl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6) Curl file transferring tool,curl/7.9.x (win32) libcurl 7.9.x CurryGuide UK link check robot,CurryGuide SiteScan 1.1 Custo web site spidering tool (link checking),Custo x.x (www.netwu.com) Bisnisseek (was www.bisnisseek.com) robot,Custom Spider www.bisnisseek.com /1.0 Cyberdog Mac Browser (was www.cyberdog.org),Cyberdog/2.0 (Macintosh; 68k) CyberPatrol LLC robot for web filtering software,CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp) CyberSpyder Link Test software,CyberSpyder Link Test/2.1.12 (admin@mspennyworth.com) Cydral image & site search spider,CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com) Cydral image & site search spider,CydralSpider/3.0 (Cydral Image Search; http://www.cydral.com) Downloadaccelerator download manager,DA 3.5 (www.lidan.com) Downloadaccelerator download manager,DA 4.0 Downloadaccelerator download manager,DA 4.0 (www.downloadaccelerator.com) Downloadaccelerator download manager,DA 5.0 Downloadaccelerator download manager,DA 7.0 Download Accelerator Plus download manager,DAP x.x PowerTCP ActiveX control tool,Dart Communications PowerTCP Unknown bot from Kornet Korea (218.149.129.xxx) scans for Perl Awstats,DataCha0s/2.0 Unknown UA looking for Awstats Perl components,DataCha0s/2.0 INFOMINE/iVia Scholary Internet Resource Collections robot,DataFountains/DMOZ Downloader INFOMINE/iVia Scholary Internet Resource Collections robot,DataFountains/Dmoz Downloader (http://ivia.ucr.edu/useragents.shtml) INFOMINE/iVia Scholary Internet Resource Collections robot,DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents.shtml) DataparkSearch open source search engine,DataparkSearch/4.47 (+http://dataparksearch.org/bot) DataparkSearch open source search engine,DataparkSearch/4.xx (http://www.dataparksearch.org/) DataSpear Directory robot (24.109.29.xx),DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.com) DataSpear Directory robot (24.109.29.xx),DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html; dssb@dataspear.com) Sicher-durchs-Netz German security related search (62.75.220.xxx),DatenBot( http://www.sicher-durchs-netz.de/bot.html) Wholeweb robot,DaviesBot/1.7 (www.wholeweb.net) Daypop blog - weblog - online mag search spider,daypopbot/0.x dbDig search engine,dbDig(http://www.prairielandconsulting.com) Some site scanning tool via diff. IPs i.e.: - wanweb.net (208.6.163.xxx) - cox.net (68.4.xxx.xxx),DBrowse 1.4b Some site scanning tool via diff. IPs i.e.: - pacbell.net (67.112.xxx.xxx),DBrowse 1.4d DC-Sakura download manager,DC-Sakura/x.xx unknown divine/Openmarket.com robot,dCSbot/1.1 some (website) downloading tool,DDD Unknown user agent,dds explorer v1.0 beta Searchengine.com (Germany) submission checking / robot (84.73.57.xx),de.searchengine.comBot 1.2 (http://de.searchengine.com/spider) DLC Perl HTTP link checking,DeadLinkCheck/0.4.0 libwww-perl/5.xx Sootle Web Directory deep link checker,Deep Link Calculator v1.0 deepak-USC/ISI robot from USC/Information Science Institute,deepak-USC/ISI Deepindex robot,DeepIndex Zetbot search Belgium (213.41.128.xx) using Deepindex robot,DeepIndex ( http://www.zetbot.com ) Deepindex robot,DeepIndex (www.en.deepindex.com) Deepindex robot (via Paragon.net Canada),DeepIndexer.ca del.icio.us picture robot for thumbnail preview via Yahoo,del.icio.us-thumbnails/1.0 Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko) DeleGate application level gateway / proxy server,DeleGate/9.0.5-fix1 Some site scanning tool from 217.34.59.xxx (btopenworld.com),Demo Bot DOT 16b Some site scanning tool from 68.154.96.xx (bellsouth.net),Demo Bot Z 16b Denmex Websearch robot/link checking,Denmex websearch (http://search.denmex.com) DataBecker Bilder Sauger (discontinued) web graphics downloader,Der große BilderSauger 2.00u Searchspider.com robot (72.245.225.xxx),dev-spider2.searchpsider.com/1.3b DevComponents HTMLDocument Class Library for Visual Studio.net,DevComponents.com HtmlDocument Object DiaGem Japan web crawler,DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) Claria (ex Gator) SearchScout robot (64.152.73.xx),Diamond/x.0 Claria (ex Gator) SearchScout robot (64.152.73.xx),DiamondBot Diggit! robot,Digger/1.0 JDK/1.3.0rc3 OpenPortal4U robot,DigOut4U Findsame.com (site is offline) / Digital-Integrity robot,DIIbot/1.2 Dillo Web Browser,Dillo/0.8.5-i18n-misc Dillo Web Browser,Dillo/0.x.x disastrous - a del.icio.us link checker based on Python,disastrous/1.0.5 (running with Python 2.5.1; http://www.bortzmeyer.org/disastrous.html; archangel77@del.icio.us) DISCo Pump offline browser / website ripper,DISCo Pump x.x Unkown robot from Discovery Engine Corp.,disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) Unkown robot from Discovery Engine Corp.,disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) Ditto picture search robot,DittoSpyder some download agent,dlman Naver Japan / Korea robot,dloader(NaverRobot)/1.0 DNS Right - Online DNS tools,DNSRight.com WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.com NTT DoCoMo (Japan) robot,DoCoMo/1.0/Nxxxi/c10 NTT DoCoMo (Japan) robot,DoCoMo/1.0/Nxxxi/c10/TB Google (216.239.39.x) proxy server,DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0) NTT DoCoMo (Japan) robot,DoCoMo/2.0 P900iV(c100;TB;W24H11) NTT DoCoMo (Japan) proxy server (210.136.161.1xx),"DoCoMo/2.0 SH901iS(c100;TB;W24H12),gzip(gfe) (via translate.google.com)" Yahoo Search Japan robot (203.216.197.xxx),DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) Yahoo Search Japan robot (203.216.197.xxx),DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) DocZilla - Mozilla-based SGML/XML/HTML- browser,DocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0) Gecko/20020804 unknown robot from AGMLAB Information Technologies (Information retrieval system ?),dodgebot/experimental Donut P - Japanese IE based browser,DonutP; Windows98SE Unknown robot from douban search (China) - maybe image crawling,Doubanbot/1.0 (bot@douban.com http://www.douban.com) Download Demon/Netzip download manager,Download Demon/3.x.x.x Download Druid IE plugin download manager,Download Druid 2.x Download Express download manager,Download Express 1.0 Download Master download manager,Download Master Download Ninja download manager (Japan),Download Ninja 3.0 Download Wonder download manager,Download Wonder Download-Tipp Germany link checking,Download-Tipp Linkcheck (http://download-tipp.de/) download.exe .NET based downloading tool,Download.exe(1.1) (+http://www.sql-und-xml.de/freeware-tools/) Download Direct download manager,DownloadDirect.1.0 Dr.WEB online virus scanner,Dr.Web (R) online scanner: http://online.drweb.com/ Dragonfly CMS - Open Source content management system,Dragonfly File Reader Drecom Japan (210.233.67.xxx) - Blog search ??,Drecombot/1.0 (http://career.drecom.jp/bot.html) Drupal - open source content management platform,Drupal (+http://drupal.org/) Some site scanning tool via diff. IPs i.e.: - cox.net (68.5.xxx.xxx) - pacbell.net (64.16x.xxx.xxx),DSurf15a 01 Some site scanning tool via diff. IPs i.e.: - cox.net (68.4.xxx.xxx),DSurf15a 71 Some site scanning tool via diff. IPs i.e.: - verizon.net (4.47.xxx.xxx),DSurf15a 81 Some site scanning tool via diff. IPs i.e.: - eastlink.ca (24.222.xxx.xxx) - cogeco.net (216.221.8x.xxx),DSurf15a VA DTAAgent Java object for data collecting,DTAAgent dt Search Spider software,dtSearchSpider Fourelle Venturi proxy server,Dual Proxy Duck Duck Go search crawler,DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) DumbFind.com robot,Dumbot(version 0.1 beta - dumbfind.com) DumbFind.com robot,Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html) DumbFind.com robot,Dumbot(version 0.1 beta) Vigiltech e-Sense user research robot (website is offline),e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html) e-Society Project (Japan) crawler (133.9.238.xx),e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) Global Opto's eApollo Flash based search engine - Taiwan,eApolloBot/2.0 (compatible; heritrix/2.0.0-SNAPSHOT-20071024.170148 +http://www.eapollo-opto.com) Earthcom (Czech Republic) search robot (194.108.39.xx),EARTHCOM.info/1.x [www.earthcom.info] Earthcom (Czech Republic) search robot (194.108.39.xx),EARTHCOM.info/1.xbeta [www.earthcom.info] Keywen Encyclopedia Bot,EasyDL/3.xx Keywen Encyclopedia Bot,EasyDL/3.xx http://keywen.com/Encyclopedia/Bot Some site scanning tool via diff. IPs i.e.: - swbell.net (65.66.xxx.xxx),EBrowse 1.4b eCatch (now Wysigot) offline browser,eCatch/3.0 Echo.fr robot,EchO!/2.0 Some site scanning tool via diff. IPs i.e.: - cox.net (68.4.xxx.xxx),Educate Search VxB Xdefine text search engine robot - based on Egothor open source crawler,egothor/3.0a (+http://www.xdefine.org/robot.html) Egoto Search robot,EgotoBot/4.8 (+http://www.egoto.com/about.htm) eJupiter searcg robot (206.191.49.xx),ejupiter.com TimelyWeb web page monitoring tool,EldoS TimelyWeb/3.x Elftales crawler for uChoose theme based search (Germany),elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) DAUMOA - RSS search robot of Daum,"ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html)" ELinks text mode browser,ELinks (0.x.x; Linux 2.4.20 i586; 132x60) ELinks text mode browser,ELinks/0.x.x (textmode; NetBSD 1.6.2 sparc; 132x43) Sonic E-mail collector,EmailSiphon EmailSpider E-mail harvesting software,EmailSpider Trellian EMailWolf E-mail collector,EmailWolf 1.00 EmeraldShield spam and web filtration services,EmeraldShield.com WebBot EmeraldShield spam and web filtration services,EmeraldShield.com WebBot (http://www.emeraldshield.com/webbot.aspx) Empas search Korea robot (220.95.22x.xxx),EMPAS_ROBOT Enabot - Enaball semantic search project crawler,EnaBot/1.x (http://www.enaball.com/crawler.html) endo - Mac news site and blog aggregator,endo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo) Enfish Personal search tool,Enfish Tracker Enterprise Search engine software (64.202.165.xxx),Enterprise_Search/1.0 Enterprise Search engine software (64.202.165.xxx),Enterprise_Search/1.0.xxx Enterprise Search engine software (64.202.165.xxx),Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp) Envolk Web Search robot,envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php) Envolk Web Search robot,envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html) EroCrawler adult search robot,EroCrawler Enterprise Search engine software (64.202.165.xxx),ES.NET_Crawler/2.0 (http://search.innerprise.net/) ExactSEEK (Jayde Online) robot,eseek-larbin_2.6.2 (crawler@exactseek.com) ESI Smart-Spider toolkit,ESISmartSpider e-Style ISP search (Russia) robot (217.174.103.xxx),eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) Some site scanning tool via diff. IPs,ESurf15a 15 Eurip.com - European Internet Portal robot,EuripBot/0.x (+http://www.eurip.com) GetFile Eurip.com - European Internet Portal robot,EuripBot/0.x (+http://www.eurip.com) GetRobots Eurip.com - European Internet Portal robot,EuripBot/0.x (+http://www.eurip.com) PreCheck Ayell Euronet business directory robot,Eurobot/1.0 (http://www.ayell.eu) Evaal Search Engine robot,EvaalSE - bot@evaal.com Eventax event search (Germany),eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de) Vulcan Inc. Everest crawler (in development),Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/crawlerhelp) Vulcan Inc. Everest crawler (in development),Everest-Vulcan Inc./0.1 (R&D project; http://everest.vulcan.com/crawlerhelp) Exalead Websearch image crawler (193.47.80.xx),Exabot-Images/1.0 Exalead (France) search robot (193.47.80.xx),Exabot-Test/1.0 Exalead (France) search robot (193.47.80.xx),Exabot/2.0 Exalead (France) search robot (193.47.80.xx),Exabot/3.0 eXact Search Bar for IE,ExactSearch ExactSEEK (Jayde Online) robot,ExactSeek Crawler/0.1 ExactSEEK (Jayde Online) robot,exactseek-crawler-2.63 (crawler@exactseek.com) ExactSEEK (Jayde Online) robot,exactseek-pagereaper-2.63 (crawler@exactseek.com) ExactSEEK (Jayde Online) robot (69.9.181.1xx),exactseek.com Exalead (France) search robot (193.47.80.xx),Exalead NG/MimeLive Client (convert/http/0.120) Excalibur (now Convera) spider software,Excalibur Internet Spider V6.5.4 Execrawl software search using nutch,Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com) exooba crawler for exooba search pre-alpha development,exooba crawler/exooba crawler (crawler for exooba.com; http://www.exooba.com/; info at exooba dot com) exooba crawler for exooba search pre-alpha development,exooba/exooba crawler (exooba; exooba) Mirago UK Robot,ExperimentalHenrytheMiragoRobot Expired Domain Sleuth domain name tool,Expired Domain Sleuth Express Web Pictures image browser,Express WebPictures (www.express-soft.com) Extractor Pro e-mail collector,ExtractorPro Exisoftware image grabber and downloading tool,Extreme Picture Finder Download-Tipp Germany robot,EyeCatcher (Download-tipp.de)/1.0 Factbites search robot,Factbot 1.09 (see http://www.factbites.com/webmasters.php) Factbites search robot,factbot : http://www.factbites.com/robots FaEdit Professional - Japanese bookmark manager,FaEdit/2.0.x FairAd user,FairAd Client Safe-t.net web filtering service,FANGCrawl/0.01 Drew Curtis' FARK.com link checking,FARK.com link verifier Fast/Alltheweb crawler (66.151.181.xx),Fast Crawler Gold Edition Fast/Alltheweb crawler (66.151.181.xx),FAST Enterprise Crawler 6 (Experimental) Fast Enterprise Crawler (66.151.181.xx) for Scirus scienctific information search,FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/ Fast/Alltheweb crawler (66.151.181.xx),FAST Enterprise Crawler 6 used by Cobra Development (admin@fastsearch.com) Comperio Web Miner based on Fast ESP,FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no) Fast/Alltheweb crawler (66.151.181.xx),FAST Enterprise Crawler 6 used by FAST (FAST) Pages Jaunes business search (France) robot using Fast Enterprise Crawler,FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr) Fast/Alltheweb crawler for Sensis.com.au Australian search (66.151.181.xx),FAST Enterprise Crawler 6 used by Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au) SPH Search - Singapore related search using Fast crawler,FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg) FAST Enterprise Crawler used by WWU (University of Muenster - Germany),FAST Enterprise Crawler 6 used by WWU (wardi@uni-muenster.de) Fast/Alltheweb crawler (66.151.181.xx),FAST Enterprise Crawler/6 (www.fastsearch.com) Fast/Alltheweb crawler (66.151.181.xx),FAST Enterprise Crawler/6.4 (helpdesk at fast.no) Fast/Alltheweb crawler (66.151.181.xx),FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0) Fast/Alltheweb crawler (66.151.181.xx),FAST MetaWeb Crawler (helpdesk at fastsearch dot com) Fast/Alltheweb crawler (66.151.181.xx),Fast PartnerSite Crawler Fast/Alltheweb multimedia crawler,FAST-WebCrawler/2.2.10 (Multimedia Search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) Fast/Alltheweb crawler,FAST-WebCrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) Fast/Alltheweb crawler,FAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no Fast/Alltheweb crawler,FAST-WebCrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no Fast/Alltheweb crawler,FAST-WebCrawler/3.2 test Fast/Alltheweb crawler,FAST-WebCrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) Fast/Alltheweb crawler,FAST-WebCrawler/3.4/Nirvana (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) Fast/Alltheweb crawler,FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) Fast/Alltheweb crawler,FAST-WebCrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c=faqs/crawler) Fast/Alltheweb crawler,FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Fast/Alltheweb crawler,FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) Fast/Alltheweb crawler,FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Fast/Alltheweb crawler,FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp) Fast/Alltheweb crawler,FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Fast/Alltheweb crawler,FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Fast/Alltheweb multimedia crawler,FAST-WebCrawler/3.x Multimedia Fast/Alltheweb multimedia crawler,FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no) Fastbot search Germany crawler (80.252.104.1xx),fastbot crawler beta 2.0 (+http://www.fastbot.de) Ay-Up geo sync search robot,FastBug http://www.ay-up.com Fast/Alltheweb crawler (66.151.181.xx),FastCrawler 3.0.1 (crawler@1klik.dk) Fast/Alltheweb crawler used by SuperPages.com,FastSearch Web Crawler for Verizon SuperPages (kevin.watters@fastsearch.com) Favcollector Favicon collecting robot,Favcollector/2.0 (info@favcollector.com http://www.favcollector.com/) FavIconizer - IE favorites icons refreshing tool,FavIconizer favo.eu (Germany) search robot,favo.eu crawler/0.6 (http://www.favo.eu) ZD's FavOrg favourites managing program,FavOrg Campulka.net Favorites checking tool,Favorites Checking (http://campulka.net) Favorites Sweeper bookmark checker,Favorites Sweeper v.2.03 FaXo Search robot (69.152.89.xx),Faxobot/1.0 Free Download Manager (FDM) download accelerator ,FDM 1.x Free Download Manager (FDM) download accelerator ,FDM 2.x RSS Feed Seeker bot (68.225.95.2xx),Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php) Feed24 news feed and blog search,Feed24.com Feed::Find - Syndication feed (RSS/Atom) auto-discovery,Feed::Find/0.0x Feedable beta web based RSS service,Feedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1) Unknown robot from the University of Tokyo (157.82.157.xx),FeedChecker/0.01 FeedDemon RSS reader,FeedDemon/2.7 (http://www.newsgator.com/; Microsoft Windows XP) Google news feed feetcher for iGoogle gadgets,Feedfetcher-Google-iGoogleGadgets; (+http://www.google.com/feedfetcher.html) Google Feedfetcher - RSS and Atom feed crawler,Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) FeedForAll RSS feed robot,FeedForAll rss2html.php v2 FeedHub news feed personalization engine powered by mSpoke,FeedHub FeedDiscovery/1.0 (http://www.feedhub.com) FeedHub news feed personalization engine powered by mSpoke,FeedHub MetaDataFetcher/1.0 (http://www.feedhub.com) Feedjit news feed service favicon crawler,Feedjit Favicon Crawler 1.0 Newsbrain Feedreader3,Feedreader 3.xx (Powered by Newsbrain) FeedShow online RSS feed reader,Feedshow/x.0 (http://www.feedshow.com; 1 subscriber) FeedShow online RSS feed reader,FeedshowOnline (http://www.feedshow.com) "FeedZcollector - Feed (RSS, ATOM and RDF) capturing software",FeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollector MixCat robot,Felix - Mixcat Crawler (+http://mixcat.com) FreeBSD download tool,fetch libfetch/2.0 Frequent Finders spider via Sitefusion.com,FFC Trap Door Spider Filangy search and bookmark service,Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com) Filangy search and bookmark service,Filangy/1.0x (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com) Filangy search and bookmark service,Filangy/1.0x (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com) File Boost Network link checking,fileboost.net/1.0 (+http://www.fileboost.net) FileHound download manager,FileHound x.x filtrbox media content (news) monitoring,Filtrbox/1.0 Find An ISP robot,FindAnISP.com_ISP_Finder_v99a Yelo.no business search (Norway) via Findexa,Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) NextLinks - German vocabulary and hyperlink search,findlinks/x.xxx (+http://wortschatz.uni-leipzig.de/findlinks/) Finesearch robot,FineBot Finjan Vital Security Web Appliance security solution,Finjan-prefetch Fireball.de robot,Firefly/1.0 Fireball.de robot,Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) Unknown robot from Czech Technical University Prague (147.32.141.xx),Firefox (kastaneta03@hotmail.com) Unknown robot from Czech Technical University Prague (147.32.141.xx),Firefox_1.0.6 (kasparek@naparek.cz) AT&T/Fast Search robot for FirstGov (U.S.Government) portal,FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov Firstsfind Germany robot / link checking,firstsbot Flaptor information retrieval solutions robot,Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor period com) JetCar/FlashGet download manager,FlashGet FlatArts Favorites Icon Tool,FLATARTS_FAVICO Flexum.ru search service,Flexum spider Flexum.ru search service,Flexum/2.0 DivX.com Movie Find robot,FlickBot 2.0 RPT-HTTPClient/0.3-3 Metacarta / Cogent robot,flunky unknown,fly/6.01 libwww/4.0D FLY Index Metasearch link checking,flyindex.net 1.0/http://www.flyindex.net Fnoole news crawler,FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html) IBM's Almaden Research robot,FocusedSampler/1.0 folkd.com social search robot,Folkd.com Spider/0.1 beta 1 (www.folkd.com) FollowSite robot - website monitoring,FollowSite Bot ( http://www.followsite.com/bot.html ) FollowSite robot - website monitoring,FollowSite.com ( http://www.followsite.com/b.html ) Fooky search Scorpionbots robot (65.12.170.xxx),Fooky.com/ScorpionBot/ScoutOut; http://www.fooky.com/scorpionbots Neomo Search (Germany) robot (85.10.197.1xx),Francis/1.0 (francis@neomo.de http://www.neomo.de/) Some spam bot,Franklin Locator 1.8 Free Downloads shareware directory link checking,free-downloads.net download-link validator /0.1 FreeFind.com robot,FreeFind.com-SiteSearchEngine/1.0 (http://freefind.com; spiderinfo@freefind.com) Frelics backlink checking bot (beta),Frelicbot/1.0 +http://www.frelic.com/ Fresh Download download manager,FreshDownload/x.xx FreshNotes - music related artist search (72.3.225.xx),FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com Some site scanning tool via diff. IPs,FSurf15a 01 Find the Best search robot,FTB-Bot http://www.findthebest.co.uk/ Some site scanning tool from diff. IPs i.e.: - 66.28.240.xx (cogentco.com) - 68.5.174.xx (cox.net),Full Web Bot 0416B Some site scanning tool i.e. from - 68.154.96.xx (bellsouth.net),Full Web Bot 0516B Some site scanning tool from 66.255.6.xxx (uslec.com),Full Web Bot 2816B FuseBulb search,FuseBulb.Com FyberSearch FyberSpider robot,FyberSpider (+http://www.fybersearch.com/fyberspider.php) Some user from bbnplanet.net (4.63.218.2xx) using an Innerprise robot tool,Gagglebot Seed Search robot,GAIS Robot/1.0B2 Gaislab Taiwan robot,Gaisbot/3.0 (indexer@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php) Gaislab Taiwan robot,Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Galaxy robot (63.121.41.xxx),GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html) Gallent Search directory (UK),Gallent Search Spider v1.4 Robot 2 (http://robot.GallentSearch.com) Gamekit game search engine - Germany,gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) GameSpyHTTP/1.0,Gamespy_Arcade GammaWare GammaSpider,GammaSpider/1.0 nttrd.com / Infobee.ne.jp robot,gazz/x.x (gazz@nttrd.com) Geckobot user robot,geckobot Google Mobile Search crawler,Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) Unknown robot from Carnegie Mellon University (128.2.211.xxx),generic_crawler/01.0217/ Lunascape Genesis browser,GenesisBrowser (HTTP 1.1; 0.9; XP SP2; .NET CLR 2.0.50727) GenieKnows.com search,genieBot (http://64.5.245.11/faq/faq.html) GenieKnows.com search,geniebot wgao@genieknows.com Unknown robot from wavepath.com (65.254.33.1xx),GeoBot/1.0 Geona Search robot / link checking,GeonaBot 1.x; http://www.geona.com/ GeoURL ICBM Address Server - a location-to-URL reverse directory,geourl/2.0b2 GeoURL ICBM Address Server - a location-to-URL reverse directory,GeoURLBot 1.0 (http://geourl.org) Getbot web downloading tool / site grabber,GetBot GetRight download manager,GetRight/3.x.x GetRight download manager,GetRight/4.5xx GetRight download manager,GetRight/4.x GetRight download manager,GetRight/4.x[a-e] GetRight download manager,GetRight/6.1 (Pro) GetRight download manager,GetRightPro/6.0beta2 GetWeb - web page to email service,GetWeb/0.1 libwww-perl/5.16 Sixxs Ghost Route Hunter,GhostRouteHunter/20021130 (https://www.sixxs.net/tools/grh/; info@sixxs.net) GigaBaz Brainbot (Germany) robot,gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/) Gigablast robot (64.62.168.xx),Gigabot/2.0 (gigablast.com) Gigablast robot,Gigabot/2.0/gigablast.com/spider.html Gigablast robot,Gigabot/2.0; http://www.gigablast.com/spider.html Gigablast robot,Gigabot/2.0att Gigablast robot,Gigabot/3.0 (http://www.gigablast.com/spider.html) Gigablast robot (64.62.168.xx),Gigabot/x.0 Gigablast robot (64.62.168.xx),GigabotSiteSearch/2.0 (sitesearch.gigablast.com) www.gnod.net spider,GNODSPIDER (www.gnod.net) Go!Zilla download manager,Go!Zilla 3.x (www.gozilla.com) Go!Zilla download manager,Go!Zilla/4.x.x.xx GotIt web accelerator (discontinued),Go-Ahead-Got-It/1.1 GoGuides.Org (195.226.137.xx) robot,Goblin/0.9 (http://www.goguides.org/) GoGuides.Org (195.226.137.xx) robot,Goblin/0.9.x (http://www.goguides.org/goblin-info.html) GoForIt Search robot,GoForIt.com GoForIt Search robot,GOFORITBOT ( http://www.goforit.com/about/ ) GoGuides.org directory & search link checking,GoGuides.Org Link Check GoldenFeed.com - RSS search engine,GoldenFeed Spider 1.0 (http://www.goldenfeed.com) Invention Machines Goldfire Server,Goldfire Server suchen.de German local search robot,gonzo1[P] +http://www.suchen.de/popups/faq.jsp suchen.de German local search robot,gonzo2[P] +http://www.suchen.de/faq.html Some private robot (Wanadoo.fr client),Goofer/0.2 Google instant messenger,Google Talk Packard Bell Net user robot (*not* Google),googlebot (larbin2.6.0@unspecified.mail) Google image crawler (66.249.72.xxx),Googlebot-Image/1.0 Google image crawler (66.249.72.xxx),Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html) Google robot 66.249.64.XXX,Googlebot/2.1 ( http://www.google.com/bot.html) Google robot 66.249.64.XXX,Googlebot/2.1 ( http://www.googlebot.com/bot.html) Google robot 66.249.64.XXX,Googlebot/Test ( http://www.googlebot.com/bot.html) ShareThis social networking service via Amazon Web Services,Gordon's Spider/Nutch-0.9 (http://www.sharethis.com; gordon@sharethis.com) Grapeshot web search system API,GrapeFX/0.3 libwww/5.4.0 Flatland Industries vertical search solution,great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com) GreatNews 1.0 Beta RSS reader,GreatNews/1.0 GreenBrowser - IE based browser (China),GreenBrowser search gridwell favicon display,gridwell (http://search.gridwell.com) Grigor Search bot,GrigorBot 0.8 (http://www.grigor.biz/bot.html) Australasian Legal Information Institute (AustLII) robot,Gromit/1.0 Grub open source crawler,grub crawler(http://www.grub.org) Grub open source crawler,grub-client Google Search Appliance robot (216.239.xx.xx),gsa-crawler (Enterprise; GID-01422; jplastiras@google.com) Google Search Appliance robot (216.239.xx.xx),gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com) Google Enterprise Search Appliance used by Enhesa (212.35.100.1xx),gsa-crawler (Enterprise; GIX-02057; dm@enhesa.com) Google Enterprise Search Appliance used by IBM (129.41.20.1xx),gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com) Google Search Appliance robot (216.239.xx.xx),gsa-crawler (Enterprise; GIX-0xxxx; enterprise-training@google.com) GSiteCrawler - Google sitemap generator for Windows,GSiteCrawler/v1.xx rev. xxx (http://gsitecrawler.com/) Guestbook spamming tool,Guestbook Auto Submitter Northernlight robot,Gulliver/1.3 Yuntis Collaborative Web Resource Categorization and Ranking Project robot,Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) Gungho - Extensible web crawler written in Perl by Google Code,Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) guruji : the Indian search engine robot,GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html) guruji : the Indian search engine picture crawler,GurujiImageBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) Openwave Mobile Browser on Haier T10C mobile,Haier-T10C/1.0 iPanel/2.0 WAP2.0 (compatible; UP.Browser/6.2.2.4; UPG1; UP/4.0; Embedded) Happy Fun Search robot,HappyFunBot/1.1 Harvest-NG web crawler used by search.yahoo.com,Harvest-NG/1.0.2 Haste - web mapping and monitoring system,Haste/0.12 (HOME: http://haste.kytoon.com/) Hatena::Antenna Japan robot,Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) Hatena Japan proxy for handheld/mobile clients,Hatena Mobile Gateway/1.0 Hatena Japan robot,Hatena Pagetitle Agent/1.0 Hatena Japan RSS feed robot,Hatena RSS/0.3 (http://r.hatena.ne.jp) Hatena::Diary (Japan) web page screenshot robot,HatenaScreenshot/1.0 (checker) hbtronix.spider - Domain name spider (Germany),hbtronix.spider.2 -- http://hbtronix.de/spider.php Mirago Germany robot,HeinrichderMiragoRobot Mirago Germany robot,HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) Helix - The SiteSearch (Canada) web crawler,Helix/1.x ( http://www.sitesearch.ca/helix/) Mirago France robot,HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) Mirago search (UK) robot,HenrytheMiragoRobot Mirago search (UK) robot,HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) O'Reilly's Perl LWP example client program from Web Client Programming with Perl,hgrepurl/1.0 University of Kassel Germany CsCrawler using the HTTPClient library,Hi! I'm CsCrawler my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3 HiDownload download manager,HiDownload Hippias robot,Hippias/0.9 Beta Pilot Hitlist web analytics solution,HitList Hitwise spider,Hitwise Spider v1.0 http://www.hitwise.com diff. IPs / unknown services,HLoader Morfeo / Centrum Search (Czech Republic) robot from 65.102.46.xxx,holmes/3.11 (http://morfeo.centrum.cz/bot) Onet.pl (Poland) search robot,holmes/3.9 (onet.pl) Onet.pl (Poland) search robot,holmes/3.xx (OnetSzukaj/5.0; +http://szukaj.onet.pl) Morfeo / Centrum Search (Czech Republic) robot from 65.102.46.xxx,holmes/x.x Holes search robot (Georgia),HolmesBot (http://holmes.ge) HomePageSearch robot,HomePageSearch(hpsearch.uni-trier.de) Homerweb search robot,Homerbot: www.homerweb.com Honda-Search.com - Honda cars related search robot,Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda-search.com) HooWWer - Next Generation Information Retrieval robot ,HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-infohiit.fi) HooWWer - Next Generation Information Retrieval robot (128.214.112.xx),HooWWWer/2.1.x ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-infohiit.fi) HotJava browser plus HTML Component 1.1.x,HotJava/1.0.1/JRE1.1.x Hotzuno - Japanese BBS reader client,Hotzonu/x.0 Unknown robot from HP Labs,HPL/Nutch-0.9 - COMPUTERorgs.com robot (205.134.190.xxx) using htdig,htdig/3.1.6 (http://computerorgs.com) htdig used by the Academie de Toulouse,htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer) htdig search tool,htdig/3.1.x (root@localhost) Lithops Software link validation tool,Html Link Validator (www.lithopssoft.com) HTML2JPG webpage to image converter,"HTML2JPG Blackbox, http://www.html2jpg.com" HTML2JPG webpage to image converter,HTML2JPG Enterprise HTML Parser Java library to parse HTML,HTMLParser/1.x PHP HTTP client to access Web servers,HTTP Retriever Anonymous web proxy service,http://Anonymouse.org/ (Unix) Ask 24x Info (Germany) DMOZ related robot,http://Ask.24x.Info/ (http://narres.it/) ACONTBOT - Acont search Germany robot,http://hilfe.acont.de/bot.html ACONTBOT OzySoftware.com software directory link checking,http://OzySoftware.com/Index.html IBM's Almaden Research robot,http://www.almaden.ibm.com/cs/crawler IBM's Almaden Research robot,http://www.almaden.ibm.com/cs/crawler [rc1.wf.ibm.com] IBM's Almaden Research robot,http://www.almaden.ibm.com/cs/crawler [wf216] Istarthere.com search robot,http://www.istarthere.com_spider@istarthere.com Monogol - German open source search engine project (195.226.167.1xx),http://www.monogol.de TrendTech Search Engine (Denmark) robot,http://www.trendtech.dk/spider.asp) HTTP::Lite - Standalone Perl module for retreiving HTTP documents,HTTP::Lite/2.x.x HTTPEyes - Web proxy cache,HTTPEyes HTTPResume Amiga download manager,HTTPResume v. 1.x HttpUnit - Java test code for emulating browser behaviour,httpunit/1.5 HttpUnit - Java browser behavior simulation tool,httpunit/1.x Hybrid Share mono C#/Gtk# application for file sharing,Hybrid/1.2 [en] (OS Independent) Hyper Estraier full-text search system,HyperEstraier/1.x.xx i1search robot,i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com) Alexa / The Internet Archive (209.237.238.1xx),IAArchiver-1.0 Unknown robot (reads robots.txt) from chinatelecom (219.142.78.xx),iaskspider Iask search / Sina portal robot (China),iaskspider2 (iask@staff.sina.com.cn) Alexa / The Internet Archive (209.237.238.1xx),ia_archiver Alexa / The Internet Archive (209.237.238.1xx),ia_archiver-web.archive.org Alexa / The Internet Archive (209.237.238.1xx),ia_archiver/1.6 IOSpirit iBrowse Amiga Browser,IBrowse/2.2 (AmigaOS 3.5) IOSpirit iBrowse Amiga Browser,IBrowse/2.2 (Windows 3.1) iCab MAC Web browser,iCab/2.5.2 (Macintosh; I; PPC) Knowledge Clustered Group ICC-Crawler (University of Tokyo - Japan),ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp) Knowledge Clustered Group ICC-Crawler (University of Tokyo - Japan),ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) ICJobs - Intelligence Competence Center (Germany) robot,iCCrawler (http://www.iccenter.net) ICJobs - Intelligence Competence Center (Germany) robot,ICCrawler - ICjobs (http://www.icjobs.de/bot.htm) ICE Java browser,ICE Browser/5.05 (Java 1.4.0; Windows 2000 5.0 x86) Goo Japan / Inktomi robot (210.173.179.xx),ichiro/x.0 (http://help.goo.ne.jp/door/crawler.html) Goo Japan / Inktomi robot (210.173.179.xx),ichiro/x.0 (ichiro@nttr.co.jp) Iconsurf.com - Visual Surf Engine / favicon finder,IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html) Iconsurf.com - Visual Surf Engine / favicon finder,IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html) icooLoader download manager,ICOO Loader v.x.x.x ICRA (Internet Content Rating Association) label spider,ICRA_label_spider/x.0 ICS Robot Search Engine (International Christian school of Seoul),icsbot-0.1 Internet Download Accelerator,IDA Janas (Ideare.com / Tiscali.it) robot,ideare - SignSite/1.x Unknown UA from Yahoo China,"iearthworm/1.0, iearthworm@yahoo.com.cn" Some bookmark manager,IEFav172Free iFeed.jp - online rss aggregator (in development),iFeed.jp/2.0 (www.psychedelix.com/agents/agents.rss; 0 subscribers) Igde search (Russia) robot,igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html) iGetter download manager,iGetter/1.x (Macintosh;G;PPC) iGetter download manager,iGetter/2 (Macintosh; U; PPC Mac OS X; en) Webkhoj - Indian language search engine,IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in) Ilial Knowledge Search robot,"ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com)" Unknown robot from UCLA using Nutch,ilial/Nutch-0.9-dev Ilse Netherlands robot (62.69.178.xx),IlseBot/1.x Il Trovatore - Italian search engine robot,IlTrovatore-Setaccio ( http://www.iltrovatore.it) Il Trovatore - Italian search engine robot,Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it) Il Trovatore - Italian search engine robot,IlTrovatore-Setaccio/1.2 ( http://www.iltrovatore.it/aiuto/faq.html) Il Trovatore - Italian search engine robot,Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it) Il Trovatore - Italian search engine robot,iltrovatore-setaccio/1.2-dev (spidering; http://www.iltrovatore.it/aiuto/.....) Il Trovatore - Italian search engine robot,IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it) ImageVisu image and graphics viewer - display files from the Web (HTTP and ECWP),ImageVisu/v4.x.x BD-Brandprotect copyright infringement crawler,ImageWalker/2.0 (www.bdbrandprotect.com) HttpClient - a PHP Web Client Class,Incutio HttpClient v0.x IncyWincy search engine using DMOZ Open Directory database,IncyWincy data gatherer(webmaster@loopimprovements.com IncyWincy search engine using DMOZ Open Directory database,IncyWincy page crawler(webmaster@loopimprovements.com Look.com robot using IncyWincy search engine,IncyWincy(http://www.look.com) IncyWincy search engine using DMOZ Open Directory database,IncyWincy(http://www.loopimprovements.com/robot.html) IncyWincy search engine using DMOZ Open Directory database,IncyWincy/2.1(loopimprovements.com/robot.html) Index the Web (69.57.134.xx) crawler,IndexTheWeb.com Crawler7 Spam bot from diff. IPs,Industry Program 1.0.x Inet Library Resource Center robot,Inet library InetURL IVM (phone software) plugin for web server access ?,InetURL/1.0 Pubblisito.com search - Italia,info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca Infoaxe - search history and bookmark service,Infoaxe./Nutch-0.9 Converas RetrievalWare Internet Spider (63.241.61.x),infoConveraCrawler/0.8 ( http://www.authoritativeweb.com/crawl) Versions-project.org Ingelin spider,InfoFly/1.0 (http://www.versions-project.org/) InfoLink link checking tool,InfoLink/1.x INFOMINE Scholary Internet Resource Collection crawler,INFOMINE/8.0 Adders INFOMINE Scholary Internet Resource Collection crawler,INFOMINE/8.0 RemoteServices INFOMINE Scholary Internet Resource Collection crawler,INFOMINE/8.0 VLCrawler (http://infomine.ucr.edu/useragents) 164.71.1.1xx jp.co.fujitsu.t2 Robot,InfoNaviRobot(F107) Infoseek robot,InfoSeek Sidewinder/0.9 Infoseek robot,InfoSeek Sidewinder/1.0A Infoseek robot,InfoSeek Sidewinder/1.1A Infoseek robot,Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX) Infoseek Japan robot,Infoseek SideWinder/2.0B (Linux 2.4 i686) Ilse Netherlands robot (62.69.178.xx),INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://webmaster.ilse.nl/jsp/webmaster.jsp) Inktomi (Hotbot-Lycos NBCi etc.) robot,Inktomi Search Enterprise Search engine software (64.202.165.xxx) ,InnerpriseBot/1.0 (http://www.innerprise.com/) Insitor Search robot (80.67.20.1xx),Insitor.com search and find world wide! Insitor Search robot (80.67.20.1xx),Insitornaut download manager,InstallShield DigitalWizard Integrity - website broken link checker for MAC OSx,integrity/1.6 Microton Intelix robot for Eurotran translation software ?,Intelix/0.x (cs; http://www.microton.cz/intelix/; microton@@microton.cz) Interarchy file transfer software - SFTP/FTP client for Mac OS X,Interarchy/x.x.x (InterarchyCrawler) Dream Train (Japan) Internet search robot,Internet Ninja x.0 Heritrix - The Internet Archive's open-source crawler based on Nutch (207.241.225.2xx),InternetArchive/0.8-dev(Nutch;http://lucene.apache.org/nutch/bot.html;nutch-agent@lucene.apache Internet Link Agent - link checking tool,InternetLinkAgent/3.1 Internetseer Web site monitoring / Claymont robot,InternetSeer.com intraVNews - Feed reader & RSS aggregator for Outlook,intraVnews/1.x Internet Open Index crawler using Nutch,IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org) IP*Works! HTTP Component,IP*Works! V5 HTTP/S Component - by /n software - www.nsoftware.com IP2Location - Reverse lookup geographical data and ISP by IP,http://www.ip2location.com IP2Map - geographical IP mapping,IP2MapBot/1.1 http://www.ip2map.com Laurions Ipium robot,IPiumBot laurion(dot)com Ipselon Web Search robot,IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com) Iria download manager,Iria/1.xxa IRL-crawler - Texas A&M University research project crawler,IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler) IRL-crawler - Texas A&M University research project crawler,IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/) url_log - Irssi Perl url grabber,IrssiUrlLog/0.2 Irvine downloading tool,Irvine/1.x.x Unknown spambot / harvester from diff. IPs,ISC Systems iRc Search 2.1 iSiloX document converter for iSilo reader,iSiloX/4.xx Windows/32 Unknown University of Alberta link-checking ?,isurf (tszhu@canada.com) iTunes UA name for access and decrypt the iTunes music store pages,iTunes/x.x.x Some spam bot from 66.139.78.xx(x),IUPUI Research Bot v 1.9a iVia robot - Open source Internet portal & virtual library system software,iVia Page Fetcher (http://ivia.ucr.edu/useragents.shtml) iVia robot - Open source Internet portal & virtual library system software,iVia/4.0 CanonizeUrl (http://infomine.ucr.edu/iVia/useragents.shtml "BD BrandProtect - brand, company or trademarks online monitoring",IWAgent/ 1.0 - www.brandprotect.com Proxy message from jp-q.ne.jp,J-PHONE/3.0/J-SH07 ODIN Directory Japan robot (163.138.95.xx),Jabot/6.x (http://odin.ingrid.org/) ODIN Directory Japan robot (163.138.95.xx),Jabot/7.x.x (http://odin.ingrid.org/) German Domanova (offline since Feb.02) robot,Jack Jakarta Commons (Java based) HTTP client,Jakarta Commons-HttpClient/2.0xxx Jakarta Commons (Java based) HTTP client,Jakarta Commons-HttpClient/3.0-rcx JamBot search robot (70.146.82.xx),Jambot/0.1.x (Jambot; http://www.jambot.com/blog; crawler@jambot.com) JamBot search robot,Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot; crawler@jambot.com) Java VM,Java 1.1 Java VM,Java/1.4.1_01 Java VM,Java1.0.21.0 Java VM,Java1.1.xx.x Java VM,Java1.3.0rc1 Java VM,Java1.3.x Java VM,Java1.4.0 Jayde B2B Search robot (66.28.139.xx),Jayde Crawler. http://www.jayde.com some site downloading tool ? via 61.77.51.xxx,JBH Agent 2.0 WAP 2.0 / jBrowser for handhelds,jBrowser/J2ME Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0) JCheckLinks Java hyperlink validator,JCheckLinks/0.1 RPT-HTTPClient/0.3-1 Java Development Kit,JDK/1.1 Insignias Jeode (PDA) Java platform,Jeode/1.x.x JetEye Search robot (64.62.142.xxx / 64.71.144.xxx),Jetbot/1.0 Omea RSS - Atom - newsgroups web page reader ,JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/) Omea RSS - Atom - newsgroups web page reader ,JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/) JetCar / Flashget download manager,JetCar Jigsaw - W3C's CSS Validator Server,Jigsaw/2.2.x W3C_CSS_Validator_JFouffa/2.0 Jobo website downloading program,JoBo/1.x (http://www.matuschek.net/jobo.html) Jobo website downloading program,JoBo/@JOBO_VERSION@(http://www.matuschek.net/jobo.html) Finacialbot.com - German (213.61.218.xx) job search JobRoboter,JobSpider_BA/1.1 Jocsoft Web Spider - website downloading tool,JOC Web Spider Jordo Media RSS / Atom feed directory link checking,JordoMedia/1.0 RSS File Reader (http://www.jordomedia.com) Journster.com RSS/Atom aggregator,Journster [alpha] (http://journster.com/) Journster.com RSS/Atom aggregator,Journster.com RSS/Atom aggregator 0.5 (http://www.journster.com/bot.phtml) Check Favorites bookmark checking,JRTS Check Favorites Utility Check Favorites bookmark checking,JRTwine Software Check Favorites Utility Jyxo search (Czech Republic) robot (212.71.128.xx),Jyxobot/x K-meleon browser - Windows 2000,K-Meleon/0.6 (Windows; U; Windows NT 5.1; en-US; rv:0.9.5) Gecko/20011011 Verity K2 Spider ( Network search software),k2spider Unknown robot from AITrc (Advanced Information Technology Research Center) - Korea,KAIST AITrc Crawler Kakle ranked metasearch robot,KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bot.html; support@kakle.com) Kalooga image crawler,kalooga/kalooga-4.0-dev-datahouse (Kalooga; http://www.kalooga.com; info@kalooga.com) Kalooga image crawler,kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; crawler@kalooga.com) Kapere site grapper / web downloader,Kapere (http://www.kapere.com) Kazehakase - Gecko based browser (Japan),Kazehakase/0.x.x.[x] Openwave UP.Browser for mobiles via Google WAP Proxy (216.239.33.x),KDDI-SN22 UP.Browser/6.0.7 (GUI) MMP/1.1 (Google WAP Proxy/1.0) Kenjin Spider search agent,Kenjin Spider Dznet.com Kevin crawler (link checking ?) via 68.39.148.xx (nj.comcast.net),Kevin http://dznet.com/kevin/ Website AlertsKevin crawler (website monitoring) via 68.39.148.xx (nj.comcast.net),Kevin http://websitealert.net/kevin/ Voila.fr robot,KE_1.0/2.0 libwww/5.2.8 Some Perl search script from KFSW (Germany),KFSW-Bot (Version: 1.01 powered by KFSW www.kfsw.de) kinja weblog search robot,kinja-imagebot (http://www.kinja.com/) kinja weblog search robot,kinjabot (http://www.kinja.com) Fireball search (Germany) robot,KIT-Fireball/2.0 Fireball search (Germany) robot,KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5) Klondike WAP Browser,Klondike/1.50 (WSP Win32) (Google WAP Proxy/1.0) University of Washington KnowItAll - web information extraction,KnowItAll(knowitall@cs.washington.edu) The knowledge.com (ODP) directory robot,Knowledge.com/0.x Kontiki Client download manager,Kontiki Client x.xx Krugle source code search engine for developers (64.71.164.1xx) ,"Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; webcrawler@krugle.com)" KnowledgeStorm technology industry crawler for Findtech.com,KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/content/crawler/index.html; crawleradmin@knowledgestorm.com) Kuloko contextual search robot,kuloko-bot/0.x Kuloko contextual search robot,kulokobot www.kuloko.com kuloko@backweave.com National Library of Sweden Heritage Project robot,kulturarw3/0.1 Link or server checking from Sanoma Budapest (195.70.35.xxx),KummHttp/1.1 (compatible; KummClient; Linux rulez) UP.Browser for mobiles on Kyocera KWC-KX9 cellphone,KWC-KX9/1109 UP.Browser/6.2.3.9.g.1.107 (GUI) MMP/2.0 UP.Link/6.3.0.0.0 TREC Blog Track - Blog and news feed crawler,Labrador/0.2; http://ir.dcs.gla.ac.uk/labrador; craigm@dcs.gla.ac.uk Intels Lachesis web site response time monitoring tool,Lachesis Unknown robot from Easten Network China (202.96.51.1xx),lanshanbot/1.0 Unknown robot from Easten Network China (202.96.51.1xx),lanshanbot/1.0 (+http://search.msn.com/msnbot.htm) Lapozz search (Hungary) robot (82.131.195.xx),LapozzBot/1.4 ( http://robot.lapozz.com) Lapozz search (Hungary) robot (82.131.195.xx),LapozzBot/1.5 (+http://robot.lapozz.hu) Larbin indexer used as Metacarta.com (66.28.xx.xxx) robot,larbin (samualt9@bigfoot.com) Unknown robot from 66.230.140.xx (argon.oxeo.com),LARBIN-EXPERIMENTAL (efp@gmx.net) Larbin indexer used by Mitsubishi Electric Research Labs,larbin_2.1.1 larbin2.1.1@somewhere.com Larbin indexer used as Compete.com crawler,larbin_2.2.0 (crawl@compete.com) Larbin indexer used as Inria robot,larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) Larbin indexer used as Kyoto University robot,larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) Larbin indexer used as Inria robot,larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) Larbin indexer used by an unknown dsl.net client,larbin_2.6.0 (larbin2.6.0@unspecified.mail) Larbin indexer used by diff. IPs / services,larbin_2.6.1 (larbin2.6.1@unspecified.mail) Larbin indexer used by National Institut of Informatics (NII/Japan),larbin_2.6.2 (hamasaki@grad.nii.ac.jp) Larbin indexer used by diff. IPs,larbin_2.6.2 (larbin2.6.2@unspecified.mail) Larbin indexer used as robot by Georgia Institute of Technology http://www.gatech.edu/,larbin_2.6.2 (listonATccDOTgatechDOTedu) Larbin indexer used by Technical University of Crete,larbin_2.6.2 (pimenas@systems.tuc.gr) Larbin indexer used as Lemur Consulting robot,larbin_2.6.2 (tom@lemurconsulting.com) Larbin indexer used as robot via cloud9.net (168.100.192.xxx),larbin_2.6.2 (vitalbox1@hotmail.com) Unknown robot from EPFL Lausanne Switzerland (128.178.155.1xx),larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch) Larbin indexer used by GenieKnows.com search,larbin_2.6.3 (wgao@genieknows.com) Larbin indexer used by Next Generation Information Retrieval (NGIR),larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi Larbin indexer used by CEA / DCom Rechercher,larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) Larbin indexer used as Inria robot,larbin_devel (http://pauillac.inria.fr/~ailleret/prog/larbin/) LawInfo - Lawyer and attorney directory,lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; webmaster@lawinfo.com) ROADS - Perl web based subject based gateway tool,lc/$ROADS::Version libwww-perl/5.00 unknown robot via MTT.ca / Aliant.ca,lcabotAccept: */* LeapTag news reader and content discovery tool,LeapTag/0.8.1.beta081.r3750 (compatible; Mozilla 4.0; MSIE 5.5; robot@yoriwa.com) Linkexchange crawler,LECodeChecker/3.0 libgetdoc/1.0 Leechget download manager,LeechGet 200x (www.leechget.de) Gseek.com (site is offline) robot,LEIA/2.90 Gseek.com (site is offline) robot,LEIA/3.01pr (LEIAcrawler; [SNIP]) Maybe logfile spamming for Lets crawl! search (Germany),LetsCrawl.com/1.0 +http://letscrawl.com/ Lexibot (exMataHari) search software,LexiBot/1.00 Polaris mobile browser on LG LX 260 Sprint Rumor phone,LG-LX260 POLARIS-LX260/2.0 MMP/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 LG 8138 Mobile Phone browser,LG/U8138/v1.0 About.com robot,Libby_1.1/libwww-perl/5.47 libcurl's (multiprotocol file transfer library) standard user-agent name,libcurl-agent/1.0 LibertyW search for mobile (France),LibertyW (+http://www.lw01.com) Korea Telecom Search robot,libWeb/clsHTTP -- hiongun@kt.co.kr CMP United Media robot,libwww-perl/5.41 SplatSearch robot (207.44.142.xx),libwww-perl/5.45 Alexa robot,libwww-perl/5.48 diff. IPs / services,libwww-perl/5.50 Fast Search robot,libwww-perl/5.52 FP/2.1 Fast Search robot,libwww-perl/5.52 FP/4.0 diff. IPs / services,libwww-perl/5.53 Profile for You internet profiling (?),libwww-perl/5.63 unknown link checking from Wanadoo.fr (193.253.33.xxx),libwww-perl/5.64 Amidalla search engine robot (62.241.33.xx),libwww-perl/5.65 SplatSearch robot (72.36.210.xx),libwww-perl/5.800 Mediater Rechercher robot,libwww/5.3.2 Liferea - Linux feed reader,Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/) Liferea (Linux Feed Reader) news aggregator for Unix and Linux,Liferea/1.x.x (Linux; es_ES.UTF-8; http://liferea.sf.net/) Lightning Download manager,LightningDownload/1.0beta2 Lightning Download manager,LightningDownload/1.x.x Lightning Download manager,LightningDownload/1.x.x [Accelerated x] Lijit blog search spider,LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com) Some spam bot,Lincoln State Web Browser Link Valet online link checking,Link Valet Online 1.x Linkalarm link validation,LinkAlarm/2.x Linkbot Pro link checking software,Linkbot Rpsoft 2000 Site-Crawler,linkbot Linkbot Pro link checking software,Linkbot x.0 Linkcheck - linkchecking tool for Unix/Linux,LinkCheck (linkcheck@inter7.com http://www.inter7.com/linkcheck) Linklint - Perl html link checker,LinkLint-checkonly/2.x.x Linklint - Perl html link checker,LinkLint-spider/2.x.x Linknz - The Kiwi Search Engine (New Zealand),linknzbot Link-Pimp web directory link checking,LinkPimpin v1.0 TafWeb link checking program,LinkProver 2.1 Links text browser for Unix & OS/2,Links (0.9x; Linux 2.4.7-10 i686) Links text browser for Unix & OS/2,Links (0.9xpre12; Linux 2.2.14-5.0 i686; 80x24) Links text browser for Unix & OS/2,Links (2.xpre7; Linux 2.4.18 i586; x) Links SQL directory management program,Links - http://gossamer-threads.com/scripts/links/ Links SQL directory management program,Links 2.0 (http://gossamer-threads.com/scripts/links/) Links SQL directory management program,Links SQL (http://gossamer-threads.com/scripts/links-sql/) Links4us ODP based directory link checking,"Links4US-Crawler, (+http://links4us.com/)" Shareware robot from Elsop.com (used by Indiafocus/Indiainfo and others),LinkScan/11.0beta2 UnixShareware robot from Elsop.com (used by Indiafocus/Indiainfo) Shareware robot from Elsop.com (used by Indiafocus/Indiainfo and others),LinkScan/9.0g Unix Shareware robot from Elsop.com,LinkScan/x.x Unix Linksmanager bookmark checking,LinksManager.com (http://linksmanager.com/linkchecker.html) LinkSonar (Japan) link checking tool,LinkSonar/1.35 Left Side Software's LinkSweeper (ceased) bookmark utility,LinkSweeper/1.x Seventwentyfour link checking robot,LinkWalker link check 3 - Perl HTML link checker (from Perl for Web Site Management),link_check3.plx libwww-perl/5.65 Listbid.com / Directnic.com link checking ?,ListBidBot (freelance job spider http://listbid.com)Freelance WKD Lab: LiveTrans - Online query and terminology translation service,LiveTrans/Nutch-0.9 (maintainer: cobain at iis dot sinica dot edu dot tw; http://wkd.iis.sinica.edu.tw/LiveTrans/) llaut robot - Universitat de les Illes Balears (Spain),Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) E-Mail harvesting robot - same as ContactBot,LMQueueBot/0.2 lmspider from Scansoft (192.133.61.xx) - Web text collector,lmspider (lmspider@scansoft.com) Lexis-Nexis robot,LNSpiderguy LocalBot company information collector,LocalBot/1.0 ( http://www.localbot.co.uk/) Local.com local search robot (216.52.252.xxx),LocalcomBot/1.2.x ( http://www.local.com/bot.htm) Lockstep (website content protection tool) user agent,Lockstep Spider/1.0 GlobalQueue spider (64.40.105.xxx),Look.com Lotus Notes browser,Lotus-Notes/4.5 ( Windows-NT ) IBM Lotus Discovery Server,LotusDiscovery/x.0 (compatible; Mozilla 4.0; MSIE 4.01; Windows NT) Everatom.com song lyrics search,Lovel as 1.0 ( +http://www.everatom.com) LTI - The Lemur Toolkit for Language Modeling and Information Retrieval via Yahoo,LTI/LemurProject Nutch Spider/Nutch-1.0-dev (lti crawler for CMU; http://www.lti.cs.cmu.edu; changkuk at cmu dot edu) LTI - The Lemur Toolkit for Language Modeling and Information Retrieval via Yahoo,LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://www.lemurproject.org; mhoy@cs.cmu.edu) Luchs.at (Linux Wiki) link checking,luchs.at URL checker Lunascape IE based browser (Japan),Lunascape Ultimate Search / Smartdesk (no website) robot,lwp-trivial/1.32 Search4free robot,lwp-trivial/1.34 Search4free robot,lwp-trivial/1.34 Expert HTML online source viewer,lwp-trivial/1.35 Expert HTML online source viewer,lwp-trivial/1.35 Perl LWP:Collective module,LWP::Simple/5.22 Perl LWP:Collective module,LWP::Simple/5.36 Perl LWP:Collective module - Linkomatic robot,LWP::Simple/5.48 Perl LWP:Collective module - secure-netz.de link checking (in conjunction w. libwww-perl/5.50),LWP::Simple/5.50 Perl LWP:Collective module - Inktomi (62.253.64.x) robot,LWP::Simple/5.51 Perl LWP:Collective module - Only.com ,LWP::Simple/5.53 Perl LWP:Collective module,LWP::Simple/5.63 ThePlanet/jaja-jak-globusy.com Google Adsense refferer spam bot from 70.85.116.* / 70.84.128.xxx / 70.85.193.xxx,LWP::Simple/5.803 Lycos spider,Lycos_Spider_(modspider) Lycos spider,Lycos_Spider_(T-Rex) Bobcat - Text and Lynx based DOS browser,Lynx/2-4-2 (Bobcat/0.5 [DOS] Jp Beta04) Lynx 2.x text mode browser,Lynx/2.6 libwww-FM/2.14 seebot.org online service - uses lynx browser for crawlers view of web pages,Lynx/2.8 (;http://seebot.org) Lynx 2.x text mode browser,Lynx/2.8.3dev.9 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6 Lynx 2.x text mode browser used as robot via cogentco.com,Lynx/2.8.4rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6c (human-guided@lerly.net) Some spam bot,Mac Finder 1.0.xx UKWizz search robot,Mackster( http://www.ukwizz.com ) Wind.it client user-agent ?,Mag-Net All Magic/Wap wml service for mobile devices,MagicWML/1.0 (forcewml) Magpie RSS - PHP RSS Parser,MagpieRSS/0.7x (+http://magpierss.sf.net) Mahiti.com India search crawler,Mahiti.Com/Mahiti Crawler-1.0 (Mahiti.Com; http://mahiti.com ; mahiti.com) Mail.ru search,Mail.Ru/1.0 Unknown Bea robot,mailto:webcraft@bea.com SLI Systems mammoth robot,mammoth/1.0 ( http://www.sli-systems.com/) Looksmart robot,MantraAgent Map of the Internet visual search engine index robot,MapoftheInternet.com ( http://MapoftheInternet.com) Kolibri.de robot,Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) Pilot Hitlist web site analysis,Marketwave Hit List Looksmart directory page analysis,Martini Looksmart directory page analysis,MARTINI Marvin Medhunt robot,Marvin v0.3 Sagool search Japan robot,MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp) Mass Downloader download manager,Mass Downloader 2.x Masterseek (Scandinavia) Beta business search ?,MasterSeek Lexibot (exMataHari) search software,Mata Hari/2.00 Virgilio Italy robot (212.48.11.xxx) using Fast Enterprise Search,Matrix S.p.A. - FAST Enterprise Crawler 6 (Unknown admin e-mail address) Maxomo multimedia search robot,maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; maxomobot@maxomo.com) Unknown robot from McAfee Austria (80.123.144.xx),McBot/5.001 (windows; U; NT4.0; en-us) MegaDownload files search robot,MDbot/1.0 (+http://www.megadownload.net/bot.html) Substitute for Windows Media Player,Media Player Classic Media Find crawler,MediaCrawler-1.0 (Experimental) Google AdSense robot,Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html) WWW.fi Media Search,MediaSearch/0.1 Search UK robot,MegaSheep v1.0 (www.searchuk.com internet sheep) Megite web2.0 RSS and news service software,Megite2.0 (http://www.megite.com) Mercator crawler software (used by Altavista),Mercator-1.x Mercator crawler software (used by Altavista),Mercator-2.0 Mercator crawler software (used by Altavista),Mercator-Scrub-1.1 MetaEuro.com Clustering Web Search Engine crawler,Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.metaeuro.com; crawler at metaeuro dot com) MetaGer search robot (Germany),MetaGer-LinkChecker MetaGer search robot (Germany),MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) MetaGer search robot (Germany),MetaGer_PreChecker0.1 Download Express download manager,MetaProducts Download Express/1.x Metaspinner search robot - Germany,Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/) Metatagsdir.com directory index spider,metatagsdir/0.7 (+http://metatagsdir.com/directory/) Microsoft Foundation Class Library - i.e. used for e-mail harvesting from 68.154.96.xx (bellsouth.net),MFC Foundation Class Library 4.0 Microsoft.com user agent,MFC_Tear_Sample Advanced Site Crawler web site ripper and extractor,MFHttpScan GigaBaz Brainbot (Germany) robot (213.139.152.xx),MicroBaz MS Office 2000 acting as WebDAV client,Microsoft Data Access Internet Publishing Provider Cache Manager MS Office 2000 acting as WebDAV client,Microsoft Data Access Internet Publishing Provider DAV Server probe for data access operations using MS Frontpage with OPTION header,Microsoft Data Access Internet Publishing Provider Protocol Discovery MS Office 2000 acting as WebDAV client,Microsoft Data Access Internet Publishing Provider Protocol Discovery Microsoft Log Parser text query tool,Microsoft Log Parser 2.2 MS Small Business Server content indexer,Microsoft Small Business Indexer user agent looks for form-mail components (spam-bot),Microsoft URL Control - 6.00.8xxx Unknown robot from Microsoft.com (131.107.163.xx),MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com) Secure Computing SmartFilterWhere / Bess web filter (192.55.214.xx),Microsoft_Internet_Explorer_5.00.438 (fjones@isd.net) xpc-mii.net HTTP server message,MIIxpc/4.2 The Mindjet blog MindManager category,Mindjet MindManager unknown robot via Korea Telecom (211.218.xxx.xxx),minibot mini- Rank website popularity tool,miniRank/1.6 (Website ranking; www.minirank.com; robot) unknown robot via nec.co.jp Telecom (210.143.35.xx),MiracleAlphaTest Some spam bot,Missauga Locate 1.0.0 Some spam bot,Missigua Locator 1.9 Some spam bot,Missouri College Browse Mister PiX picture finding software,Mister Pix II 2.02a Mister PiX picture finding software,Mister PiX version.dll Misterbot search France robot,Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot.fr) Miva / ex Findwhat.com search robot (66.150.55.2xx),Miva (AlgoFeedback@miva.com) Some spam bot from Jasmine Internet - Bangkok (203.147.0.xx),Mizzu Labs 2.2 Majestic-12 DSearch MJ12bot (Experimental distributed crawler),MJ12bot/vx.x.x (http://majestic12.co.uk/bot.php?+) Majestic-12 DSearch MJ12bot (Experimental distributed crawler),MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php) MJB SEO Club MJBot,MJBot (SEO assessment) MLBot - metadata labs web crawler for building a media index (beta),MLBot (www.metadatalabs.com) mnoGoSearch (ex UdmSearch) software robot,MnogoSearch/3.2.xx Unknown bad bot - maybe guestbook spamming or email harvesting,Mo College 1.9 Goo Japan / Inktomi robot (210.173.179.xx),moget/x.x (moget@goo.ne.jp) Goo Japan / Inktomi robot (210.173.179.xx),mogimogi/1.0 moiNag - net.art generator,moiNAG 0.02 Mojeek Search Preview robot (217.155.205.xx),MojeekBot/0.x (archi; http://www.mojeek.com/bot.html) Greasemonkey Firefox extension,monkeyagent Moonbrowser - IE based browser (Japan),MoonBrowser (version 0.41 Beta4) Moreover / FeedDirect RSS feed robot,Moreoverbot/x.00 (+http://www.moreover.com) MixCat robot s. also Felix,Morris - Mixcat Crawler ( http://mixcat.com) Robots.txt online checker,Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml) Obigo WAP browser for mobiles on Motorola V3,Motorola-V3m Obigo MPRM Group Ltd. Spider Monkey robot,Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) Movable Type web-based personal publishing system,MovableType/x.x Mozdex Open search engine spider (65.98.100.2xx),mozDex/0.xx-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com) Bell Nexxia / Sympatico Canada user robot & spoofed referer from diff. IPs,Mozi! Unknown robots from diff. IPs,Mozilla libwhisker - HTTP client and utility - Perl library,Mozilla (libwhisker/2.4) http://www.somewhere.com robot,Mozilla (Mozilla@somewhere.com) BotSeer search engine for robots.txt,Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu) PDA Pocket IE 2.x Windows CE,Mozilla/1.1 (compatible; MSPIE 2.0; Windows CE) Oregano browser for RISC OS,Mozilla/1.10 [en] (Compatible; RISC OS 3.70; Oregano 1.10) IE 2.x WinNT,Mozilla/1.22 (compatible; MSIE 2.0d; Windows NT) EudoraWeb 2.0 browser (Eudora Internet Suite) for PalmOS,Mozilla/1.22 (compatible; MSIE 5.01; PalmOS 3.0) EudoraWeb 2 Dummy user agent - i.e. used by Namo Web Editor,Mozilla/2.0 AOL Mac,Mozilla/2.0 (compatible; AOL 3.0; Mac_PowerPC) AOL Win 3.x,Mozilla/2.0 (Compatible; AOL-IWENG 3.0; Win16) Ask Jeeves /Teoma robot,Mozilla/2.0 (compatible; Ask Jeeves) Ask Jeeves /Teoma robot,Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Ask Jeeves /Teoma robot,Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml) Ask Jeeves /Teoma robot,Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_crawling.html) Direct Hit Robot,Mozilla/2.0 (compatible; EZResult -- Internet Search Engine) MS Frontpage x.x web editor,Mozilla/2.0 (compatible; MS FrontPage x.0) IE 2.x Mac Power PC,Mozilla/2.0 (compatible; MSIE 2.1; Mac_PowerPC) IE 3.x WinNT,Mozilla/2.0 (compatible; MSIE 3.02; Update a; AK; Windows NT) IE 3.x AOL Win95,Mozilla/2.0 (compatible; MSIE 3.02; Update a; AOL 3.0; Windows 95) IE 3.x Win95,Mozilla/2.0 (compatible; MSIE 3.0; AK; Windows 95) IE 3.x Win 3.1,Mozilla/2.0 (compatible; MSIE 3.0; Windows 3.1) IE 3.x WinXP,Mozilla/2.0 (compatible; MSIE 3.0B; Win32) Borland Delphi .OCX component used by WebCollector email harverster,Mozilla/2.0 (compatible; NEWT ActiveX; Win32) Thunderstone's Webinator Web indexing program,Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) Check&Get bookmark and link checking tool,Mozilla/2.0 compatible; Check&Get 1.1x (Windows 98) Netscape 2.x Win3.x International,Mozilla/2.01 (Win16; I) Netscape 2.x Gold Win95,Mozilla/2.02Gold (Win95; I) Faked user agent for diff. purposes i.e.: - some download manager - E-mail harvesting,Mozilla/3.0 (compatible) AvantGo PDA browser,Mozilla/3.0 (compatible; AvantGo 3.2) Searchhippo robot,Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com) HP Web PrintSmart (discontinued) - web page printing software,Mozilla/3.0 (compatible; HP Web PrintSmart 04b0 1.0.1.34) Internet Direct Library for Borland (often used as e-mail address collector and mass mailing tool),Mozilla/3.0 (compatible; Indy Library) Outertechs Linkman bookmark tool,Mozilla/3.0 (compatible; Linkman) Euroferret robot,Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com) Euroferret robot,Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk) Euroferret robot,Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com) Netart Generator - script generated random websites,Mozilla/3.0 (compatible; netart generator/1.0; libwww-perl/5.64) NetPositive BEOS browser,Mozilla/3.0 (compatible; NetPositive/2.2) Opera 3.x Win3.x,Mozilla/3.0 (compatible; Opera/3.0; Windows 3.1) v3.1 Opera 3.x Win95/NT,Mozilla/3.0 (compatible; Opera/3.0; Windows 95/NT4) 3.2 Perman Surfer bookmark tool,Mozilla/3.0 (compatible; PerMan Surfer 3.0; Win95) Web Link Validator link validation software,Mozilla/3.0 (compatible; REL Software Web Link Validator 2.x) Scan4Mail online mail extraction service,Mozilla/3.0 (compatible; scan4mail (advanced version) http://www.peterspages.net/?scan4mail) WebWobot UK search engine robot (82.43.129.2xx),Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com) Web Link Validator link validation software,Mozilla/3.0 (compatible; Web Link Validator 2.x)Web Link Validator http://www.relsoftware.com/ link validation software Xelios Web Capture (now Wysigot) website downloading tool (Discontinued),Mozilla/3.0 (compatible; WebCapture x.x; Auto; Windows) Iprospect search engine positioning using Thunderstone's Webinator,Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) CyberAlert's Media Monitor using Thunderstone's Webinator,Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) Viking server user/client,Mozilla/3.0 (Compatible;Viking/1.8) One of DC-Sakuras download manager user-agent names,Mozilla/3.0 (DreamPassport/3.0) Ilse Netherlands robot (62.69.178.xx),Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints) Liberate DTV server suite / TV-emulator,Mozilla/3.0 (Liberate DTV 1.1) Planetweb 2.1 Browser (discontinued) for Dreamcast,Mozilla/3.0 (Planetweb/2.100 JS SSL US; Dreamcast US) http://www.goo.ne.jp /Inktomi robot,Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Inktomi (Hotbot-Lycos-NBCi) robot - 72.30.61.xx(x),Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Inktomi (Hotbot-Lycos-NBCi) robot - 72.30.61.xx(x),Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) WiseGuys robot Netherland - 82.94.216.2,Mozilla/3.0 (Vagabondo/1.1 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/) WiseGuys robot Netherland - 82.94.216.2,Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys.nl/) Ilse Netherlands robot,Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints) WiseGuys robot Netherland - 82.94.216.2,Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/) Netscape 3.x Win3.x,Mozilla/3.0 (Win16; I) Netscape 3.x Win95,Mozilla/3.0 (Win95; I) Netscape 3.x WinNT,Mozilla/3.0 (WinNT; I) Netscape 3.x FreeBSD,Mozilla/3.0 (WorldGate Gazelle 3.5.1 build 11; FreeBSD2.2.8-STABLE) Netscape 3.x OSF1 V4.0 alpha ,Mozilla/3.0 (X11; I; OSF1 V4.0 alpha) AOL Web TV,Mozilla/3.0 NAVIO_AOLTV (11; 13; Philips; PH200; 1; R2.0C36_AOL.0110OPTIK; R2.0.0139d_OPTIK) WebTV,Mozilla/3.0 WebTV/1.2 (compatible; MSIE 2.0) Amiga Voyager Browser Amiga,Mozilla/3.01 (compatible; AmigaVoyager/2.95; AmigaOS/MC680x0) Links2Go robot,Mozilla/3.01 (Compatible; Links2Go Similarity Engine) Netgem Netbox cable modem TV Box Linux,Mozilla/3.01 (compatible; Netbox/3.5 R92; Linux 2.2) Netscape 3.x Mac,Mozilla/3.01-C-MACOS8 (Macintosh; I; PPC) Netscape 3.x Linux,Mozilla/3.01Gold (X11; I; Linux 2.0.32 i486) Netscape 3.x SunOS,Mozilla/3.01Gold (X11; I; SunOS 5.5.1 sun4m) Netscape 3.x Irix,Mozilla/3.01SGoldC-SGI (X11; I; IRIX 6.3 IP32) ANT Fresco Browser Risc OS,Mozilla/3.04 (compatible; ANTFresco/2.13; RISC OS 4.02) NCBrowser ANT Fresco Browser Risc OS,Mozilla/3.04 (compatible; NCBrowser/2.35; ANTFresco/2.17; RISC OS-NC 5.13 Laz1UK1309) QNX OS Voyager embedded browser,Mozilla/3.04 (compatible;QNX Voyager 2.03B ;Photon) I-Opener (was www.netpliance.com/) web PC,Mozilla/3.x (I-Opener 1.1; Netpliance) Yahoo Mindset: Intent-driven Search (66.228.182.1xx),Mozilla/4.0 Agada search (Germany) robot,Mozilla/4.0 (agadine3.0) www.agada.de URLBase 6 bookmark manager,Mozilla/4.0 (Compatible); URLBase 6 Astrafind! adult search robot (66.98.252.xx),Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) WiseGuys robot Netherland - 82.94.216.2,Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) WiseGuys robot Netherland - 82.94.216.2,Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) ReGet Deluxe! download manager,Mozilla/4.0 (compatible; ReGet Deluxe 5.1; Windows NT 5.1) Advanced Email Extractor e-mail collector (spam bot),Mozilla/4.0 (compatible; Advanced Email Extractor v2.xx) Arachmo Spider - web site file extraction tool,Mozilla/4.0 (compatible; Arachmo) Novell Border Manager security suite,Mozilla/4.0 (compatible; BorderManager 3.0) Best of the Web directory link checking,Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org) Blitzsuche Germany robot,Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T) Content Control from Blue Coat,Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0) "Check&Get bookmark manager, web change monitor and archiver",Mozilla/4.0 (compatible; Check&Get 3.0; Windows NT) Christcentral.com Christcrawler (was www.christcrawler.com),Mozilla/4.0 (compatible; ChristCrawler.com ChristCrawler@ChristCENTRAL.com) Yahoo Search Marketing crawler (68.142.211.1xx),"Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com)" DAUMOA - Daum search Korea robot (211.115.109.xxx),Mozilla/4.0 (compatible; DAUMOA-video; +http://ws.daum.net/aboutkr.html) DepSpid distributed web crawler for link dependencies,Mozilla/4.0 (compatible; DepSpid/5.0x; +http://about.depspid.net) Download Mage download manager,Mozilla/4.0 (compatible; DnloadMage 1.0) Fast/Alltheweb crawler,Mozilla/4.0 (compatible; FastCrawler3 support-fastcrawler3@fast.no) Fluid Dynamics Search Engine (FDSE) robot used by Abadoor.de,Mozilla/4.0 (compatible; FDSE robot) GPU Distributed Search Engine crawler,Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engine.php) Grub open source crawler,Mozilla/4.0 (compatible; grub-client-0.2.x; Crawl your stuff with http://grub.org) Grub open source crawler,Mozilla/4.0 (compatible; grub-client-0.3.x; Crawl your own stuff with http://grub.org) Grub open source crawler used by Looksmart ( 64.241.242.xx),Mozilla/4.0 (compatible; grub-client-2.x) ibisBrowser Japanese mobile browser,Mozilla/4.0 (compatible; ibisBrowser) Novell iChain Cool Solutions caching,Mozilla/4.0 (compatible; ICS 1.2.xxx) IE Favorites Check - Bookmark manager,Mozilla/4.0 (compatible; IE-Favorites-Check-0.5) Iplexx Austria (webhosting company) logfile spamming bot,Mozilla/4.0 (compatible; Iplexx Spider/1.0 http://www.iplexx.at) KeepNi Monitors - Web site monitoring / link checking tool,Mozilla/4.0 (compatible; KeepNI web site monitor) NetPromoter Link Utility link checking tool,Mozilla/4.0 (compatible; Link Utility; http://net-promoter.com) Lotus Notes 5.0 browser,Mozilla/4.0 (compatible; Lotus-Notes/5.0; Windows-NT) IE 4.x AOL Win98,Mozilla/4.0 (compatible; MSIE 4.01; AOL 4.0; Windows 98) IE 4.x Mac Power PC,Mozilla/4.0 (compatible; MSIE 4.01; Mac_PowerPC) Internet Explorer 4.0 URL check,Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows 95) Vonna search robot,Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t) IE 4.x Win95,Mozilla/4.0 (compatible; MSIE 4.01; Windows 95) IE PDA Browser Windows CE,Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; MSN Companion 2.0; 800x600; Compaq) Orange France robot for mobiles,Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com) IE for Windows CE on a PocketPC (HP iPAQ),Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPS; 240x320) IE PDA Browser Windows CE,Mozilla/4.0 (compatible; MSIE 4.01; Windows NT Windows CE) IE 4.x WinNT,Mozilla/4.0 (compatible; MSIE 4.01; Windows NT) diff. IPs / services i.e.: - Microsoft server information robot (see link) - Okanagan Internet Junction web filter (robot),Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoft Unknown robot from American College of Radiology (ACR) running MS Site Server,Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR Indonesia Interactive Web-portal robot on MS Site Server,Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia Interactive Webquestdesigns hosting,Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) WebQuest Designs Avirt Gateway proxy server,Mozilla/4.0 (compatible; MSIE 5.01; Windows 95) via Avirt Gateway Server v4.0 Metacarta.com (66.28.xx.xxx) robot,Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com) NetCaptor IE browser addon,Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; NetCaptor 6.5.0RC1) IE 5.x AOL Win95 Sureseeker search plugin,Mozilla/4.0 (compatible; MSIE 5.0; AOL 5.0; Windows 95; DigExt; Gateway2000; sureseeker.com) IE 5.x Mac PowerPC AtHome user,Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; AtHome021) www.netnose.com crawler,Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com) WinXP via CERN httpd proxy server ,Mozilla/4.0 (compatible; MSIE 5.0; Win32) via proxy gateway CERN-HTTPD/3.0 libwww/2.17 Echo.com robot,Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 Voila.fr robot,Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/) Voila.fr robot,Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6 WinME Opera 5.x,Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] Kyosoft's Link Checker,Mozilla/4.0 (compatible; MSIE 5.0; Windows ME; Link Checker 2.x.xx http://www.kyosoft.com) Beijing Express Email Address Extractor via DHCP Data Transport Services (DTS),Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Girafa (browser plug-in) robot,Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com) Galaxy robot (63.121.41.xxx) ,Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com) Galaxy robot (63.121.41.xxx),Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com/; http://www.galaxy.com/info/crawler.html) Yandex Search Russia link checking (213.180.206.2xx),Mozilla/4.0 (compatible; MSIE 5.0; YANDEX) Gobeez starting page plugin,Mozilla/4.0 (compatible; MSIE 5.5; AOL 4.0; Windows 98; GoBeez (www.gobeez.com)) IE 5.5 Win95 Hotbar plug-in,Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; Transmission Segment; Hotbar 2.0) Crazy Browser - IE based tabbed Browser,Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Crazy Browser 1.x.x) Wanadoo Internet services,Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; KITV4.7 Wanadoo) Safexplorer (safexplorer.com - site is offline) kids browser,Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SAFEXPLORER TL) Katiesoft Scroll (ex www.katiesoft.com now discarded) & SimulBrowse (ex www.simulbrowse.com now dead) IE browser plugins,Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SYMPA; Katiesoft 7; SimulBrowse 3.0) Windows ME BTOpenworld Internet services,Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; BTinternet V8.1) Windows ME Internet Explorer URL check,Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler) Cobion Germany Brand Protection Services robot,Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) Cobion Germany Brand Protection Services robot,Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018) IE 5.5 Win2000 / user agent,Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Active Cache Request Maybe: - MS Internet Security & Acceleration Server (ISA) cache refreshing request (see link) or - IE 5.5 Win2000 probably with some (website) API request component (see 2nd link) - suspected as email-harvester / site scanning tool (see http://www.byte.com/documents/s=493/byt20010208s0001/index.htm,Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request IE 5.5 Win2000 with MS.NET SDK,Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705) IE 5.5 Win2000 / user agent w. AI RoboForm (AIRF) password manager,Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AIRF) AspTear URL fetching program component / Download32.com spider,Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AspTear 1.5) Nokia.com network,Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; N_o_k_i_a) Unknown Object Sciences Corp. robot using the HTTPClient,Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3E Singingfish media spider (64.12.186.2xx) via AOL search,Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revision: 3.10 Megaupload Mega Manager - Download manager toolbar for IE,Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows 98; .NET CLR 1.1.4322; MEGAUPLOAD 2.0) Hotbar IE graphical skin,Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1; SV1; HbTools 4.7.2) Skaffe.com directory link checker,Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] TargetSeek Crawler concerning electronics industry product announcements,Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/TargetSeek.html) IE 6.0 WebWasher ad filter,Mozilla/4.0 (compatible; MSIE 6.0; Win32) WebWasher 3.0 Web Link Validator link validation software,Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) REL Software Web Link Validator 2.x) Web Link Validator link validation software,Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Web Link Validator 2.x) IE 6.0 Netmanager IE add-on,Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Net M@nager V3.02 - www.vinn.com.au) WebLight web analyzer & link checker,Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; support@illumit.com; http://www.illumit.com/Products/weblight/) Abolimba Multibrowser - IE based browser,Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; http://www.Abolimba.de) Lunascape IE based browser (Japan),Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; Lunascape 2.1.3) Google wireless transcoder (GWT) proxy for rewriting websites for mobiles,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;) Tüzilla (Germany) - ODP link checking using Robozilla,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuezilla.de/t_st-odp-entries-agent.html) Tüzilla (Germany) - ODP link checking using Robozilla,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP links test; http://tuezilla.de/test-odp-links-agent.html) ZoomSpider.Net indexing robot for several directorys,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR 1.1.4322) unknown robot from 64.246.44.xx,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (dns_admin@c-a-s-h.com) Covac Software UPPS (Universal PHP Proxy Server) - free public proxy server,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Covac UPPS Cathan 1.2.5;) GetNetWise Crayon Crawler web filter,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Crayon Crawler; snprtz|T04056566514940; (R1 1.5)) Deepnet Explorer - IE based browser,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Deepnet Explorer) Heritrix Internet Archive's open-source web project used by Analysis Projects at UW,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 http://www.cs.washington.edu/research/networking/websys/) IE 6x WinXP Hotbar plug-in,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 3.0) IE 6x WinXP iOpus Internet Macros - Internet-based macro recorder ,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iOpus-I-M) iRider - IE based browser / Free Download Manager (FDM),Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iRider 2.21.1108; FDM) KKman http://www.kkman.com/ - Japanese IE based browser,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; KKman3.0) IE 6x WinXP MathPlayer mathematical notation plugin ,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MathPlayer2.0) Maxton (ex MyIE2) - IE based browser,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Maxthon) IE 6x WinXP peoplepc online PeoplePal IE toolbar,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; PeoplePal 3.0; MSIECrawler) IE 6x WinXP / I-Opener (was www.netpliance.com/) web PC,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; IOpener Release 1.1.04) Qihoo search (China) robot,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.net) SimBar IE toolbar for accessing The Sims sites / Infopath IE form & spreadsheet plugin,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SIMBAR Enabled; InfoPath.1) IE 6x WinXP Stumble Upon IE toolbar,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; StumbleUpon.com 1.760; .NET CLR 1.1.4322) Balsa Productions embedded web browser package for Borland Delphi,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler) ChangeDetection robot for web page monitoring,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html ) IE 6x WinXP also used by WebSite Pro HTML editor,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) DX-Browser - German IE based browser,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DX-Browser 5.0.0.0) ezPeer+ P2P IE addon,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; ezPeer+ v1.0 Beta (0.4.1.98); ezPeer+ v1.0 (0.5.0.00); .NET CLR 1.1.4322; MSIECrawler) Axandra IBP website promotion software ?,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322) MRA = Mail.ru Agent - Instant Messenger / VoIP,Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRA 4.3 (build 01218)) MSN Search robot (207.46.89.xx),Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) Link Commander bookmark manager,Mozilla/4.0 (compatible; MSIE 7.0; Win32) Link Commander 4.0 IE 7.0 - WinXP,Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; bgft) GTB = Google Toolbar Internet Explorer add-on,Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB5; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 2.0.50727) MSIE 7.0 *and* Trident token used by Internet Explorer 8 in compatibility view mode,Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0) Internet Explorer 8,Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648) IE 8.0 (beta) on Win Vista,Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0) Bsalsa embedded browser,Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Orange 8.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30618; OfficeLiveConnector.1.3; OfficeLivePatch.1.3) DAUMOA - Daum search Korea robot (211.115.109.xxx),"Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.net/aboutkr.html)" DAUMOA - Daum search Korea robot (211.115.109.xxx),"Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea)" Naver Search Korea Naverbot,Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp) Netcraft webserver info,Mozilla/4.0 (compatible; Netcraft Web Server Survey) NetPromoter Link Utility link checking tool ,Mozilla/4.0 (compatible; NetPromoter Spider;http://www.net-promoter.com/) Opera 3.x WinNT,Mozilla/4.0 (compatible; Opera/3.0; Windows 4.10) 3.51 [en] Powermarks bookmark manager,Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT) RSS Popper - MS Outlook RSS reader plugin,Mozilla/4.0 (compatible; RSS Popper) SiteKiosk public terminal browser,Mozilla/4.0 (compatible; SiteKiosk 4.0; MSIE 5.0; Windows 98; SiteCoach 1.0) Entireweb Search Speedyspider (62.13.25.xxx),Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com) SiteProbe - website status checking,Mozilla/4.0 (compatible; SPENG) Super Cleaner privacy tool (bookmark checking),Mozilla/4.0 (compatible; SuperCleaner 2.xx; Windows 98) Synapse - Apache web service for processing XML documents,Mozilla/4.0 (compatible; Synapse) Web2PDF - Adobe Acrobat plugin for site traversal and other services for the Web Capture feature,Mozilla/4.0 (compatible; WebCapture 3.0; Windows) Windows HTTP Services (WinHTTP / XML-parser),Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5) WSN Links PHP directory software,Mozilla/4.0 (compatible; WSN Links) Euro Directory (German / Austrian) directory link checking,Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0) Galaxy robot,Mozilla/4.0 (compatible; www.galaxy.com) Linkguard.com link validation (service is offline),Mozilla/4.0 (compatible; www.linkguard.com Linkguard Online 1.0; Windows NT) Yahoo Search Japan robot (203.141.52.),Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) Yahoo Japan robot (202.93.76.xx),Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya) Faked IE id string used by DeepTrawl link checking tool,Mozilla/4.0 (compatible;MSIE 6.0; Windows NT 5.0; H010818) spoofed referer by Fantomaster (Multiblocker) anonymity products,Mozilla/4.0 (fantomBrowser) spoofed referer by Fantomaster (Multiblocker) anonymity products,Mozilla/4.0 (fantomCrew Browser) unknown robot from - 64.57.223.40 - 66.28.233.xxx (cogentco.com),Mozilla/4.0 (hhjhj@yahoo.com) Activtourist Jemma spider,Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) NetFront (v3.x) for Pocket PC (here on Sanyo PM-8200 cell phone),Mozilla/4.0 (MobilePhone PM-8200/US/1.0) NetFront/3.x MMP/2.0 Google robot from 66.249.66.xxx ,Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) Google robot from 66.249.66.xxx ,Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; http://www.google.com/bot.html) Unknown robot from Mozilla.org,Mozilla/4.0 (Mozilla; http://www.mozilla.org/docs/en/bot.html; master@mozilla.com) ASI - Any Search Info robot,Mozilla/4.0 (Sleek Spider/1.2) Furl (Looksmart) online bookmark tool robot,Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net) Wisenut robot,Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) Wisenut robot,Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) Wisenut robot,Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) Wisenut robot,Mozilla/4.0 compatible ZyBorg/1.0 for Homepage (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) Unknown robot from 66.230.140.xx (argon.oxeo.com) maybe an e-mail collector,Mozilla/4.0 efp@gmx.net WebTV,Mozilla/4.0 WebTV/2.6 (compatible; MSIE 4.0) Ask / Ask Jeeves robot,Mozilla/4.0 [en] (Ask Jeeves Corporate Spider) LookSmart spider,Mozilla/4.0(compatible; Zealbot 1.0) HideMe - Web based anonymous proxy server service,Mozilla/4.01 (compatible; NORAD National Defence Network) Some download manager spoofing Netscape 4.01,Mozilla/4.01 [en](Win95;I) Netscape 4.x SunOS 5.6,Mozilla/4.02 [en] (X11; I; SunOS 5.6 sun4u) Dulance Bot - Dulance automated price comparison engine,Mozilla/4.04 (compatible; Dulance bot; +http://www.dulance.com/bot.jsp) Netscape 4.x HP-Unix,Mozilla/4.04 [en] (X11; I; HP-UX B.10.20 9000/712) Netscape 4.x IRIX,Mozilla/4.04 [en] (X11; I; IRIX 5.3 IP22) Netscape 4.x Macintosh 68k,Mozilla/4.05 (Macintosh; I; 68K Nav) Netscape 4.x Macintosh PowerPC,Mozilla/4.05 (Macintosh; I; PPC Nav) Netscape 4.x SunOS 4.1.4,Mozilla/4.05 [en] (X11; I; SunOS 4.1.4 sun4m) Version 4.08 [en]-98306,Mozilla/4.08 [en] (Win98; U ;Nav) Netscape 4.x WinNT,Mozilla/4.08 [en] (WinNT; U) Echo.com robot,Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_TrueRobot/1.4 libwww/5.2.8 Voila.fr robot,Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2 HTTrack Offline Browser,Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98) iCab MAC Web browser MAC Power PC,Mozilla/4.5 (compatible; iCab 2.5.3; Macintosh; I; PPC) OmniWeb 4.x.x Mac browser,Mozilla/4.5 (compatible; OmniWeb/4.0.5; Mac_PowerPC) OmniWeb 4.x.x Mac browser,Mozilla/4.5 (compatible; OmniWeb/4.1-beta-1; Mac_PowerPC) different IPs using the HTTPClient library (mostly link checking),Mozilla/4.5 RPT-HTTPClient/0.3-2 RuralNet Internet Services,Mozilla/4.5 [en]C-CCK-MCD {RuralNet} (Win98; I) Netscape 4.x Linux,Mozilla/4.5b1 [en] (X11; I; Linux 2.0.35 i586) Cnet robot for Search.com (216.239.114.xx),Mozilla/4.6 [en] (http://www.cnet.com/) Netscape 4.x OS/2,Mozilla/4.61 [de] (OS/2; I) BrowseX cross-platform browser,Mozilla/4.61 [en] (X11; U; ) - BrowseX (2.0.0 Windows) Nameprotect (12.148.196.128 - 12.148.196.255) snoopbot,Mozilla/4.7 Eidetica earch and text mining spider,Mozilla/4.7 (compatible; http://eidetica.com/spider) Intelliseek (64.158.138.xx) robot,Mozilla/4.7 (compatible; Intelliseek; http://www.intelliseek.com) OffByOne Browser,Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2 WhizBang! Labs information extraction robot,Mozilla/4.7 (compatible; Whizbang) WhizBang! Labs information extraction robot,Mozilla/4.7 (compatible; WhizBang; http://www.whizbang.com/crawler) BecomeBot - Becomecom shopping search (64.124.85.xx(x)),Mozilla/4.7 [en](BecomeBot@exava.com) Exabot - exava shopping search (64.124.85.xx(x)),Mozilla/4.7 [en](Exabot@exava.com) unknown,Mozilla/4.7 [en]C-CCK-MCD {Yahoo;YIP052400} (Win95; I) http://www.ba.be robot,Mozilla/4.72 [en] (BACS http://www.ba.be) Netscpape 4.7x Caldera Open Linux Pentium III,Mozilla/4.72C-CCK-MCD Caldera Systems OpenLinux [en] (X11; U; Linux 2.2.14 i686) Netscape 4.7x Japan OSF1 alpha,Mozilla/4.75C-ja [ja] (X11; U; OSF1 V5.1 alpha) Opera 5.x Win 98,Mozilla/4.76 (Windows 98; U) Opera 5.12 [en] Netscape 4.7x FreeBSD,Mozilla/4.76 [en] (X11; U; FreeBSD 4.4-STABLE i386) Netscape 4.7x SunOS,Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u) IRIX 6.5,Mozilla/4.77C-SGI [en] (X11; U; IRIX 6.5 IP32) GigaMedia / NTT DoCoMo robot,Mozilla/5.0 Eurekster Swicki community search using SLI-Systems site search engine Mammoth,Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1 SLI Systems mammoth robot,Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1 Clush search robot,Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/) Greasemonkey RSS panel Firefox plugin,Mozilla/5.0 (compatible) GM RSS Panel X Evri search robot,Mozilla/5.0 (compatible; +http://www.evri.com/evrinid) 008 distributed crawler for 80legs,Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620 Abonti WebSearch beta robot,Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com) HitCompanies Aihit crawler,Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/) Ansearch Australian search robot,Mozilla/5.0 (compatible; AnsearchBot/1.x; +http://www.ansearch.com.au/) The Library of Congress Minerva crawler,Mozilla/5.0 (compatible; archive.org_bot/1.10.0 +http://www.loc.gov/minerva/crawl.html) Heritrix - The Internet Archive's open-source crawler (207.241.225.2xx),Mozilla/5.0 (compatible; archive.org_bot/1.13.1x http://crawler.archive.org) Heritrix - The Internet Archive's open-source crawler,Mozilla/5.0 (compatible; archive.org_bot/1.5.0-200506132127 http://crawler.archive.org) Hurricane Katrina Ask Jeeves /Teoma robot,Mozilla/5.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml) Project BanBots Perl script robot,Mozilla/5.0 (compatible; BanBots/2.0b; Fetch; +http://www.banbots.com) BecomeBot - Become.com shopping search (64.124.85.xx(x)),Mozilla/5.0 (compatible; BecomeBot/1.23; http://www.become.com/webmasters.html) BecomeBot - Become.com shopping search (64.124.85.xx(x)),Mozilla/5.0 (compatible; BecomeBot/1.xx; MSIE 6.0 compatible; http://www.become.com/webmasters.html) BecomeBot - Become.com shopping search (64.124.85.xx(x)),Mozilla/5.0 (compatible; BecomeBot/2.0beta; http://www.become.com/webmasters.html) BecomeBot - Become.com shopping search (64.124.85.xx(x)),Mozilla/5.0 (compatible; BecomeBot/2.x; MSIE 6.0 compatible; http://www.become.com/site_owners.html) BecomeBot - Become.com shopping search (64.124.85.xx(x)),Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.become.co.jp/site_owners.html) BlogRefsBot.com blog robot,Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers) Pressemitteilungen Webservice RSS / news crawler (Germany),Mozilla/5.0 (compatible; Bot; +http://pressemitteilung.ws/spamfilter BuzzRanking internet content analysis,Mozilla/5.0 (compatible; BuzzRankingBot/1.0; +http://www.buzzrankingbot.com/) Charlotte indexing spider for Searchme / Wikiseek,Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com) Charlotte indexing spider for Searchme / Wikiseek,Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/) Unknown graphics crawler or downloading agent from Yamana Laboratory - Waseda Univerity Japan (133.9.238.xx),Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp) Custo web site spidering tool (link checking),Mozilla/5.0 (compatible; Custo 3 (Netwu.com); Windows NT 5.1) De.com German travel related search via Amazon Web Services,Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com) Diffbot beta - RSS and news feed crawler,Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com) DNS-Digger - DNS server neighbourhood search,Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com) DNS-Digger - DNS server neighbourhood search,Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) Earthcom (Czech Republic) search robot (194.108.39.xx),Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info) enter4u / Earthcom.info search (Czech Republic),Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu) Prague Faculty of Mathematics and Physics using Egothor open source crawler,Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/) Exalead (France) search robot (193.47.80.xx),Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot) TheFind.com - Shopping search robot,Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/main/CrawlerFAQs.fhtml) Galbot tagging robot (beta) - Denmark,Mozilla/5.0 (compatible; Galbot/1.0; +http://www.galbot.com/bot.html) Geneva Single-Site Search Engine used by Healthdash health search,mozilla/5.0 (compatible; genevabot http://www.healthdash.com) Paros - a Java based HTTP/HTTPS proxy,Mozilla/5.0 (compatible; Google Desktop) Paros/3.2.12 Google robot,Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) Unknown robot using Heritrix,mozilla/5.0 (compatible; heritrix/1.0.4 http://innovationblog.com) The Stanford University InfoLab robot using Heritrix,Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) Newstin news feed search using Heritrix,Mozilla/5.0 (compatible; heritrix/1.12.1 +http://newstin.com/) Page-store.com vertical search via Amazon Web Services,Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) Page-store.com vertical search via Amazon Web Services,Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) [email:paul@page-store.com] Heritrix Internet Archive's open-source web project ,mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org) Chepi Beta search Spain (194.116.240.1xx) using Heritrix,Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net) Truveo data mining robot using Heritrix,Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/) L3S WebCrawling Project (Germany) using Heritrix,Mozilla/5.0 (compatible; heritrix/1.5.0 http://www.l3s.de/~kohlschuetter/projects/crawling/) Pandora Internet Archive crawler (Australia) using Heritrix,Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 http://pandora.nla.gov.au/crawl.html) WORIO (beta) search for computer scientists and programmers using Heritrix open-source crawler,Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/) greatarea.com website collection project using Heritrix,Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) hanzo:web social web archiving service,Mozilla/5.0 (compatible; Heritrix/1.8.0 http://www.hanzoarchives.com) Accelobot - Accelovation Market Discovery software robot,Mozilla/5.0 (compatible; heritrix/1.x.x +http://www.accelobot.com) Unknown AOL robot using Heritrix,Mozilla/5.0 (compatible; heritrix/2.0.0-RC1 +http://www.aol.com) Hermits Search.com - Products and service search robot,Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com) IsMySiteUp? - Online website monitoring service,Mozilla/5.0 (compatible; http://www.IsMySiteUp.Net/bot/ ) UptimeAuditor - real time web monitoring,Mozilla/5.0 (compatible; http://www.UptimeAuditor.com/bot/ ) Hyperix vertical search crawler,Mozilla/5.0 (compatible; HyperixScoop/1.3; +http://www.hyperix.com) Unknown robot (reads robots.txt) from chinatelecom (219.142.78.xx),Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0) ID-Search.org - Russian search project,Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) Najdi.si (Slovenia) search using Interseek/Web Interseek/API Search Engine,Mozilla/5.0 (compatible; InterseekWeb/3.x) hanzo:web social web archiving service,Mozilla/5.0 (compatible; Jim +http://www.hanzoarchives.com) Konqueror 2.0.x X11,Mozilla/5.0 (compatible; Konqueror/2.0.1; X11); Supports MD5-Digest; Supports gzip encoding Konqueror 2.1.x X11,Mozilla/5.0 (compatible; Konqueror/2.1.1; X11) Konqueror 2.2.x,Mozilla/5.0 (compatible; Konqueror/2.2.2) Konqueror 2.2.x Linux,Mozilla/5.0 (compatible; Konqueror/2.2.2; Linux 2.4.14-xfs; X11; i686) Exalead (France) search robot (193.47.80.xx),Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) Lemur Consulting LemIR spider,Mozilla/5.0 (compatible; LemSpider 0.1) Linksmanager.com online link checking service,Mozilla/5.0 (compatible; LinksManager.com_bot http://linksmanager.com/linkchecker.html) LinkStash Bookmark Manager,Mozilla/5.0 (compatible; LinkStash Bookmark Manager; http://www.xrayz.co.uk/) Mojeek Search Preview robot (217.155.205.xx),Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html) Joomla!/Mambo component - MosBookmarks (bot) link checking,Mozilla/5.0 (compatible; MOSBookmarks/v2.6-Plus; Link Checker) PodTech entertainment and video network crawler,Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net) onet.pl Szukaj (Search) robot (213.180.128.1xx),Mozilla/5.0 (compatible; OnetSzukaj/5.0; http://szukaj.onet.pl) Pagestacker online bookmark service,Mozilla/5.0 (compatible; PagestackerBot; http://www.pagestacker.com) PalmeraBot - Links24h.com search engine robot,Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Version 0.001 FeedMo feed search (Japan) using Pear HTTP,Mozilla/5.0 (compatible; PEAR HTTP_Request class; http://feed.moo.jp/) PHONifier mobile access to web content,Mozilla/5.0 (compatible; Phonifier; +http://www.phonifier.com) pmoz.info ODP link checking bot,Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm) Pogodak search (Slovenia) robot via Interseek,Mozilla/5.0 (compatible; pogodak.ba/3.x) Pogodak search (Slovenia) robot via Interseek,Mozilla/5.0 (compatible; Pogodak.hr/3.1) Proximic Publisher Widget - RSS and news content generator,Mozilla/5.0 (compatible; Proximic crawler; +http://www.proximic.com/en/about-us/contact-us.html) ProgramacionWeb.net PWeBot crawler (Argentina),Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php) Quantcast - Open Internet Ratings Service,Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com) robtex - Multi-RBL check and AS-numbercheck,Mozilla/5.0 (compatible; robtexbot/1.0; http://www.robtex.com/ ) ScoutJet (Blekko) search web crawler,Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/) Scrub the web robot (66.93.156.xx),Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/) Shunixbot (France) beta / test semantic web indexing robot,Mozilla/5.0 (compatible; ShunixBot/1.x.x +http://www.shunix.com/robot.htm) Shunixbot (France) beta / test semantic web indexing robot,Mozilla/5.0 (compatible; ShunixBot/1.x; http://www.shunix.com/bot.htm) Skreemr - Audio search engine,Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) Snap Firefox Search Plugin,Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 Spurl.net bookmark service & search engine (84.40.30.xxx),Mozilla/5.0 (compatible; SpurlBot/0.2) Summize - Opinion and review search robot,Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) Syclik Control web content management system,Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;) Synoo web directory robot,Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html) Theophrastus Internet Spider for a basic search engine project,Mozilla/5.0 (compatible; Theophrastus/x.x; http://users.cs.cf.ac.uk/N.A.Smith/theophrastus.php) Interseek - Java search engine technology used for Pogodak search,Mozilla/5.0 (compatible; TridentSpider/3.1) WiseGuys robot,Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) Webdunia search (India) robot,Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) phpwebbrain online bookmark service (Germany),Mozilla/5.0 (compatible; Windows NT 5.0; phpwebbrainBot/0.1 - http://www.monsterli.ch/phpwebbrain/) WORIO (beta) search for computer scientists and programmers using Heritrix open-source crawler,Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com) Kathune spider for World of Warcraft guild data. Used to power WoW Lemmings,Mozilla/5.0 (compatible; WoW Lemmings Kathune/2.0;http://www.wowlemmings.com/kathune.html) eXternalTest - Server and online services monitoring,Mozilla/5.0 (compatible; XTbot/1.0v; +http://www.externaltest.com) Yahoo / Inktomi search robot,Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Inktomi robot (202.160.180.xxx) for Yahoo China,Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) Inktomi robot for Yahoo (via 66.196.xx.xxx),Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Yesup Seo - Toronto SEO Service,Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html) Yoono - community based search (193.110.140.xxx / 194.0.179.[x]xx),Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) Youdao search (China) robot,Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) Zenbot robot for the Southern African Zen search service,Mozilla/5.0 (compatible; Zenbot/1.3; +http://zen.co.za/webmasters/) Powerset Natural Language Search crawler (under development) using Heritrix via Amazon Web Services,"Mozilla/5.0 (compatible; zermelo +http://www.powerset.com) [email:paul@page-store.com,crawl@powerset.com]" Heritrix - The Internet Archive's open-source crawler (207.241.225.2xx),Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +http://www.archive-it.org) Wayback Machine Internet Archive crawler,Mozilla/5.0 (compatible;archive.org_bot/heritrix-1.9.0-200608171144 +http://pandora.nla.gov.au/crawl.html) FindITAnswers - Search engine for software developers,Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+http://search.it-influentials.com/bot.htm) Mainseek search (Poland) robot,Mozilla/5.0 (compatible;MAINSEEK_BOT) MozShot - Technical demo to take screenshot of any URL,Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; http://mozshot.nemui.org/) Mozilla Firefox 3.0 beta (Gran Paradiso) for MacOS,Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b5) Gecko/2008032619 Firefox/3.0b5 Chimera browser (Mozilla/Gecko engine) - now Camino Mac PowerPC,Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.0.1) Gecko/20021219 Chimera/0.6 Camino browser (Mozilla/Gecko engine) - ex Chimera Mac PowerPC,Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.0.1) Gecko/20030306 Camino/0.7 OmniWeb 5.x.x Mac OS X browser,Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US) AppleWebKit/xx (KHTML like Gecko) OmniWeb/v5xx.xx Safari 1.2x browser (Mozilla/Gecko engine) MAC OS X,Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/xxx.x (KHTML like Gecko) Safari/12x.x Netscape 6.x Mac PowerPC,Mozilla/5.0 (Macintosh; U; PPC; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1 unknown robot from gw.ocg-corp.com,Mozilla/5.0 (research@mediatrec.com) Sage - RSS and Atom feed reader extension for Mozilla Firefox,Mozilla/5.0 (Sage) Inktomi (Hotbot-Lycos - NBCi etc.) robot,Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Inktomi (Hotbot-Lycos - NBCi etc.) robot,Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Opera 5.x SunOS,Mozilla/5.0 (SunOS 5.8 sun4u; U) Opera 5.0 [en] Twiceler experimental web crawler,Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Some spambot from Romania (82.208.139.1xx & 86.123.65.xx) - Maybe email harvesting,Mozilla/5.0 (Version: xxxx Type:xx) GenieKnows.com search robot (64.5.245.xx / 64.5.220.xxx),Mozilla/5.0 (wgao@genieknows.com) Netscape 6.x Win98,Mozilla/5.0 (Windows; U; Win98; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1 Firebird browser (Mozilla/Gecko engine) - ex Phoenix Win98,Mozilla/5.0 (Windows; U; Win98; en-US; rv:x.xx) Gecko/20030423 Firebird Browser/0.6 Beonex Communicator browser (Mozilla/Gecko engine),Mozilla/5.0 (Windows; U; Win9x; en; Stable) Gecko/20020911 Beonex/0.8.1-stable Healthline health related search robot (72.5.115.xx),Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obeys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com Google Chrome browser based on WebKit (Safari),"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/0.2.153.1 Safari/525.19" Flock web browser built on Mozilla technologies,Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060731 Firefox/1.5.0.5 Flock/0.7.4.1 Megaupload Mega Manager - Download manager plugin for Firefox,Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.4/Megaupload x.0 Orca browser - based on Gecko,Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008092215 Firefox/3.0.1 Orca/1.1 beta 3 Firefox browser (Mozilla/Gecko engine) - ex Firebird WinXP,Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.x Firebird browser (Mozilla/Gecko engine) - ex Phoenix WinXP,Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.xx) Gecko/20030504 Mozilla Firebird/0.6 Mnenhy - enhanced mail & news Mozilla based browser,Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.xxx) Gecko/20041027 Mnenhy/0.6.0.104 Voila.fr robot,Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com) Voila.fr robot,Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com) Mozilla Firefox 3.0 beta (Gran Paradiso) for Win,Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5 Tiscali Communicator - Online services suite,Mozilla/5.0 (Windows; U;XMPP Tiscali Communicator v.10.0.1; Windows NT 5.1; it; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3 Healthline health related search robot (72.5.115.xx),Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@health Healthline health related search robot (72.5.115.xx),Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com Kazehakase - Gecko based browser (Japan),Mozilla/5.0 (X11; Linux i686; U;rv: 1.7.13) Gecko/20070322 Kazehakase/0.4.4.1 Netscape 6.x Linux,Mozilla/5.0 (X11; U; Linux 2.4.2-2 i586; en-US; m18) Gecko/20010131 Netscape6/6.01 SeaMonkey browser suite (ex Mozilla) on Linux,Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.8.0.2) Gecko/20060309 SeaMonkey/1.0 Epiphany (Mozilla/Gecko engine) browser Linux,Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.7.6) Gecko/20050405 Epiphany/1.6.1 (Ubuntu) (Ubuntu package 1.0.2) Nautilus (developed by Eazel.com) 1.x Browser Linux,Mozilla/5.0 (X11; U; Linux i686; en-US; Nautilus/1.0Final) Gecko/20020408 Mozilla (Gecko) 0.9x browser Linux,Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3) Gecko/20010801 Spinn3r social network crawler,Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130 Phoenix 0.3 browser (Mozilla/Gecko engine) - now Firebird Linux,Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20021007 Phoenix/0.3 Epiphany (Mozilla/Gecko engine) browser Linux,Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040413 Epiphany/1.2.1 Snap Firefox Search Plugin,Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot Bon Echo Alpha - developer preview of future Firefox browser,Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061129 BonEcho/2.0 IceWeasel - the GNU version of the Firefox browser,Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/20061205 Iceweasel/2.0.0.1 (Debian-2.0.0.1+dfsg-2) Mozilla Firefox 3.0 beta (Gran Paradiso) for Linux,Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9a8) Gecko/2007100619 GranParadiso/3.0a8 Galeon 1.x Browser Linux,Mozilla/5.0 Galeon/1.0.2 (X11; Linux i686; U;) Gecko/20011224 gURLChecker - GNOME link checking tool,Mozilla/5.0 gURLChecker/0.x.x (Linux) URL Spider - used by usww.net,Mozilla/5.0 URL-Spider W8net spider,Mozilla/5.0 usww.com-Spider-for-w8.net GenieKnows.com search robot (64.5.245.xx / 64.5.220.xxx),Mozilla/5.0 wgao@genieknows.com axxus.de German business directory,Mozilla/5.0 whoiam [http://www.axxus.de/] Yuntis : Collaborative Web Resource Categorization and Ranking Project robot,Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) MetaExplorer project's MetaQuerier robot,MQbot metaquerier.cs.uiuc.edu/crawler MetaExplorer project's MetaQuerier robot,MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.uiuc.edu) MS Frontpage 4.x,MSFrontPage/4.0 Some faked UA - maybe for a download manager,MSIE 4.0 (Win95) unknown robot from gw.ocg-corp.com (209.126.176.x),MSIE-5.13 (larbin@unspecified.mail) MSN media search robot,msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) Windows Live product search (Beta) robot,msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm) MSN Search robot - 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx,MSNBOT/0.xx (http://search.msn.com/msnbot.htm) MSN Search robot - 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx,msnbot/x.xx ( http://search.msn.com/msnbot.htm) Microsoft search for mobiles,MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) MSN Search robot - 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx,MSNPTC/1.0 Microsoft proxy server,MSProxy/2.0 MacEdition CodeBitch link checking,MSRBOT Microsoft MSRBot,MSRBOT (http://research.microsoft.com/research/sv/msrbot) StreamBox VCR user agent,"Mulder, VCR-1.0" Multiblocker (Fantomaster) anonymity software user,multiBlocker browser MultiCrawler for DERI Galway's Semantic Web Search Engine cluster,multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html) Virginia Tech Digital Library Research Laboratory robot,MultiText/0.1 SoMusical! musical directory link checking,MusicWalker2.0 ( http://www.somusical.com) Unknown bad bot from diff. Taiwanese IPs,MVAClient Windows HTTP Services (WinHTTP),My WinHTTP Connection unknown user robot (24.124.34.42),myDaemon GetRight download manager,MyGetRight/1.0.0 GetRight download manager,MyGetRight/1.0b Mylinea France web catalogue crawler,Mylinea.com Crawler 2.0 VDOG - SEO webdirecory (Germany) link checking,mylinkcheck/1.02 Blogbot (Germany) robot,Naamah 1.0.1/Blogbot (http://blogbot.de/) Blogbot (Germany) robot,Naamah 1.0a/Blogbot (http://blogbot.de/) Naver Japan / Korea robot ,NABOT/5.0 Naver Japan / Korea robot ,nabot_1.0 Badbot searching for Wordpress wp-login.php,NameOfAgent (CMS Spider) naoFavicon4IE,naoFavicon4IE/1.xx Unknown spambot / harvester from diff. IPs,NASA Search 1.0 Nationaldirectory spider,NationalDirectory-WebSpider/1.3 Nationaldirectory spider,NationalDirectoryAddURL/1.0 Naver Japan / Korea robot,NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Naver Japan / Korea robot,NaverBot_dloader/1.5 Navisso closed beta robot (69.41.162.1xx),NavissoBot Navisso closed beta robot (69.41.162.1xx),NavissoBot/1.7 (+http://navisso.com/) Vias Information Archival robot,NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) Nebulla.info distributed crawler (Germany),Nebullabot/2.2 (http://bot.nebulla.info) NEC Researchindex robot - now CiteSeer.IST scientific document index,NEC Research Agent -- compuman at research.nj.nec.com rcn.com user agent ? NEC Researchindex robot ?,NEC-Hayek/1.0 Net Seekr search robot,Net-Seekr Bot/Net-Seekr Bot V1 (http://www.net-seekr.com) NetAnts download manager,NetAnts/1.2x NetComplete IE browser package,NETCOMplete/x.xx Netinfo.bg search (Bulgaria) robot,NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) Netlookout internet notifier,NetLookout/2.24 Netluchs (Germany) search (193.164.8.xx),Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de) NetMechanic link checker,NetMechanic Vx.0 NewsGator NetNewsWire - Mac RSS feed reader,NetNewsWire/2.x (Mac OS X; http://ranchero.com/netnewswire/) unknown InCom (216.0.107.xx) robot,NetNoseCrawler/v1.0 Netprospector metasearch software,Netprospector JavaCrawler Netpumper download manager,NetPumper/x.xx Look.com robot (209.87.232.x),NetResearchServer(http://www.look.com) IncyWincy search engine using DMOZ database,NetResearchServer/x.x(loopimprovements.com/robot.html) NetSeer search (beta) crawler via Amazon Web Services - see also Teemer,NetSeer/Nutch-0.9 (NetSeer Crawler; http://www.netseer.com; crawler@netseer.com) Wirtualna Polska / Netsprint search (Poland) robot,NetSprint -- 2.0 NetWhat Search crawler (69.9.167.1xx),NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com) Netzippy robot,NetZippy unknown,NeuralBot/0.2 Unknown (12.238.4.xxx) attbi.com client robot,newsearchengine (ThisUser@unspecified.mail) FetchLinks plugin for NewsGator RSS reader,NewsGator FetchLinks extension/0.2.0 (http://graemef.com) NewsGator online RSS reader,NewsGatorOnline/2.0 (http://www.newsgator.com; 1 subscribers) Eliyon Crawler for Business People Search,NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot) Nextopia crawler,NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x find your keywords - semantic search (Germany) robot,NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; ) Exalead (France) search robot (193.47.80.xx),NG/1.0 Exalead Websearch image crawler (193.47.80.xx),NG/4.0.1229 Unknown UA from PlanetLab distributed network,nicebot NicoZone childsafe search robot ,NICO/1.0 "Nikita the Spider - Online HTML validation , link checking",Nikita the Spider (http://NikitaTheSpider.com/) Experimental LSI (?) robot from 140.233.69.xx (Middlebury.edu),NITLE Blog Spider/0.01 Download manager,Nitro Downloader 1.x (www.klsofttools.com) Noago spider,Noago Spider telefonica.es user robot,Nocilla/1.0 Google WAP robot,Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com Botmobi crawler for Find.mobi mobile search,Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://find.mobi/bot.html abuse@mtld.mobi) YahooSeeker/M1A1-R2D2 - Yahoo mobile web crawling robot,Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) Google WAP proxy,Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0) Nokodo public beta search robot (67.18.222.xx),NokodoBot/1.x (+http://nokodo.com/bot.htm) Burf.com UK Search Engine robot,Norbert the Spider(Burf.com) noXtrum search robot (Spain),noxtrumbot/1.0 (crawler@noxtrum.com) Noyona job search (preview),noyona_0_1 Nameprotect copyright search robot (24.177.134.x),NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com) Nameprotect copyright search robot (24.177.134.x),NPBot (http://www.nameprotect.com/botinfo.html) Nameprotect copyright search robot (24.177.134.x),NPBot-1/2.0 Nsauditor Network Security Auditor,Nsauditor/1.x NetShow Media Player = Windows Media Player 10,NSPlayer/10.0.0.xxxx WMFSDK/10.0 Nsyght social search application,nsyght.com/Nutch-1.0-dev (nsyght.com; Nsyght.com) Nsyght social search application,nsyght.com/Nutch-x.x (nsyght.com; search.nsyght.com) NTT Directory robot,nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) Nucleus CMS SiteList link managing plugin,Nucleus SiteList LinkChecker/1.1 nuSearch spider (84.9.136.xxx),nuSearch Spider www.nusearch.com (compatible; MSIE 4.01) nuSearch spider (84.9.136.xxx),NuSearch Spider (compatible; MSIE 6.0) nuSearch spider (84.9.136.xxx),NuSearch Spider www.nusearch.com Nutch open source robot,Nutch Picapage search for handheld devices using Nutch,Nutch crawler/Nutch-0.9 (picapage.com; admin@picapage.com) Ayell Euronet business directory robot using Nutch,Nutch/Nutch-0.9 (Eurobot; http://www.ayell.eu ) Netsweeper content filtering engine (66.207.120.2xx) powered by Nutch,NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) Nutch open source robot,NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net) Robot from University of Washington Computer Science & Engineering (128.208.6.2xx),NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu) Amazon Elastic Compute Cloud (Amazon EC2) robot,NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com) Nutch open source robot,NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net) Unknown robot using Nutch (maybe private crawling) via Cox network (70.187.130.25x),nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com) Unknown crawler from University of Washington - Computer science,NutchVinegarCrawl/Nutch-0.8.1 (Vinegar; http://www.cs.washington.edu; eytanadar at gmail dot com) Weblog bookwatch robot,obidos-bot (just looking for books.) Objects Search robot,ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) Objects Search robot,ObjectsSearch/0.0x (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) Cobion Germany Brand Protection Services robot,oBot ((compatible;Win32)) GlobalSpec Engineering Search robot (66.194.55.xxx),Ocelli/1.x (http://www.globalspec.com/Ocelli) Octopus download manager,Octopus Octora blog or RSS information crawler - beta (66.228.114.xx),Octora Beta - www.octora.com Octora RSS feed search,Octora Beta Bot - www.octora.com Meta Products Offlinebrowser,Offline Explorer 1.* Claymont robot / Internetseer Web Site Monitoring,OliverPerry OmniExplorer car & shopping search (64.62.175.xxx),OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet CategorizerOmniExplorer http://www.omni-explorer.com/ car & shopping search (64.62.175.xxx) OmniExplorer car & shopping search (64.62.175.xxx),OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler OmniExplorer car & shopping search (64.62.175.xxx),OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler OmniExplorer car & shopping search (64.62.175.xxx),OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer onsearch.de German web directory link checking ,onCHECK Linkchecker von www.scientec.de fuer www.onsinn.de onsearch.de German web directory link checking ,"onCHECK-Robot, www.onsearch.de" onet.pl Szukaj (Search) robot (213.180.128.1xx),Onet.pl SA- http://szukaj.onet.pl Dead-Links.com link validation spider,online link validator (http://www.dead-links.com/) Online24 shopping portal (Germany) link checking,"Online24-Bot (Version: 1.0x, powered by www.online24.de)" OntoSpider - Dutch robot for a research project. (195.11.244.xx),OntoSpider/1.0 libwww-perl/5.65 SeetooZ search crawler,OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) OpenAcoon open source search engine (used by Acoon search),OpenAcoon v4.0.x (www.openacoon.de) Openfind.com.tw robot,Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) Openfind.com.tw robot,Openfind data gatherer- Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) Openfind.com.tw robot,Openfind Robot/1.1A2 open i search robot - search engine in development,OpenISearch/1.x (www.openisearch.com) Opentagger social bookmarking system,OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) OpenText crawler,OpenTextSiteCrawler/2.9.2 OpenWebSpider - Open Source web search engine,OpenWebSpider/0.x.x (http://www.openwebspider.org) OpenWebSpider - Open Source web search engine,OpenWebSpider/x Opera 5.0 Linux,Opera/5.0 (Linux 2.0.38 i386; U) [en] Opera 5.11 faked WinME referer,Opera/5.11 (Windows ME; U) [ru] Opera 5.12 Win98,Opera/5.12 (Windows 98; U) [en] unknown robot from gw.ocg-corp.com (209.126.176.x),Opera/6.01 (larbin@unspecified.mail) Opera 6.x- Mandrake Linux,Opera/6.x (Linux 2.4.8-26mdk i686; U) [en] Opera 6.x WinNT,Opera/6.x (Windows NT 4.0; U) [de] Opera 7.x WinXP,Opera/7.x (Windows NT 5.1; U) [en] Opera 8.x (Beta) WinXP,Opera/8.xx (Windows NT 5.1; U; en) Opera 9 (Beta) Browser,Opera/9.0 (Windows NT 5.1; U; en) Opera 9 (final),Opera/9.00 (Windows NT 5.1; U; de) Opera browser 9.6x on WinXP (Presto = Operas rendering engine),Opera/9.60 (Windows NT 5.1; U; de) Presto/2.1.1 Opidoo Search Belgium robot (62.4.83.xxx),OpidooBOT (larbin2.6.3@unspecified.mail) Open Wave Phone Simulator SDK,OPWV-SDK UP.Browser/7.0.2.3.119 (GUI) MMP/2.0 Push/PO Oracle Application Server cache,Oracle Application Server Web Cache 10g Oracle interMedia Text - Text and web documents indexing,Oracle iMTCrawler Oracle Search,Oracle Ultra Search Orangeslicer semantic search (Beta) Germany (193.201.52.1xx),OrangeSpider Orbiter - DailyOrbit search spider,Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) Orca browser - based on Gecko,Orca Browser (http://www.orcabrowser.com) Marketscore (was Netsetter) internet accelerator,OSSProxy 1.3.305.321 (Build 305.321 Win32 en-us)(Dec 21 2005 16:30:54) Unknown robot from Chinanet (220.181.8.xxx),OutfoxBot/0.x (For internet experiments; http://; outfox.agent@gmail.com) Unknown robot from Chinanet (60.191.80.1),OutfoxMelonBot/0.5 (for internet experiments; http://; outfoxbot@gmail.com) Overture/Fast/Alltheweb crawler (66.77.73.xxx),Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Unknown robot from 198.169.127.xx (innovationplace.com),OWR_Crawler 0.1 Ozelot - Flying Cat's search engine robot (Germany),ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat.de) PADLibrary.com - PAD file software robot for FindFiles.com,PADLibrary Spider Pagebites job search crawler,PageBitesHyperBot/600 (http://www.pagebites.com/) Pagebull visual search engine,Pagebull http://www.pagebull.com/ Pagestacker online bookmark service,Pagestacker Bot Secure Computing SmartFilter Tools - malware crawler,page_verifier (http://www.securecomputing.com/goto/pv) Downloadmanager ?,PagmIEDownload CFC crawler used by Italian academic and research network (GARR),parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1 http://www.ianett.com robot,ParaSite/1.0b (http://www.ianett.com/parasite/) Patsearch (Germany) robot,Patwebbot (http://www.herz-power.de/technik.html) Pavuk web downloading program for Unix,pavuk/0.9pl29b i686-pc-linux-gnu Some site scanning tool via diff. IPs- i.e.: - cox.net (68.4.xxx.xxx),PBrowse 1.4b Post.sk / Eurotel.sk robot,pd02_1.0.0 pd02_1.0.0@dzimi@post.sk Pear HTTP_Request PHP extension package,PEAR HTTP_Request class ( http://pear.php.net/ ) Peerbot - favicon search robot,PEERbot www.peerbot.com Unknown robot from 64.5.48.xxx (Plethoric.net),PeopleChat/Search_Engine Some site scanning tool via diff. IPs,PEval 1.4b diff. IPs / services,PHP/3.x.xx diff. IPs / services,PHP/4.0.4pl1 diff. IPs / services- i.e.: -NTT/Verio Inc. link checker,PHP/4.0.6 diff. IPs / services- i.e.: - Phenominet.com link checking,PHP/4.1.1 diff. IPs / services- i.e.: - 209.114.200.xx = MyNetCrawler link checking - 216.139.207.xxx = Mixcat crawler,PHP/4.1.2 Pico Search robot,PicoSearch/1.0 Piffany targeted search web spider,Piffany_Web_Scraper_v0.x Piffany targeted search web spider,Piffany_Web_Spider_v0.x Whois Source domain name information robot (66.249.26.xx)- s.also: - SurveyBot,PigeonBot1.0 BETA PingALink website monitoring,PingALink Monitoring Services 1.0 PingALink website monitoring,PingALink Monitoring Services 1.0 (http://www.pingalink.com) Pingdom web site monitoring,Pingdom GIGRIB (http://www.pingdom.com) pipeline search (DMOZ based) search robot (24.106.39. xxx),pipeLiner/0.3a (PipeLine Spider;http://www.pipeline-search.com/webmaster.html; webmaster'at'pipeline-search.com) pipeline search (DMOZ based) search robot (24.106.39. xxx),pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) Pita crawler,Pita Private user-agent via Hurricane Electric Internet Services,Pizilla++ ver 2.45 Portaljuice spider,PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com) Plagger - pluggable RSS/Atom feed aggregator written in Perl,Plagger/0.x.xx (http://plagger.org/) unknown ucsd.edu robot,PlagiarBot/1.0 Plantynet web filtering services - Blacklist DB robot,PlantyNet_WebRobot_V1.9 dhkang@plantynet.com Unknown UA from 66.220.23.2xx,plinki/0.1 (you got plinked! (thats a good thing..); http://www.plinki.com; crawl@plinki.com) Pluck RSS feed crawler,PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1 subscribers) Pluggd Podcast search engine,Pluggd/Nutch-0.9 (automated crawler http://www.pluggd.com;support at pluggd dot com) Yutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan,Pockey-GetHTML/4.12.0 (Win32; GUI; ix86) Yutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan,Pockey-GetHTML/x.xx Yutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan,Pockey/x.x.x Yutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan,Pockey7.x.x(WIN32GUI) HTTP user-agent for POE (portable networking framework for Perl ),POE-Component-Client-HTTP/0.64 (perl; N; POE; en; rv:0.640000) ThePlanet/jaja-jak-globusy.com Google Adsense refferer spam bot from 70.85.116.* / 70.84.128.xxx / 70.85.193.xxx,Poirot Polybot webcrawler,polybot 1.0 (http://cis.poly.edu/polybot/) Dir.com / Iliad French recherche robot,Pompos/1.x http://dir.com/pompos.html Iliad / Free French recherche robot,Pompos/1.x pompos@iliad.fr Popdex - web site popularity crawler,Popdexter/1.0 Unknown spam bot / harvester (63.223.10.***),Port Huron Labs PortalB (now Alacra search) spider,PortalBSpider/2.0 (spider@portalb.com) portalmmm IMode mobile browser,portalmmm/2.0 S500i(c20;TB) Yahoo (66.94.237.1xx / 216.109.121.xx) favorites tracking robot,PostFavorites Potbot : A simple IRC bot written in Perl,potbot 1.0 Project Rialto - data mining development project,PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com) CUPS robot for AT&T Privacy Bird Privacy Preferences (P3P) enhancements,PrivacyFinder Cache Bot v1.0 CUPS robot for AT&T Privacy Bird Privacy Preferences (P3P) enhancements,PrivacyFinder/1.1 Privoxy web proxy,Privoxy/3.0 (Anonymous) Some site scanning tool from diff. IPs- i.e.: - 67.99.33.x (lightningcon.broadwing.net),Production Bot 0116B Some site scanning tool from diff. IPs- i.e.: - 216.232.64.xx (telus.net),Production Bot 2016B Some site scanning tool from diff. IPs- i.e.: - 141.154.181.xxx (east.verizon.net),Production Bot DOT 3016B Some spam bot,Program Shareware 1.0.2 unknown,Progressive Download unknown,Progressive Download HTTP check XP5 robot,Project XP5 [2.03.07-111203] Answerchase PROve Answerbot,PROve AnswerBot 4.0 ProWebguide robot,ProWebGuide Link Checker (http://www.prowebguide.com) Picsearch robot (62.119.21.13x),psbot/0.1 (+http://www.picsearch.com/bot.html) Some site scanning tool via diff. IPs- i.e.: QWest Net,PSurf15a 11 Some site scanning tool via diff. IPs- i.e.: Optonline net (24.191.xxx.xxx),PSurf15a 51 Some site scanning tool via diff. IPs- i.e.: - choiceone.net (216.153.xxx.xxx) - attbi.com (12.250.xxx.xxx) - optonline.net (24.191.xxx.xxx),PSurf15a VA Unknown website grabbing / ripping for unknown purposes from 208.66.195.x - Digitalinfinity.org Russia,psycheclone Some robot from Stanford University (171.64.75.xxx = PubCrawl.Stanford.EDU),PubCrawl (pubcrawl.stanford.edu) Parallel URL Fetcher downloading tool,puf/0.91beta6a (Linux 2.2.18; i686) Parallel URL Fetcher downloading tool,puf/0.93.2a (Linux 2.4.18; i686) WebarooBot - Webaroo web site search / theme based downloading tool (64.124.122.2xx),pulseBot (pulse Web Miner) PureSight Internet content filter,PureSight Puxa Rapido download manager,PuxaRapido v1.0 ProgramacionWeb.net PWeBot link checking (Argentina),PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php) Fast Search robot (using PycURL Python component- s. below),PycURL PycURL - Python interface to libcurl,PycURL/7.xx.x Python URL fetcher - robot used by Naver Japan/Korea,Python-urllib/1.1x Python URL fetcher - robot used by Google,Python-urllib/2.0a1 Qango.com Web Directory robot,Qango.com Web Directory (http://www.qango.com/) QEAVis: Quantitative Evaluation of Academic Websites Visibility using Nutch,QEAVis Agent/Nutch-0.9 (Quantitative Evaluation of Academic Websites Visibility; http://nlp.uned.es/qeavis Quepasa!com (Latin American search) robot,QPCreep Test Rig ( We are not indexing- just testing ) Quepasa!com (Latin American search) robot,QuepasaCreep ( crawler@quepasacorp.com ) Quepasa!com (Latin American search) robot,QuepasaCreep v0.9.1x QueryN Metasearch robot,QueryN Metasearch Blacktrees Quicksilver helper application for Mac,"Quicksilver (Blacktree,MacOSX)" Quicktime for Macintosh,QuickTime\xaa.7.0.4 (qtver=7.0.4;cpu=PPC;os=Mac 10.3.9) Qweerybot for the Qweery search engine (in development) - Netherland,QweeryBot/3.01 ( http://qweerybot.qweery.nl) Qweerybot for the Qweery search engine (in development) - Netherland,Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com) Radian6 RSS feed comment crawler,R6_CommentReader_(www.radian6.com/crawler) Radian6 Rss feed crawler,R6_FeedFetcher_(www.radian6.com/crawler) gigaBaz - the brainbot (Germany) robot,rabaz (rabaz at gigabaz dot com) DAUMOA - Daum search Korea robot (211.115.109.xxx),RaBot/1.0 Agent-admin/phortse@hanmail.net Bot Provider for the All Womans Bot Service?,Rainbot1.1 Intersearch.de (was www.intersearch.de) robot (Germany),ramBot xtreme x.x giveRAMP Search Engine robot (64.69.43.1xx),RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveramp.com/bot.html; support@giveRAMP.com) giveRAMP Search Engine robot (64.69.43.1xx),RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) Rank Exec reciprocal link checking,Rank Exec (rankexec.com) Reciprocal Link Manager 1.x/bot Rankiva website popularity robot,Rankivabot/3.2 (www.rankiva.com; 3.2; vzmxikn) Innova/IBM Rational SiteCheck - Rational robot,Rational SiteCheck (Windows NT) Unknown spider from Raytheon Company - maybe Raytheon High Speed Guard proxy,RAYSPIDER/Nutch-0.9 Read A Blog - RSS feed and blog search engine,ReadABlog Spider (compatible; 1.1; feed update; www.readablog.com) RealDownload download manager,RealDownload/4.0.0.4x The REAP Web Crawler for the REAP project,REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project) Reaper robot for SiteSearch,Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/) Reaper robot for SiteSearch,Reaper/2.0x (+http://www.sitesearch.ca/reaper) REBOL messaging language for distributed Internet apps,REBOL Core 2.x.x.x.x REBOL/View - machine independent internet client application,REBOL View 1.x.x.x.x Rebusnet software site - link / submission checking,RebusnetBot (+http://www.rebusnet.biz) Rebusnet software site - link / submission checking,RebusnetPADBot/1.5x (+http://www.rebusnet.biz) Online reciprocal link checker,reciprocal links checker (http://www.recip-links.com/) rediff.com search link checking,RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) RedCarpet crawler for Pronto price comparison search(66.179.107.1xx),RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) Der Bot for telegenetic.net's security related search (65.220.67.2xx),RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.html; lhall@telegenetic.net) Der Bot for telegenetic.net's security related search (65.220.67.2xx),RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) RedKernel Softwares robot,RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) REL Link Checker Lite free version of Web Link Validator,REL Link Checker Lite x.x unknown,RepoMonkey Bait & Tackle/v1.01 Rewebber proxy service,Rewebber/1.2 libwww-perl/5.41 Applied Semantics Auto-Categorizer for QWestDex Direct,rico/0.1 RixBot Rebol Indexer for the RIX - Rebol related search (195.204.121.xx),RixBot (http://babelserver.org/rix) StreamBox VCR user agent,RMA/1.0 (compatible; RealMedia) Real Media server acting as client,RMA/1.0 (compatible; RealMedia) Canadian Content search crawler,RoboCrawl (http://www.canadiancontent.net) Canadian Content Search (207.44.220.xx) robot,RoboCrawl (www.canadiancontent.net) FindPal Australia metasearch robot (61.68.139.xx),RoboPal (http://www.findpal.com/) PopJapanSearch robot,Robot/www.pj-search.com Experimental robot using Wget via attbi.net,Robot: NutchCrawler- Owner: wdavies@acm.org Supersnooper robot,Robot@SuperSnooper.Com Netscape Directory / DMOZ Open Directory link crawler,Robozilla/1.0 ROME - Open source Java tools for RSS and Atom feeds,Rome Client (http://tinyurl.com/64t5n) Ver: 0.9 Qualigo.de robot,Rotondo/3.1 libwww/5.3.1 different IPs using the HTTPClient library (mostly link checking),RPT-HTTPClient/0.3-x Metacarta.com (66.28.xx.xxx) robot,RRC (crawler_admin@bigfoot.com) RSS Bandit RSS/Atom reader for .NET framework,RssBandit/1.5.0.10 (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; http://www.rssbandit.org) (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; ) RSS Micro Search - RSS feed search engine,RSSMicro.com RSS/Atom Feed Robot RSSOwl embedded RSS feed reader,RSSOwl/1.2.3 2006-11-26 (Windows; U; zhtw) RSSOwl embedded RSS feed reader,RSSOwl/1.2.4 Preview Release 2007-04-15 (Windows; U; zhtw) Ykoon RssReader news feed reader,RssReader/1.0.xx.x (http://www.rssreader.com) Microsoft Windows NT 5.1.2600.0 Some site scanning tool via diff. IPs- i.e.: - dslx.net (208.35.1x.xxx) - Home.com,RSurf15a 41 Some site scanning tool via diff. IPs- i.e.: - dslx.net (208.35.1x.xxx) - Home.com,RSurf15a 51 Some site scanning tool via diff. IPs- i.e.: - dslx.net (208.35.1x.xxx) - Home.com,RSurf15a 81 rubhub blog spider,Rubbot/1.0 (+http://rubhub.com/) WebarooBot - Webaroo web site search / theme based downloading tool (64.124.122.2xx),RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html) WebarooBot - Webaroo web site search / theme based downloading tool (64.124.122.2xx),RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) unknown robot from rumours.jp (202.214.69.xxx),Rumours-Agent RX (Reflexive Search) Bar for IE,RX Bar Search & Links directory spider,S&L Spider (http://search.hirners.com/) SEO-Tools.net link checking ?,S.T.A.L.K.E.R. (http://www.seo-tools.net/en/bot.aspx) SafariBookmarkChecker for Mac OS X,SafariBookmarkChecker (+http://www.coriolis.ch/) sait robot - unknown robot from Samsung International Korea,sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) Sandcrawler robot from Microsoft (131.107.0.xx),SandCrawler - Compatibility Testing Sapphire Web Crawler from Carnegie Mellon University's Language Technologies Institute,SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) Sapphire Web Crawler from Carnegie Mellon University's Language Technologies Institute,SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) WebSavvy Directory robot,savvybot/0.2 SiteSell SBIder Nutch based crawler,SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) SiteSell SBIder Nutch based crawler,SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) Softbyte Labs Black Widow web site ripper,SBL-BOT (http://sbl.net) ScanWeb - regular expression based web page searching tool,ScanWeb ScholarUniverse - Scholarly experts search robot,ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scholaruniverse.com) URL Spider Pro (USP) used by German Schwarzmann GmbH,schwarzmann.biz-Spider_for_paddel.org+(http://www.innerprise.net/usp-spider.asp) 1X Web Browser,Science Traveller International 1X/1.0 WebWobot UK search engine robot (82.43.129.2xx),ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php) Altavista robot,Scooter-3.0.EU Altavista robot,Scooter-3.0.FS Altavista robot,Scooter-3.0.HD Altavista robot,Scooter-3.0.VNS Altavista robot,Scooter-3.0QI Altavista robot,Scooter-3.2 Altavista robot,Scooter-3.2.BT Altavista robot,Scooter-3.2.DIL Altavista robot,Scooter-3.2.EX Altavista robot,Scooter-3.2.JT Altavista robot,Scooter-3.2.NIV Altavista robot,Scooter-3.2.SF0 Altavista robot,Scooter-3.2.snippet Altavista robot,Scooter-3.3dev Altavista robot,Scooter-ARS-1.1 Altavista robot,Scooter-ARS-1.1-ih Altavista robot,scooter-venus-3.0.vns Altavista robot,Scooter-W3-1.0 Altavista robot,Scooter-W3.1.2 Altavista robot,Scooter/1.0 Altavista robot,Scooter/1.0 scooter@pa.dec.com Altavista robot,Scooter/1.1 (custom) Altavista robot,Scooter/2.0 G.R.A.B. V1.1.0 Altavista robot,Scooter/2.0 G.R.A.B. X2.0 Altavista robot,Scooter/3.3 Altavista robot,Scooter/3.3.QA.pczukor Altavista robot,Scooter/3.3.vscooter Altavista robot,Scooter/3.3_SF Altavista using Mercator robot,Scooter2_Mercator_x-x.0 Altavista robot,Scooter_bh0-3.0.3 Altavista robot,Scooter_trk3-3.0.3 Scope Navigator mobile browser (Japan),Scope (Mars+) Some nec.com robot using Research Republic ScoutAbout Research Tool,ScoutAbout Ant.com search robot,ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ ScoutMaster information retrieval software,scoutmaster Scrub the web robot (66.93.156.xx),Scrubby/2.x (http://www.scrubtheweb.com/) Scrub the web robot (66.93.156.xx),Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html) URL Search+ search software,Search+ Xtreem Search Engine Studio - SE software,Search-Engine-Studio Search.ch robot,search.ch V1.4 Search.ch robot,search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch) Enterprise Search web indexing / site searching tool,Search/1.0 (http://www.innerprise.net/es-spider.asp) Unknown robot / website grabber from Chinatelecom (219.142.78.xxx),searchbot admin@google.com SearchByUSA robot (69.150.7.xxx),SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUsa.com) Searchday (Germany) search robot,SearchdayBot Searchexpress spider,SearchExpress Spider0.99 Searchguild forum & directory robot (81.3.75.xxx),SearchGuild/DMOZ/Experiment (searchguild@gmail.com) Searchguild forum & directory robot (81.3.75.xxx),SearchGuild_DMOZ_Experiment (chris@searchguild.com) Searchit robot (69.93.107.xx),Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) Searchmee! Search Engine (prototype) robot by findanisp.com,Searchmee! Spider v0.98a SearchSight search robot,SearchSight/2.0 (http://SearchSight.com/) SearchSpider robot,SearchSpider.com/1.1 SearchSpider robot,Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com) Janas (Ideare.com / Tiscali.it) robot,SearchTone2.0 - IDEARE seekport. beta search (Germany) robot,Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3 seekport. beta search (Germany) robot,Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF) seekport. beta search (Germany) robot,Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2 Lookseek search robot / link checking,Seeker.lookseek.com semaforo.net web filtering software,semaforo.net Semager.de (was NG-Search) semantic search - Germany,Semager/1.1 (http://www.semager.de/blog/semager-bots/) Semager.de (was NG-Search) semantic search - Germany,Semager/1.x (http://www.semager.de) Semantic Discovery domain checking tool,semanticdiscovery/0.x Sensis Australia search robot,Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au) Sensis Australia search robot,Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au) Seznam Search (Czech Republic) robot,SeznamBot/1.0 Seznam Search (Czech Republic) robot,SeznamBot/1.0 (+http://fulltext.seznam.cz/) Seznam Search (Czech Republic) robot,SeznamBot/2.0-test (+http://fulltext.sblog.cz/) Unknown robot from Shablast.com - Website has no content - Ignores robots.txt,ShablastBot 1.0 Shareaza P2P peer-to-peer download client,Shareaza v1.x.x.xx SharewarePlaza File Check Bot - link checking,SharewarePlazaFileCheckBot/1.0+(+http://www.SharewarePlaza.com) Chikayama-Taura Lab Shim-Crawler used for The Kototoi Project (Japan) - (133.11.36.xx),Shim Crawler Chikayama-Taura Lab Shim-Crawler used for The Kototoi Project (Japan) - (133.11.36.xx),Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp) ShopWiki shopping search based on LittleWiki search,ShopWiki/1.0 ( +http://www.shopwiki.com/) ShopWiki shopping search based on LittleWiki search,ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) Shoula Search Engine crawler,Shoula.com Crawler 2.0 Siets Crawler - Web based site crawling application,SietsCrawler/1.1 (+http://www.siets.biz) Sigram's Nutch robot - crawler testing,Sigram/Nutch-1.0-dev (Test agent for Nutch development; http://www.sigram.com/bot.html; bot at sigram dot com) Siigle search (Turkey) robot,Siigle Orumcex v.001 Turkey (http://www.siigle.com) Slider Search directory robot (194.213.194.2xx),silk/1.0 Slider Search directory robot (194.213.194.2xx),silk/1.0 (+http://www.slider.com/silk.htm)/3.7 SimpleFavPanel - IE newsfeed panel plugin,SimpleFavPanel/1.2 Simpy bookmarking and personal search engine,Simpy 1.x; http://www.simpy.com/ Simpy bookmarking and personal search engine,Simpy/1.x (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com) Sirketçe search - Turkey,Sirketcebot/v.01 (http://www.sirketce.com/bot.html) SiteBar online bookmark manager,SiteBar/3.x.x (Bookmark Server; http://sitebar.org/) SiteBar bookmark server,SiteBar/x.x SiteBar bookmark server,SiteBar/x.x.x (Bookmark Server; http://sitebar.org/) Internetseer Web Site Monitoring / Claymont robot,sitecheck.internetseer.com Internetseer Web Site Monitoring,sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com) SiteRecon website monitoring spider at xx minute intervals,SiteRecon+(xx) PC Magazin web site downloadmanager,SiteSnagger Site Spider robot (66.249.17.xx),SiteSpider +(http://www.SiteSpider.com/) SiteSucker Mac website downloading tool,SiteSucker/1.x.x SiteTagger.com bookmark organizer,SiteTaggerBot (http://www.sitetagger.com/bot.htm) SiteTruth - Automatic site legitimacy rating system,SiteTruth.com site rating system Webwasher.com (217.146.159.xx) internet filter,SiteWinder Xtreeme SiteXpert sitemap & search engine builder,SiteXpert Skaffe.com directory link checker,Skampy/0.9.x (http://www.skaffe.com/skampy-info.html) Skaffe.com directory link checker,Skimpy/0.x (http://www.skaffe.com/skampy-info.html) Visvo distributed website crawler based on Nutch,Skywalker/0.1 (Skywalker; anonymous; anonymous) Only.com robot,Slarp/0.1 Sleipnir - Japanese Explorer based browser & search bar,Sleipnir Sleipnir - Japanese Explorer based browser & search bar,Sleipnir Version 1.xx Sleipnir - Japanese Explorer based browser & search bar,Sleipnir Version2.x Sleipnir - Japanese Explorer based browser & search bar,Sleipnir/2.xx Slider Search directory robot (194.213.194.2xx),Slider_Search_v1-de Slim Browser (IE based browser) - uses this user agent for favicon.ico only,SlimBrowser Inktomi (Hotbot-Lycos-NBCi) robot,Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html) Inktomi (Hotbot-Lycos-NBCi) robot,Slurp/2.0-KiteWeekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html) Inktomi (Hotbot-Lycos-NBCi) robot,Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html) Inktomi (Hotbot-Lycos-NBCi) robot - 72.30.61.xx(x),Slurpy Verifier/1.0 Slysearch robot (now Turnitin robot),SlySearch (slysearch@slysearch.com) Slysearch robot (now Turnitin robot),SlySearch/1.0 http://www.plagiarism.org/crawler/robotinfo.html Slysearch robot (now Turnitin robot),SlySearch/1.x http://www.slysearch.com Netzip/Smartdownload download manager,SmartDownload/1.2.67 (Win32; Jan 12 1999) Netzip/Smartdownload download manager,SmartDownload/1.2.77 (Win32; Feb 1 2000) Netzip/Smartdownload download manager,SmartDownload/1.2.77 (Win32; Jun 19 2001) Loop Improvements NRS Enterprise search (69.44.155.xx[x]),smartwit.com SmiffyDCMetaSpider - Robot to check the retro-adding of Dublin Core metadata,SmiffyDCMetaSpider/1.0 Snoopy PHP-client,sna-0.0.1 (mikemuzio@msn.com) Snoopy PHP-client,sna-0.0.1 mikeelliott@hotmail.com Unknown bot from bb2.net (66.234.139.xxx) also as Snapbot/1.0,snap.com beta crawler v0 Unknown bot from bb2.net (66.234.139.xxx) - also as snap.com,Snapbot/1.0 Unknown bot from Psinet / Cogentco - not from Snap.com,"Snapbot/1.0 (Snap Shots, +http://www.snap.com)" My UrlTrends online web ranking service,Snappy/1.1 ( http://www.urltrends.com/ ) Snarfer RSS reader,Snarfer/0.x.x (http://www.snarfware.com/) Unknown robot from 217.229.156.xx (T-Online Germany),SnoopRob/x.x Snoopy PHP-client,Snoopy v1.xx Snoopy PHP-client,Snoopy v1.xx- : User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2) Snoopy PHP-client,Snoopy_v0.xx Snyke.com France robot,SnykeBot/0.6 (http://www.snyke.com) Link crawler for the social sciences,SocSciBot () NetFront browser on Softbank mobile phone,SoftBank/1.0/812SH/SHJ001 Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 Soft Hypermarket link checking,SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) Softizer.com software directory link checking,Softizerbot (http://www.softizer.com) Unknown UA from Chinanet (220.181.26.1xx) faking Sogou search robot,sogou develop spider Unknown UA from Chinanet (220.181.18.xx) faking Sogou search robot,Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Unknown UA from Chinanet (220.181.26.1xx) faking Sogou search robot,sogou spider Unknown UA from Chinanet (220.181.26.1xx) faking Sogou search robot,Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Unknown UA from Chinanet (220.181.26.1xx) faking Sogou search robot,sohu agent Sohu (Search Fox) search robot China (61.135.131.xxx),sohu-search SOSO search (China) spider,Sosospider+(+http://help.soso.com/webspider.htm) Default Proxomitron (discontinued) filtering proxy user agent identifier,Space Bison/0.02 [fu] (Win67; X; SK) Speed Download (Mac) download manager,SpeedDownload/1.x Speedfind.de robot,speedfind ramBot xtreme 8.1 Entireweb search robot,Speedy Spider (Beta/x.x; speedy@entireweb.com) Entireweb search spider,Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/) Entireweb search robot,Speedy_Spider (http://www.entireweb.com) Sphere blog and news search robot,Sphere Scout&v4.0 - scout at sphere dot com Sphider - a lightweight search engine in PHP,Sphider Only.com robot,Spida/0.1 Search-Info ODP/DMOZ spider,Spider-Sleek/2.0 (+http://search-info.com/linktous.html) Batsch robot,spider.batsch.com TerraNautic spider for Schnellsuchen touristic search (Germany),Spider.TerraNautic.net - v:1.04 Yellopet spider,spider.yellopet.com - www.yellopet.com Maxbot .gov .mil .edu indexing robot,Spider/maxbot.com admin@maxbot.com Unknown robot from CPE at Kasetsart University (158.108.35.xxx),SpiderKU/0.x Yahoo Search user agent or spider (202.165.102.xxx),SpiderMan SpiderMonkey Canada robot,SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml) Spider.de robot,Spinne/2.0 Medkatalog (medical catalogue) Austria robot,Spinne/2.0 med Medkatalog (medical catalogue) Austria robot,Spinne/2.0 med_AH Spock - people search application - via Amazon web services,Spock Crawler (http://www.spock.com/crawler) Sportsuchmaschine (German sports related search) link checking / robot,sportsuchmaschine.de-Robot (Version: 1.02- powered by www.sportsuchmaschine.de) Sproose personalized search (38.100.225.xx),sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@sproose.com) SQ Webscanner Mac download manager,SQ Webscanner Simple page-prefetch for Squid web proxy,Squid-Prefetch Squidclam is a replacement for SquidClamAV-Redirector,squidclam SCAVR - Squid helper script for scanning download URLs for viruses,SquidClamAV_Redirector 1.x.x AOL Search / Pacific Internet Exchange robot,Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) diff. IPs / services i.e.: - Inria.fr robot - Websense (Internet filtering) robot,Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu) Time Warner Telecom user robot ?,Sqworm/2.9.89-BETA (beta_release; 20020130-839; i686-pc-linux-gnu) Some site scanning tool via diff. IPs i.e.: - choiceone.net (216.153.xxx.xxx) - epix.net (216.108.198.xx),SSurf15a 11 Rambler search (Russia) robot (81.19.6x.xx),StackRambler/x.x Stamina download manager,Stamina/1.4 Star Downloader download manager,Star Downloader Star Downloader download manager,StarDownloader/1.xx Experimental search engine spider from 66.92.186.xxx,stat statcrawler@gmail.com Steeler crawler,Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Steeler - University of Tokyo web crawler,Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Strategic Board blog & news search robot,Strategic Board Bot (+http://www.strategicboard.com) Strategic Board blogs and news aggregator robot,Strategic Board Bot (+http://www.strategicboard.com) Surfsafely submission verifier,Submission Spider at surfsafely.com Suchbaer.de (Germany) search robot,suchbaer.de Suchbaer.de (Germany) search robot,suchbaer.de (CrawlerAgent v0.103) Suchbot Germany robot,suchbot Suchknecht Austria robot,Suchknecht.at-Robot suchpad search Germany robot (213.239.194.xx),suchpadbot/1.0 (+http://www.suchpad.de) Sunrise XP handheld news / website reader and converter,Sunrise XP/2.x Sunrise XP web sites and newsfeeds converter and handheld reader,Sunrise/0.42g (Windows XP) SuperBot website copier,SuperBot/x.x (Win32) SuperBot website copier,SuperBot/x.x.x.xx (Windows XP) Ubbi Superdownloads (Brazil) link checking,Superdownloads Spiderman SurfControl Web Filtering,SURF Wanadoo Rechereche robot,SurferF3 1/0 Maskbit Surfmaster bookmark tool,SurfMaster Whois Source domain name information robot (66.249.26.xx),SurveyBot/2.2 Whois Source Whois Source domain name information robot (66.249.26.xx),SurveyBot/2.3 (Whois Source) Yokogao Search Engine robot (Kanazawa University),suzuran HP Secure Web Browser for OpenVMS,SWB/V1.4 (HP) unknown,swbot/0.9c libwww/5.3.1 Swooglebot Swoogle's semantic web crawler,Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.htm) SWSBot - SmartWareSoft (85.186.255.xx) software search engine created for Playfuls.com,SWSBot-Images/1.2 http://www.smartwaresoft.com/swsbot12.html Sygol Search (Italy) robot,SygolBot http://www.sygol.net Sylera browser (Japan),Sylera/1.2.x Mindspring.com user robot,SyncBot SyncIT link validation,SyncIT/x.x Syndirella desktop information aggregator (beta),Syndirella/0.91pre Synomia (France) robot,SynoBot Syntryx Solution Suite - domain / keyword crawler (216.7.179.xx),Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler Szukacz.pl (Polish search) robot,Szukacz/1.x Szukacz.pl (Polish search) robot,Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl) German T-Online browser & internet suite ,T-Online Browser tags2dir.com directory index,tags2dir.com/0.8 (+http://tags2dir.com/directory/) TAGword DMOZ survey - ODP link checking robot,Tagword (http://tagword.com/dmoz_survey.php) Tagyu - del.icio.us bookmark collection online tag generator,Tagyu Agent/1.0 Daumsoft Talkro IR robot,Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com- Home: http://222.122.15.190/webshot) Texas A&M University - Dept. of Computer Science crawler (server or link checking ?),TAMU_CS_IRL_CRAWLER/1.0 Targetblaster user link validation ?,targetblaster.com/0.9k Target Your News - user submitted links,TargetYourNews.com bot Trinity College Dublin (Ireland) TCDBOT,TCDBOT/Nutch-0.8 (PhD student research;http://www.tcd.ie; mcgettrs at t c d dot IE) HTTP header for transfer encoding used as user agent name ?,TE WinInet Internet client app.,TeamSoft WinInet Component Tecomac Gmbh (Germany) crawler software - now Arexera Information Technologies,TECOMAC-Crawler/0.x Tecomi (Germany) beta / test robot (84.201.65.xxx),Tecomi Bot (http://www.tecomi.com/bot.htm) Teemer crawler for NetSeer search (beta) via Amazon Web Services - see also NetSeer/Nutch,"Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com)" Teleport (website) downloading tool,Teleport Pro/1.2x(.1xxx) Teoma crawler (65.214.36.xx[x]),Teoma MP Teoma crawler (65.214.36.xx[x]),teomaagent crawler-admin@teoma.com Teoma crawler (65.214.36.xx[x]),teomaagent1 [crawler-admin@teoma.com] Teoma crawler (65.214.36.xx[x]),teoma_agent1 Teradex Directory robot,Teradex Mapper; mapper@teradex.com; http://www.teradex.com Teragram multilingual text & data processing software,TeragramCrawler Terraminds blog search (Germany),terraminds-bot/1.0 (support@terraminds.de) Terrawiz Indian Search Engine robot,TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) Noceans Information Portfolio Manager (66.35.69.x),Test spider Balihoo - Search Engine for Advertising Media,TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.aspx; tgautier at balihoo dot com) Expert HTML online source viewer,The Expert HTML Source Viewer (http://www.expert-html.com) The Rarest Words - Linguistic experiment crawler via Amazon Web Services,TheRarestParser/0.2a (http://therarestwords.com/) TheSuBot robot (Germany) for an unknown theme based search engine,TheSuBot/0.1 (www.thesubot.de) ThumbShots website thumbnail service (Germany) robot,"thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)" ThumbShots.de (Germany) robot,thumbshots-de-Bot (Version: 1.02- powered by www.thumbshots.de) Breaking Blogs timbo bot blog robot,timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html TimelyWeb web page monitoring tool,http://www.timelyweb.com/ TinEye crawler for an open image search project,TinEye/1.1 (http://tineye.com/crawler.html) Tivra spider from AT&T Labs Research,tivraSpider/1.0 (crawler@tivra.com) Tjgroup spider,TJG/Spider TJvHttpGrabber (JEDI Visual Component Library),TJvMultiHttpGrabber Component Tkensaku Search (Japan) robot from 210.239.46.xxx (www.tken.com),Tkensaku/x.x(http://www.tkensaku.com/q.html) Unknown robot from 195.68.98.xx (coltfrance.com),toCrawl/UrlDispatcher Topodia search engine and personal information assistant (in development),Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia.com/; support@topodia.com) Topos search (Russia) robot,TOPOS robot/1.1 (http://www.topos.com.ua/) Traazi! search (Germany) robot,traazibot/testengine (+http://www.traazi.de) Trailfire web collection and annotating system,Trailfire-bot/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) Trailfire web collection and annotating system,Trailfire-bot/0.7.1 (Trailfire page content analyzer; http://trailfire.com; info@trailfire.com) Trailfire web collection and annotating system,Trailfire/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) Trampelpfad Webkatalog spider,Trampelpfad-Spider Trampelpfad Webkatalog spider,Trampelpfad-Spider-v0.1 Tricos meta tag validation,tricosMetaCheck 1.2216-08-1999 (http://www.tricos.com/metacheck) some bad user agent,TSurf15a 11 Tulip Chain browser / link checker for Dmoz.org directory,TulipChain/5.x (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://java.sun.com/) Linux/2.4.17 Tulip Chain browser / link checker for Dmoz.org directory,TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8 Tumblr Tumblelogs RSS and news syndication crawler,Tumblr/1.0 RSS syndication (+http://www.tumblr.com/) (support@tumblr.com) Turnitin (ex SlySearch) robot for helping educational institutions prevent plagiarism,TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) TurnPike Emporium Directory (207.67.198.x) link checking,Turnpike Emporium LinkChecker/0.1 TutorGig tutorial search robot,TutorGig/1.5 (+http://www.tutorgig.com/crawler) TutorGig tutorial search robot,Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler) Twiceler experimental web crawler,Twiceler www.cuill.com/robots.html Twiceler experimental web crawler,Twiceler-0.9 http://www.cuill.com/twiceler/robot.html File downloading component from Twisted Python,Twisted PageGetter Twitt(url)y URL tracking service for Twitter via Amazon Web Services,Twitturly / v0.x Twotrees content filter,Twotrees Reactive Filter V2.0 Tycoon - Hewlett-Packards distributed cluster solution robot,Tycoon Agent/Nutch-1.0-dev Tygo Search robot,TygoBot Tygo Search robot,TygoProwler UCMore - IE navigation and search plugin,UCmore UCMore - IE navigation and search plugin,UCMore Crawler App Ucweb mobile browser,UCWEB5.1 user agent - maybe UdmSearch (see UdmSearch) ?,UDM UdmSearch / MySearch (now mnoGoSeach) offline browser/search client,UdmSearch/3.1.x University of Iowa Crawler- possibly MySpiders,UIowaCrawler/1.0 UKWizz search robot,UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/) Infoseek robot,Ultraseek Unknown mail harvester/spambot from 80.58.13.xxx (proxycache.rima-tde.net),Under the Rainbow 2.2 DNS Right - Online DNS tools,unknownght.com Web Server IIS vs Apache Survey. See Results at www.DNSRight.com Unknown robot from University of Toronto (128.100.5.1xx),UofTDB_experiment (leehyun@cs.toronto.edu) Mobile phone browser,UP.Browser/3.01-IG01 UP.Link/3.2.3.4 Updated! search robot,updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com) Updated! search robot,updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om) Handspring (PalmOS powered cellphone) Treo Blazer browser,UPG1 UP/4.0 (compatible; Blazer 1.0) UptimeBot.com online link popularity check,Uptimebot UptimeBot.com online link popularity check,UptimeBot(www.uptimebot.com) URI::Fetch - client for fetching HTTP pages and syndication feeds (RSS Atom),URI::Fetch/0.06 Innerprise URL Spider Pro (now ES.NET) web indexing / site searching tool,URL Spider Pro/x.xx (innerprise.net) URLBase - Internet shortcut manager,URLBase/6.x URLBlaze file sharing link toolkit,URLBlaze ://URLFAN news crawler,urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html URLGetFile downloading tool,URLGetFile Innerprise URL Spider Pro (now ES.NET) web indexing / site searching tool,URL_Spider_Pro/x.x Innerprise URL Spider Pro (now ES.NET) web indexing / site searching tool,URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp) BoardReader search favicon fetcher,User-Agent: BoardReader Favicon Fetcher /1.0 info@boardreader.com BoardReader search image fetcher,User-Agent: BoardReader Image Fetcher /1.0 info@boardreader.com ljpic.com - LiveJournal picture feed search,User-Agent: LjSEEK Picture-Bot /1.0 contact@ljseek.com FileHeap download manager,User-Agent: FileHeap! file downloader (http://www.fileheap.com) Malformed UA header from some guestbook/forum spammer,User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Skizzle search robot,User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com) unknown robot (reads robots.txt) or sitegrabber. From different IPs- ie.: 62.98.8.xx (wind.it),user-agent=Mozilla/3.01Gold University of Sydney NLP Spider for research in Natural Language Processing ,USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) Web Thief Site Grabber,UtilMind HTTPGet WebWasher ad filter,Utopia WebWasher 3.0 uTorrent BitTorrent client,uTorrent/1500 Unknown bad behaving bot via Road Runner - see link,VadixBot WiseGuys WAP pages robot,Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)/1.0 Profile WiseGuys robot Netherland - 82.94.216.2,Vagabondo/1.x MT (webagent@wise-guys.nl) WiseGuys robot Netherland - 82.94.216.2,Vagabondo/2.0 MT WiseGuys robot Netherland - 82.94.216.2,Vagabondo/2.0 MT (webagent at wise-guys dot nl) WiseGuys robot Netherland - 82.94.216.2,Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl) WiseGuys robot Netherland - 82.94.216.2,Vagabondo/3.0 (webagent at wise-guys dot nl) Open Directory link checking from Vakes,Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com) unknown level3.net (63.214.172.xxx) robot,VayalaCreep-v0.0.1 (haploid@haploid.com) unknown level3.net (63.214.172.xxx) robot,Vayala|Creep-v0.0.1 (codepoet@wildties.com) iNet Grabber - Internet content grabber,vb wininet Versus Project robot - Comparing methods for near-uniform URL sampling,versus 0.2 (+http://versus.integis.ch) Unknown robot from EPFL University Switzerland (128.178.155.xxx),versus crawler eda.baykan@epfl.ch Verticrawl - Semantic search engine solution (French),Verticrawlbot VeryGoodSearch.com link submission checking,VeryGoodSearch.com.DaddyLongLegs Verzamelgids NL link checking robot,verzamelgids.nl - Networking4all Bot/x.x Verzamelgids NL link checking robot,Verzamelgids/2.2 (http://www.verzamelgids.nl) Unknown robot from Yahoo Norway,Vespa Crawler Sidewinder G2 anti-virus and anti-spyware protection,virus_detector (virus_harvester@securecomputing.com) Visbot crawler for a search software under development,VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) Some IE toolbar made with Visicom Media Dynamic Toolbar software,Visicom Toolbar Vision research lab's Cortina - content based image retrieval (128.111.60.xx),Vision Research Lab image spider at vision.ece.ucsb.edu VLC - Cross-platform media player and streaming server,VLC media player - version 0.8.5 Janus - (c) 1996-2006 the VideoLAN team VM - Vertical Search Engine (China),VMBot/0.x.x (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com) Vortex Web Indexing Robot for a study on internet link distribution,Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) "Kosmix health, auto and travel search crawler (204.14.48.x / 38.113.234.xxx)",voyager-hc/1.0 "Kosmix health, auto and travel search crawler (204.14.48.x / 38.113.234.xxx)",voyager/1.0 "Kosmix health, auto and travel search crawler (204.14.48.x / 38.113.234.xxx)",voyager/2.0 (http://www.kosmix.com/html/crawler.html) Vivisimo search crawler (206.210.89.xxx),VSE/1.0 (testcrawler@hotmail.com) Vivisimo search crawler (206.210.89.xxx),VSE/1.0 (testcrawler@vivisimo.com) Verity vspider indexing software,vspider Verity vspider indexing software,vspider/3.x VWBot - MetaQuerier Crawler for the MetaQuerier project at the University of Illinois,VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiuc.edu W3C Link Checker,W3C-checklink/3.x.x.x libwww-perl/5.xx W3C Link Checker,W3C-checklink/4.x [4.xx] libwww-perl/5.xxx WebCon - the Libwww command line tool,W3C-WebCon/5.x.x libwww/5.x.x W3C Line Mode (character based Web browser),W3CLineMode/5.4.0 libwww/5.x.x Unknown link checking using Libwww via Korea Telecom (221.148.44.xxx),W3CRobot/5.4.0 libwww/5.4.0 W3C HTML-Code Validator,W3C_Validator/1.xxx libwww-perl/5.xx w3m Linux pager / text-based browser,w3m/0.x.xx W3 Site Search (Germany) search engine solution,W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de Wadain (Japan) Blog / RSS search crawler,wadaino.jp-crawler 0.2 (http://wadaino.jp/) Wanna-Be text mode browser,WannaBe (Macintosh; PPC) WapOnWindows WAP browser for PCs,WapOnWindows 1.0 Watchfire WebXM intranet solution,Watchfire WebXM 1.0 WAVcheck - Simple Vendor Discovery Tool for detecting client-side tags from web analytics vendors,WAVcheck 1.0.x (http://www.webbanalys.se/apps/WAVcheck/) Wavefire local search community engine (64.141.15.1xx),Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com) Waypath blog discovery engine robot,Waypath development crawler - info at waypath dot com Waypath blog discovery engine robot,Waypath Scout v2.x - info at waypath dot com WDG HTML-code validator,WDG_Validator/1.1 Datafire.com's Web Image Collector (graphics downloading tool),Web Image Collector Relsoft link checking software,Web Link Validator 1.5 RankMeter ranking software,Web Snooper Web-bekannt German web directory link checking,"web-bekannt (Version: 1.02, powered by www.internetservice-franken.de)" Web-bekannt German web directory link checking,"web-bekannt (Version: 1.02, powered by www.web-bekannt.de)" Unkown link or server checking from Würzburg University Germany (132.187.10.xx),Web-Bot V1.03 Unknown robot from 69.50.233.x (nectartech.com),Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3 Web2Express / Web2x - Open data searching tool,web2express.org/Nutch-0.9-dev (leveled playing field; http://web2express.org/; info at web2express.org) WebAlta search Russia crawler (85.21.201.xx),WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html) WebarooBot / RufusBot from webaroo offline search service,WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html) WebarooBot / RufusBot from webaroo offline search service,WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html) Yanasoft WebAuto website copier / downloading tool,WebAuto/3.4xxx (WinNT; I) Web Bandit personal search software,webbandit/4.xx.0 Amansoft WebBug web server protocol test,WebBug/5.x WebClipping.com - online news monitoring service,Webclipping.com WebCollage Syndicator graphics crawler/collector,webcollage/1.xx Quarterdecks WebCompass search tool,WebCompass 2.0 WebCopier offline browser,WebCopier vx.x WebCopier offline browser,WebCopier vx.xa WebCorp linguistic search engine (UK),WebCorp/1.0 Webcrawl Search robot (64.40.105.xxx),webcrawl.net Unix/Linux Web Downloader,WebDownloader for X x.xx Unknown robot from china-netcom.com,Webdup/0.9 WingFlyer WebFetch website downloading tool,WebFetch webfetch - command line tool to fetch files via HTTP,webfetch/5.x.x Verso NetSpective WebFilter,WebFilter Robot 1.0 Telemate.net NetSpective WebFilter,WebFilter Robot 1.x Webfind search robot,WebFindBot(http://www.web-find.com) Webglimpse search engine software,Webglimpse 2.xx.x (http://webglimpse.net) webGobbler - Online random image generator,webGobbler/1.x.x fake ?,webhack herbert.groot.jebbink.nl Web Images collage generator,WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages ) WebLight web analyzer & link checker,WebLight/4.x.x (support@illumit.com; http://www.illumit.com/Products/weblight/) WebLink's link management system for HTTP- FTP and Mail hyperlinks,Weblink's checker/ Los Alamos National Laboratoy weblog research project,Weblog Attitude Diffusion 1.0 Unknown robot from Leipzig University (Germany) faculty for computer science,"webmeasurement-bot, http://rvs.informatik.uni-leipzig.de" WebMiner bulk file downloader,WebMiner/x.x [en] (Win98; I) WeBoX (Japan) - Browser and web collector,WeBoX/0.xx WebPix - picture downloading tool,WebPix 1.0 (www.netwu.com) Caesius WebQL - Custom robot/agent generator / web extraction software,WebQL WebRACE - HTTP retrieval- annotation and caching engine,WebRACE/1.1 (University of Cyprus- Distributed Crawler) WebRankSpider experimental web crawler,WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) Webreaper download manager,WebReaper vx.x - www.webreaper.net Webreaper download manager,WebReaper [info@webreaper.net] Webreaper download manager,WebReaper [webreaper@webreaper.net] Websearch Australia robot,WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU) Dortmund University WebSearchBench - Open source search software,WebSearchBench WebCrawler v0.1(Experimental) Dortmund University WebSearchBench - Open source search software,WebSearchBench WebCrawler V1.0 (Beta)- Prof. Dr.-Ing. Christoph Lindemann- Universität Dortmund- cl@cs.uni-dortmund.de- http://websearchbench.cs.uni-dortmund.de/ Web site downloading tool and offline browser (Japan),Website Explorer/0.9.x.x Website eXtractor web site downloading tool,Website eXtractor Sootle web directory Website Worth ranking tool,WebsiteWorth v1.0 Webspinne.de robot,Webspinne/1.0 webmaster@webspinne.de Websquash.com Search Engine robot / link checking,Websquash.com (Add url robot) WebStat - Java statistical computing environment for the web,WebStat/1.0 (Unix; beta; 20040314) Webster - Rev. Healeys web crawler,Webster v0.3 ( http://webster.healeys.net/ ) Websters Webmaster Archive (Germany) submission / pad checking,webster-internet.de pad browser WebStripper download manager,WebStripper/2.xx WebTrafficExpress IBM server software,WebTrafficExpress/x.0 Web Trends link analyzer,WebTrends/3.0 (WinNT) The Stanford WebBase Project crawler,WebVac (webmaster@pita.stanford.edu) webval - Python link checking tool,WebVal/1.0 Webverzeichnis.de (Germany) directory robot,Webverzeichnis.de - Telefon: 01908 / 26005 Web Vulnerability Crawler,WebVulnCrawl.unknown/1.0 libwww-perl/5.803 Studio Net.Idea's Web Watcher Monitor robot,WebWatcherMonitor/2.01 WebZip offline browser,WebZIP/x.x (http://www.spidersoft.com) Unknown spam bot / harvester (62.163.**.** / 62.194.**.*),Wells Search II Some spam bot- see link,WEP Search 00 wwIPStuff - Internet client tools for Visual FoxPro,West Wind Internet Protocols 4.xx IBM's Almaden Research robot (Clever search project),WFARC GNU wget - file downloader,Wget/1.x(.x)GNU wget http://www.gnu.org/software/wget/wget.html - file downloader GNU wget - file downloader,Wget/1.x+cvs-stable (Red Hat modified) GNU wget - file downloader,Wget/1.x.x+cvs Whatsup Gold network monitor,Whatsup/x.x WhatUSeek / Chubba robot,whatUseek_winona/3.0 WhizBang! Labs (closed since May 2002) information extraction robot,WhizBang! Lab some download agent,Wildsoft Surfer Twotrees crawler,Willow Internet Crawler by Twotrees V2.1 unknown robot from gw.ocg-corp.com (209.126.176.x),WinampMPEG/2.00 (larbin@unspecified.mail) Super Affiliate Tracker agent by Wincer Song,WincerSong Agent v1.0 Windows Media Player 10,Windows-Media-Player/10.00.00.xxxx Nicksoft WinGet download manager,WinGet 1.1 Example code for a WinHTTP C++ library crawler,WinHTTP Example/1.0 Wink beta search robot (64.13.136.x),WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com) WinPodder - Podcast player and RSS reader,WinPodder (http://winpodder.com) WinWap - Windows PC WAP browser,WinWAP/3.x (3.x.x.xx; Win32) (Google WAP Proxy/1.0) Yammba web directory (Germany) link checking,"Wir sind die Borg (Version: 1.03, Sie wurden Assimiliert +http://www.yammba.com/suchmaschine/bot.html)" WIRE crawler used by the University of Pisa - Italy,"WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it)" WIRE - Web information retrieval environment crawler,"WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler)" Korea Wisenut robot,WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com) Wisewire domain checker (Discontinued),WiseWire-Spider2 WISH academic research project for link checking,wish-project (http://wish.slis.tsukuba.ac.jp/) Wordchamp web page vocabulary / translation robot,WordChampBot WordPress personal Blog publishing platform,WordPress/x.x.x.x PHP/4.x.xx WORIO (beta) search for computer scientists and programmers using Heritrix open-source crawler,worio heritrix bot (+http://worio.com/) WORIO (beta) search for computer scientists and programmers via Amazon Web Services,woriobot ( http://www.worio.com/) Entireweb Search robot (62.13.25.xxx),WorldLight unknown link checking (from 4.18.57.126) ?,WorQmada/1.0 Wotbox spider,Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com) Wotbox spider,Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02 WebSearchBench crawler from Dortmund University- Germany,WSB WebCrawler V1.0 (Beta)- cl@cs.uni-dortmund.de WebSearchBench crawler from Dortmund University- Germany,"WSB, http://websearchbench.cs.uni-dortmund.de" WUME Lab's web crawler (128.180.121.xxx),wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) Wusage log-file analysis,Wusage/x.0@boutell.com WWLib - Wolverhampton Univerity Web Library for classifying web documents,Wwlib/Linux WWSBOT web server version checker,WWSBOT 1.x [--- http://www.analyzer.nu ---] Perl web page fetching module,WWW-Mechanize/1.1x Arianna robot,www.arianna.it Business-Socket.com link checking ?,www.business-socket.com registry verify/1.x The DoWeb UK Business directory link checking,www.doweb.co.uk crawler www4mail - web navigation & database search by e-mail,www4mail/2.x libwww-FM/2.14 (Unix; I) WWWC Updating check of Web pages. (Japanese only),WWWC/1.0x World Wide Weasel Germany robot,WWWeasel Robot v1.00 (http://wwweasel.de) WWWoffle download manager,WWWOFFLE/2.x Unknown robot from CIS at Munich University,wwwster/1.x (Beta- mailto:gue@cis.uni-muenchen.de) wxDownload Fast (wxDFast) open source download manager,wxDownload Fast Arexera (Germany) crawler software,X-Crawler Xaldon WebSpider offline browser,Xaldon WebSpider Xenu link checker,Xenu Link Sleuth 1.xx Xenu link checker,Xenu's Link Sleuth 1.x[a-z] XerKa text mining and information retrieval software,Xerka WebBot v1.0.0 [UPVOpenDir] xine - free Linux / OS/2 multimedia player,xine/1.0 XIRQ search (beta) robot (70.86.206.1xx),xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com) GPath / XMLSlurp - Expression language for tree structured data,XMLSlurp/0.1 libwww-perl/5.805 Metamark URL Shorten Service,XRL/2.00b1 (Linux; i686; en-us) (+http://metamark.net/about) Xylix Retrieval System software,Xylix Inria Crawler,xyro_(xcrawler@cosmos.inria.fr) Yahoo Search Japan robot (211.14.8.2xx),Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Yahoo Search Japan robot (203.216.197.xxx),Y!J-SRD/1.0 Yahoo Search Japan robot (211.14.8.2xx),Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Yahoo picture service for mobiles,Y!OASIS/TEST no-ad Mozilla/4.08 [en] (X11; I; FreeBSD 2.2.8-STABLE i386) Y!TunnelPro - Yahoo! Messenger companion user agent,Y!TunnelPro Yacy distributed P2P web search engine robot,yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/en) Yacy distributed P2P web search engine robot,yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net (Yahoo) Pipes interactive data aggregator robot,Yahoo Pipes 1.0 Yahoo Mindset: Intent-driven Search (66.228.182.1xx),Yahoo! Mindset Yahoo blog indexing robot (209.191.83.1xx),Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) Yahoo multimedia crawler (206.190.43.xx),Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com) Yahoo multimedia crawler,Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0) Yahoo multimedia crawler via Fastsearch.net (66.77.73.xx),Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com) Yahoo Search robot,Yahoo-Test/4.0 Yahoo crawler via Overture (66.77.73.3x),Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler Yahoo Publisher Network RSS crawler,YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide) Yahoo Product Search crawler ( 68.142.195..x),YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/) Yahoo Product Search crawler ( 66.196.93.x),YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/) Yahoo Product Search crawler ( 66.196.93.x),YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/yahooseeker.html) Yahoo Product Search crawler ( 66.196.93.x),YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/) Yahoo Product Search crawler ( 68.142.195..x),YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) Yahoo robot,YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-webmaster@yahoo-inc.com ) Yandex Search Russia link checking (213.180.206.2xx),Yandex/1.01.001 (compatible; Win16; I) Yanga search robot by Gigabase (Russian Federation),Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/) Yarienavoir search (Belgium) robot,yarienavoir.net/0.2 1noon.com search Korea robot (222.231.21.xxx),Yeti 1noon.com search Korea robot (222.231.21.xxx),"Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it)" Naver search (Korea) robot,Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) yggdrasil spider for GoPubMed biorelated search engine,yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu minus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dresden dot de) Yodao search (China),YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; ) Yoono - community based search (193.110.140.xxx / 194.0.179.[x]xx),yoofind/yoofind-0.1-dev (yoono webcrawler; http://www.yoono.com ; MyEmail) Yoogli search (under development) agent,yoogliFetchAgent/0.1 Yoono - community based search (193.110.140.xxx / 194.0.179.[x]xx),yoono/1.0 web-crawler/1.0 YottaCars bot - YottaCar car search engine ( 64.62.175.xxx),YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine YottaShopping bot - YottaShopping search engine ( 64.62.175.xxx) ,YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine Y!TunnelPro - Yahoo! Messenger companion user agent,YTunnelPro Z-Add online link checker,Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/) Zao crawler for Kototoi Project,Zao-Crawler Zao crawler for Kototoi Project,Zao-Crawler 0.2b Zao crawler for Kototoi Project,Zao/0.1 (http://www.kototoi.org/zao/) Zeus Internet Marketing Robot based on Webster Pro component,ZBot/1.00 (icaulfield@zeus.com) Zearchit German search / directory,Zearchit Ze.bz Moteur de Recherche robot,ZeBot_lseek.net (bot@ze.bz) Ze.bz Moteur de Recherche robot,ZeBot_www.ze.bz (ze.bz@hotmail.com) ZedZo Search (beta) robot,zedzo.digest/0.1 (http://www.zedzo.com/) Zend PHP frameworks Zend_Http_Client component,Zend_Http_Client Powerset Natural Language Search crawler (under development) using Heritrix via Amazon Web Services,"zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+http://www.powerset.com) [email:crawl@powerset.com,email:paul@page-store.com]" Zerx search robot ?,zerxbot/Version 0.6 libwww-perl/5.79 Zeus Internet Marketing Robot (based on Webster Pro),Zeus ThemeSite Viewer Webster Pro V2.9 Win32 Zeus Internet Marketing Robot (based on Webster Pro),Zeus xxxxx Webster Pro V2.9 Win32 Zeusbot robot for building the Ulsysseek.com index,Zeusbot/0.07 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com) Unknown agent (server- or link checking ?) from 198.173.158.xx,Ziggy -- The Clown From Hell!! Zipp.net web search robot,ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net) Zipp.net web search robot,ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net) Zippyfinder robot,Zippy v2.0 - Zippyfinder.com Microsoft Zoo Tycoon 2 game client,Zoo Tycoon 2 Client -- http://www.zootycoon.com Zoom Search Engine software spider,ZoomSpider - wrensoft.com collecting for machine learning; http://zscho.de/ ),Zscho.de Crawler/Nutch-1.0-Zscho.de-semantic_patch (Zscho.de Crawler zspider robot for a new search engine,zspider/0.9-dev http://feedback.redkolibri.com/ Wisenut robot,ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com) FM Scene soccer fan pages plugin,Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (FM Scene 4.6.1) PrevX security plugin,Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729) (Prevx 3.0.5)