Robot.txt voor me forum.

[special18]

Hey, ik zou graag een robot.txt voor me forum willen, met alle zoekmachines erin en alle directories toegelaten. Kan iemand er zo1 voor mij snel in mekaar zetten.
Heb al gegoogled maarum das tikje onduidelijk allemaal plz help me:)

[janus007]

User-agent: *
Disallow: /cgi-bin/

Groeten Jan

[Fridge]

Code: Selecteer alles

User-agent: Abacho
User-agent: acontbot
User-agent: ah-ha
User-agent: AIBOT
User-agent: aipbot
User-agent: Alexibot
User-agent: Amfibibot
User-agent: AnswerBus
User-agent: appie
User-agent: Aqua_Products
User-agent: Arachmo
User-agent: Arameda
User-agent: Argus
User-agent: ASPSeek
User-agent: asterias
User-agent: b2w/0.1
User-agent: BackDoorBot/1.0
User-agent: baiduspider
User-agent: BecomeBot
User-agent: BigCliqueBOT
User-agent: Blaiz-Bee
User-agent: BlowFish/1.0
User-agent: boitho
User-agent: Boitho-robot
User-agent: Bookmark search tool
User-agent: BotALot
User-agent: BotRightHere
User-agent: BruinBot
User-agent: btbot
User-agent: Bullseye/1.0
User-agent: BuiltBotTough
User-agent: bumblebee
User-agent: BunnySlippers
User-agent: ccubee
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CipinetBot
User-agent: citenikbot
User-agent: ConveraCrawler
User-agent: ConveraMultiMediaCrawler
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: CostaCider
User-agent: CrawlConvera
User-agent: CrawlWave
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: CXL-FatAssANT
User-agent: DataCha0s
User-agent: DataFountains
User-agent: Deepindex
User-agent: DiamondBot
User-agent: DittoSpyder
User-agent: DNSGroup
User-agent: DTAAgent
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: Eule-Robot
User-agent: EuripBot
User-agent: Euripbot
User-agent: eventax
User-agent: Exabot
User-agent: ExtractorPro
User-agent: FairAd Client
User-agent: fantomas
User-agent: Faxobot
User-agent: FDSE
User-agent: FirstGov.gov Search
User-agent: Flaming AttackBot
User-agent: fluffy
User-agent: Foobot
User-agent: FyberSpider
User-agent: Gaisbot
User-agent: Galaxy
User-agent: GalaxyBot
User-agent: gazz
User-agent: genevabot
User-agent: geniebot
User-agent: GeoBot
User-agent: GetRight/4.2
User-agent: Girafabot
User-agent: GOFORITBOT
User-agent: Googlebot-Image
User-agent: GroschoBot
User-agent: gsa-crawler
User-agent: Harvest/1.5
User-agent: holmes
User-agent: HooWWWer
User-agent: Hotzonu
User-agent: htdig
User-agent: http://www.btbot.com/btbot.html
User-agent: httplib
User-agent: humanlinks
User-agent: hloader
User-agent: ia_archiver
User-agent: ia_archiver/1.6
User-agent: ichiro
User-agent: IconSurf
User-agent: Iltrovatore-Setaccio
User-agent: InfociousBot
User-agent: InfoNaviRobot
User-agent: INGRID
User-agent: InnerpriseBot
User-agent: intraVnews
User-agent: Iron33/1.0.2
User-agent: Jayde Crawler
User-agent: JennyBot
User-agent: KakleBot
User-agent: Kenjin Spider
User-agent: Keyword Density/0.9
User-agent: ksibot
User-agent: Kyluka
User-agent: LapozzBot
User-agent: larbin
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkWalker
User-agent: LinkScan/8.1a Unix
User-agent: LNSpiderguy
User-agent: LocalcomBot
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Lycos
User-agent: Mackster
User-agent: Mata Hari
User-agent: Matrix
User-agent: Mediapartners-Google*
User-agent: Metaspinner
User-agent: Microsoft URL Control
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mirago
User-agent: Mister PiX
User-agent: MJ12bot
User-agent: Mnogosearch
User-agent: moget
User-agent: moget/2.1
User-agent: MojeekBot
User-agent: MonkeyCrawl
User-agent: Mozdex
User-agent: mozilla/4
User-agent: mozilla/5
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
User-agent: MSNPTC
User-agent: MSRBOT
User-agent: NaverBot
User-agent: NaverRobot
User-agent: NavissoBot
User-agent: NetAnts
User-agent: NetMechanic
User-agent: NetMind-Minder
User-agent: Networking4all
User-agent: NextGenSearchBot
User-agent: ng
User-agent: nicebot
User-agent: NICErsPRO
User-agent: NimbleCrawler
User-agent: NLCrawler
User-agent: NuSearch Spider
User-agent: Nutch
User-agent: NutchOSU-VLIB
User-agent: Ocelli
User-agent: Octopus
User-agent: Offline Explorer
User-agent: Omnipelagos
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gatherer
User-agent: Orbiter
User-agent: outfoxbot
User-agent: Pajaczek
User-agent: Patwebbot
User-agent: PEERbot
User-agent: PhpDig
User-agent: pipeLiner
User-agent: Poirot
User-agent: polybot
User-agent: Pompos
User-agent: Popdexter
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: Python-urllib
User-agent: QueryN Metasearch
User-agent: QweeryBot
User-agent: Radiation Retriever 1.1
User-agent: RAMPyBot
User-agent: Reaper
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: RMA
User-agent: SiteSnagger
User-agent: SiteSpider
User-agent: Slurp
User-agent: sna-0.0.1
User-agent: snap.com
User-agent: SpankBot
User-agent: spanner
User-agent: SpeedySpider
User-agent: Spinne
User-agent: Squid-Prefetch
User-agent: SquigglebotBot
User-agent: StackRambler
User-agent: sureseeker
User-agent: suzuran
User-agent: SygolBot
User-agent: SynoBot
User-agent: Szukacz
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: TeleportPro
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Tkensaku
User-agent: toCrawl/UrlDispatcher
User-agent: TridentSpider
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: TutorGigBot
User-agent: Ultraseek
User-agent: unchaos_crawler
User-agent: updated
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URL Spider Pro
User-agent: URL Spider SQL
User-agent: URLy Warning
User-agent: Vagabondo
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: W3C-checklink
User-agent: W3CRobot
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcrawl.net
User-agent: WebEnhancer
User-agent: WebFindBot
User-agent: WebIndexer
User-agent: WebmasterWorldForumBot
User-agent: Web Image Collector
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebSauger
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: Wget
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: WhizBang
User-agent: WISEbot
User-agent: Wotbox
User-agent: WWW-Collector-E
User-agent: WWWeasel
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: xirq
User-agent: XunBot
User-agent: YadowsCrawler
User-agent: Yahoo Mindset
User-agent: Yahoo-Blogs
User-agent: Zao
User-agent: Zatka
User-agent: Zealbot
User-agent: Zeus
User-agent: Zeus Link Scout
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: ZipppBot
User-agent: ZyBorg
Disallow: /


User-agent: *
Disallow: /admin/
Disallow: /cache/
Disallow: /db/
Disallow: /media/
Disallow: /modcp/
Disallow: /sqlbeheer/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /templates/
Disallow: /common.php
Disallow: /config.php
Disallow: /faq.php
Disallow: /groupcp.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /posting.php
Disallow: /privmsg.php
Disallow: /profile.php
Disallow: /search.php
Disallow: /viewonline.php

[special18]

kijk das service thanxs.

[thebosdj]

User-agent: *
Disallow: /admin/
Disallow: /cache/
Disallow: /db/
Disallow: /media/
Disallow: /modcp/
Disallow: /sqlbeheer/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /templates/
Disallow: /common.php
Disallow: /config.php
Disallow: /faq.php
Disallow: /groupcp.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /posting.php
Disallow: /privmsg.php
Disallow: /profile.php
Disallow: /search.php
Disallow: /viewonline.php

Dit is ook al genoeg... die * betekent dat alle robots het volgende commando moeten volgen...

[Derky]

Hey, ik zou graag een robot.txt voor me forum willen, met alle zoekmachines erin en alle directories toegelaten. Kan iemand er zo1 voor mij snel in mekaar zetten.
Heb al gegoogled maarum das tikje onduidelijk allemaal plz help me:)

Toegelaten staat er, met al die oplossingen die jullie geven wordt alles juist geblokkeerd.
Als je alles wil toelaten heb je geen robots.txt nodig.

[special18]

Ja alleen heb ik niet zon moeite dat hij mijn plaatjes indexeert, heb 2.1 tb bandwith so.. mensen zoeken plaatjes op google komen toch weer op mijn forum tercht

[svenn]

hoi,

Je hebt helemaal geen robot.txt nodig Als iedereen alles mag zien dan heb je niks nodig en doet google alles wat hij wil net als alle andere robots. dus verwijderen robot.txt

grt svenn

[Eefje]

Robot? Zou ik niet op een forum willen

[janus007]

Volgens mij is het internet op (zoek)robots gebaseerd. vinden en gevonden worden.



Jan

[svenn]

Lief aanbrengen is het niet maar aangezien bart simpson er bij staat

met robots bedoelde ik ook crawlers niet de spam bots

[janus007]

Zo bedoelde ik het niet hoor.
Ik wou alleen zeggen dat als iedereen zijn forum afsluit dmv robots tekst, men ook niet meer kan zoeken in de vaak erg intressante info.
En ik vond het plaatje wel toepasselijk

Dus of geen robots tekst of:
User-agent: *
Disallow: /cgi-bin/

Zodat je forum toegankelijk is.


Jan

[svenn]

Ja klopt hoewel dmoz.org volledig menselijk is opgebouwd