Forum trawled by search engine bots... why?

[pAuL1974]

I was wondering if there was any good reason for the forums here to be trawled by search engine bots.

I expect search engines are a valuable way of letting new members find the site, but could/should the search engines be restricted to trawling the site, and not the forum?

Anyone thinking this is OTT paranoia, try searching your username here, or your MSN details (if posted on the forum here), or other random text you have posted on one of the larger search engines.

[mico]

That's actually how I found another forum similar to this one when I first started finding out about my anxiety. It helped me a lot.

[Winnie57]

I found it disconcerting that when I googled someone's user name here to find a picture. It directed me to posts on this site.

[Amber Fyre]

My user name is unique, and there are 1,310 hits on google for it! Argh!!!!!!!!!!
Anyway, I'm sure no one in my real life would be sad enough to search my username. If they do, I'm sure they wouldn't tell me (because it would be like invading my privacy and then telling me!). So what I don't know won't hurt me. I don't have anything to hide anyway. (Well, apart from that I have SA! )

[pAuL1974]

Originally Posted by The_Godfather

There is nothing you can do to stop Goggle bots. They scan forums for their search engine (hence why specific Goggle searches lead you to this site).

Unless Anne makes this place private (which is unlikely providing the nature of this site (self-help for SAers to find)) - they will continue to scan the forums.

My understanding is the use of a Robots.txt file (wikipedia link and example below) indicates that a web spider should not access certain parts of a site.

Whilst the web spider is under no obligation to obey the Robots.txt request, a reputable spider will oblige, and only spider where the site owner wishes them to. If the spider is allowed on the main site, it will still give SAUK seach engine hits to attract new members.

Some people do put e-mail addresses into search engines, so anyone posting details here in the MSN topic, for example, can be easily found. It's then only a few clicks away to reveal what may be some very personal and revealing posts. I changed my username here immediatlely on joining for this reason, me posting my MSN details on the forum was an oversight.

http://en.wikipedia.org/wiki/Robots.txt

example of a robots.txt file that requests 4 directories to not be spidered:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/

[Concept]

The temporary cache Google uses doesn't last forever. Just delete any sensitive posts here and in a few days/weeks the posts that were catalogued on Google will be gone.

I had a spammer faking my email address I'd left on another site using backscatter to spam other people. The only choice was to change email address, so I know how annoying it can be.

[anne123456]

We could block search engines (mostly) from this board, but I'm not sure the benefits outweigh the costs as many people find this board from specific searches on, for example, blushing or other SA specific symptoms.
But for this reason, we highly recommend that your username is something 'anonymous'. If anyone wants to change their name, just PM me.
Anne

[Peasie]

Originally Posted by anne123456

If anyone wants to change their name, just PM me.
Anne

I want to change my real name to Peasie - can you do that for me?