Gmail indexing URLs?

I’ve never really been a conspiracy theory kinda guy…but even this one made pause for a moment…

A friend of mine from Australia wondered if I could send him a few shows from my Tivo, so after a bit of network debauchery, I was able to expose my transferred recordings via my web server. I put a link to the filesystem in one of the web accessible directories (but not somewhere you could browse to) and sent the link off in an email (via gmail).

A small number of minutes later, my connection ground to a halt. I’d told him not to xfer any recording until after 2am my time. I did a quick sniff to see who was using all my bandwidth, and it was a Google crawler!!

Security through obscurity is dead….if it wasn’t already. In the end, I passwd protected the directory and bounced the web server…the crawler buggered off after that.

It brings up an interesting debate…should they crawl any links they encounter, even in ppl’s private emails? One way to ‘secure’ a directory is to only give the URL’s to ppl you trust. In theory, the URL is then as secure as those ppl decide to make it (they could tell more ppl, or add publicly accessible links to the content, etc), but if they do nothing, it should remain ‘secure’. What this means is that to secure a directory, you have to employ passwd protection, IP checks, user agent blocking, or referrer checks….all of which are a pain in the ass for a non-tech person.

Just beware of what links you send out. It’s all being crawled, man.