as revealed by the Tums ad
Now, Yahoo knows how offensive this is. So why do it? It must be that they are desparate for $$$. I see no other alternative.
I have been similarly offended by Altavista and Directhit search engines for sucking up my pages at the rate of thousands per day ... every day.
Why do they do this?
Here's the grim logic.
A search engine robot downloads your pages so as to catalog them. This is good. Without this service, no one could find my pages. Therefore, each search engine should download each of my 400+ content pages and 2500+ D-Board message pages just once. After all, they are static pages and don't change much if ever at all.
A search 'bot should be programmed to send a request to each server for the page ... then ... thereafter ... just check to see if it's been changed ... NOT download the whole f'ing thing again and again and again. But that's just what they do!
Are the programmers stupid. I think not. They could program the 'bot to do only change requests and exchange a few bits instead of kilobytes.
As a general rule though, the cache system is never used on the Internet because it does not supply a "true" picture of user activity, i.e. you don't know what pages people are requesting if they get it from a cache in some unknown place on some unknown server. So, programmers have gotten into the habit of forcing a complete download at all times. This gives advertisers a better picture of the web's traffic. It's cash vs cache and the cash always wins. The Internet is a slower place because of this overiding fact.
Here's my letter to AltaVista:
2/5/2002 sent to av.com I wish to protest your spamming my site with your bot. Every week your bot screws up my stats by downloading every fucking page on my site. I'm getting pretty pissed off about it. Yesterday, for instance, your fucked bot sucked up ~840 pages. Why must it fetch every static page on my site every fucking week? What's the point? Is there any way to control your "thing". Get your dick out of my site. If you can't get your bot to check whether a page has been updated before downloading it, I'm going to put up a nice review of your "now defunct" search engine which can't do a tenth what Google does with 1/4 the bother to me.
There is a solution
What is needed is an anonymous debit card (like an AT&T phone card). You buy one at Sam's for $20 or $40 and proceed to use it to access the web by means of ... the ISP that issued the card. You pay say, 10 cents per hour for ISP service and use the card for everything else ... purchases (via secure sockets) and most importantly ... for viewing web pages. Thus, what previously was free now costs you, say, 1/10 of a cent. So on your nightly web surf, you spend maybe 50 centavos total or thereabouts. If you have AOL you're now paying more than that.
What is the effect of this?
You can now make small payments and only put at risk (if someone snatched your card numbers) the 20, 40, 80, 100 bucks you spent for the card ... not thousands of dollars that you risk with a "Visa" credit card.