The DNC’s weblog, Kicking Ass, notes:
Via Atrios, we see that the White House has edited its website to keep search engines from archiving pages on Iraq.
First, a bit of technical background. Most major websites include a text file named robots.txt that tells search engines which directories not to include in search results. (Here’s an example: the Democrats.org robots.txt file lists folders with content — like images — that search engines can’t index.) By adding a directory to robots.txt, you ensure that nothing in that folder will ever show up in a Google search and — more important for this discussion — never be archived by sites like Google.
Sometime between April 2003 and October 2003, someone at the White House added virtually all of the directories with “Iraq” in them to its robots.txt file, meaning that search engines would no longer list those pages in results or archive them.
And Dan Gillmor comments:
Perhaps the White House doesn’t want to make it easy for people to compare its older statements about Iraq with current realities — though that doesn’t explain why the pages are searchable on the White House site itself. Maybe, then, the White House wants to know who’s looking for these things (e.g. by tracking IP addresses of people who query the government site).
Either way, the blocking of search engines is a bad idea, and fundamentally an abuse of the public trust.
What should be done about this? I’d suggest a manual-labor cooperative, of people willing to download the daily feed from the White House, mirror it and ensure that people can search without having their IP addresses logged. If you have a better idea, post a comment.
But Kicking Ass itself has a plausible explanation:
It’s easy enough to understand the reasoning if you look at past White House actions. Earlier this year, the White House revised pages on its website claiming that “combat” was over in Iraq, changing them to say “major combat.”
One of the reasons some alert readers noticed the change — and were able to prove it — was that Google had archived the pages before the change occurred. Now that all of the White House pages about Iraq are no longer archived by Google, such historical revisionism will be harder to catch.