A needle of Hope
amid a haystack of Chaos.

Google Sitemap Horror

Created: Friday, June 3rd, 2005
14:57:28 UTC
Last modified: Friday, June 3rd, 2005
15:05:28 UTC

Google released a new tool for webmasters today. Google Sitemaps. I was apparently the very first person to download their generator sitemap.py from SourceForge :o. They euphemistically tell people to Run the program on your webserver. Yet this is a perfect way to rack up fees for massively over-utilizing server resources :o Running it brought down my 3200MHz Pentium 4 running Debian Linux and 2 GB of RAM.

Before:


total used free shared buffers cached
Mem: 2001912 417396 1583560 0 4688 103248
-/+ buffers/cache: 309460 691496
Swap: 228244 0 228244

5 minutes later:


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11376 HopeSeek 17 0 1806m 1806m 2028 R 88.3 84.5 1:51.71 python

At which point my phone starts ringing and I hear a co-worker complain that the database isn't responding. When I try to open up a terminal my GUI is completely frozen, the harddrive is going nuts, and then the computer spontaneously rebooted for the first time ever. Upon analyzing the syslog, it appears that when it ran out of swapspace and free memory python (or the script?) it literally panic'd and overran the limits of kswapd.

Now, just imagine this being run on a production server, as per the Google guidelines. Try C++ next time, guys.