[22:12] <swebb> Update: I'm about 2/3 done re-crawling/unwinding all of the Oct 2011 -> Current twitter spritzer stream short urls.
[22:12] <swebb> Had to buy some new hardware to get the job done.  :)
[22:16] <swebb> Things to note: First time I had this problem but DNS query caching became an issue, so I had to set up a few more DNS servers.  I'm using 3 crawlers w/ 200 threads on each machine and each machine has its own pdnsd static DNS cache running, with one upstream DNS server.
[22:24] <chronomex> o_O
[22:24] <chronomex> cool