[02:20] *** kyan has quit IRC (Read error: Operation timed out) [02:31] *** kyan has joined #internetarchive [03:40] *** kyounko has quit IRC (KVIrc 4.2.0 Equilibrium http://www.kvirc.net/) [07:40] *** kyan has quit IRC (Read error: Operation timed out) [08:19] *** atomotic has joined #internetarchive [10:33] *** matieu has quit IRC (Ping timeout: 366 seconds) [10:36] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [11:23] *** atomotic has joined #internetarchive [12:39] *** section8 has joined #internetarchive [12:40] *** section8 has left [14:20] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [15:42] *** kyan has joined #internetarchive [16:27] *** kyan has quit IRC (Read error: Operation timed out) [17:11] *** kyounko has joined #internetarchive [18:25] *** phuzion has quit IRC (Read error: Operation timed out) [18:26] *** edsu has quit IRC (Read error: Operation timed out) [18:26] *** edsu has joined #internetarchive [18:29] *** phuzion has joined #internetarchive [19:11] *** kyan has joined #internetarchive [19:24] *** kyan has quit IRC (Read error: Operation timed out) [19:30] *** kyan has joined #internetarchive [20:01] Frogging: some comment on stackexchange from 2010 says Heritrix 3 supports a Crawl-Delay up to 300 seconds [20:01] also, they can probably handle the load, because there are crawlers out there with no regard for robots.txt [20:02] thanks [23:05] *** julieback has joined #internetarchive [23:58] *** Stiletto has quit IRC (Read error: Operation timed out)