[00:03] *** pizzaiolo has joined #archiveteam-bs [00:10] *** pizzaiolo has quit IRC (Remote host closed the connection) [00:17] *** Smiley has quit IRC (Remote host closed the connection) [00:17] *** Smiley has joined #archiveteam-bs [00:32] *** JAA has quit IRC (Quit: Page closed) [00:32] *** JAA has joined #archiveteam-bs [00:44] *** j08nY has quit IRC (Quit: Leaving) [01:22] *** passerby has quit IRC (Read error: Operation timed out) [01:28] *** passerby has joined #archiveteam-bs [01:47] *** pizzaiolo has joined #archiveteam-bs [02:05] *** pnJay has quit IRC (Read error: Connection reset by peer) [02:27] *** pizzaiolo has quit IRC (Ping timeout: 246 seconds) [02:32] *** pnJay has joined #archiveteam-bs [02:37] *** ZexaronS has quit IRC (Leaving) [02:57] *** Roelandus has quit IRC (Ping timeout: 268 seconds) [04:34] *** RichardG has quit IRC (Quit: Keyboard not found, press F1 to continue) [05:00] *** BlueMaxim has joined #archiveteam-bs [05:20] *** RichardG has joined #archiveteam-bs [05:37] *** RichardG has quit IRC (Ping timeout: 244 seconds) [05:54] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [06:00] *** Sk1d has joined #archiveteam-bs [06:25] *** RichardG has joined #archiveteam-bs [06:38] *** RichardG has quit IRC (Ping timeout: 633 seconds) [07:03] *** RichardG has joined #archiveteam-bs [07:14] *** kyounko has joined #archiveteam-bs [07:57] *** masterX24 has joined #archiveteam-bs [07:59] update on DMOZ from my end: at 3,2 mio pages 190k to go atm, that counter trolled me at 1,7 mio since some pages were "hidden" deeply and only appeared in the later stages [08:03] luckily i caught those 420s early at my crawl and whenever one slipped thru a ignore got slapped on so my crawl should only have a few of those buggers [08:07] *** Boppen has quit IRC (Ping timeout: 194 seconds) [08:07] sorry for not hopping on in the meantime [08:07] *** Boppen has joined #archiveteam-bs [08:13] *** j08nY has joined #archiveteam-bs [08:36] *** GE has joined #archiveteam-bs [08:41] *** fie has quit IRC (Leaving) [08:42] *** fie has joined #archiveteam-bs [08:43] *** j08nY has quit IRC (Quit: Leaving) [08:54] masterX24: good to hear. I also grabbed those ~300 pages last night with wget (with the command I posted + --truncate-output) just in case. Let's hope DMOZ stays up long enough so your crawl can finish. [08:56] cranked up speed a bit, originally crawl started with 2 conns parallel, later after more shit appeared i went to 3 and yesterday i switched to 5 connections after knowing my servers cpu works out [08:56] also: IRC username limit stole a 4 from my name (usually i use masterX244 everywhere) [08:57] *** Honno has quit IRC (Ping timeout: 370 seconds) [09:17] Ha, I thought something was missing but wasn't sure what it was :D Yeah, the 9 character limit is stupid [09:25] *** kyounko|2 has joined #archiveteam-bs [09:31] *** kyounko has quit IRC (Read error: Operation timed out) [09:39] @JAA how did you get the feeling something was missing? [09:47] *** qwebirc73 has joined #archiveteam-bs [09:47] damnit, firefox crashed [09:48] I saw your post on /r/Archiveteam a few days ago and faintly remembered what your nick looked like. My head was probably thinking "something with m, maybe a letter, and three digits" or so. [09:49] ahh, makes sense then [09:49] *** masterX24 has quit IRC (Ping timeout: 268 seconds) [09:49] relogging after server noticed my zap [09:49] *** qwebirc73 has quit IRC (Client Quit) [09:50] *** masterX24 has joined #archiveteam-bs [09:50] back now with real nick [09:50] welcome back :) [10:00] *** GE has quit IRC (Remote host closed the connection) [10:02] *** Honno has joined #archiveteam-bs [10:09] *** Honno has quit IRC (Ping timeout: 370 seconds) [10:17] *** Jonison has joined #archiveteam-bs [10:25] *** Jonison has quit IRC (Quit: Leaving) [10:43] 146k to go... hoping that crawl finishes on DMOZ [10:43] *** Cameron_D has quit IRC (Read error: Operation timed out) [10:44] *** yipdw has quit IRC (Read error: Operation timed out) [10:48] *** me has joined #archiveteam-bs [10:52] *** Cameron_D has joined #archiveteam-bs [10:56] *** BlueMaxim has quit IRC (Read error: Operation timed out) [10:56] *** BlueMaxim has joined #archiveteam-bs [11:05] *** me has quit IRC (Read error: Operation timed out) [11:05] *** Cameron_D has quit IRC (Read error: Operation timed out) [11:08] *** Cameron_D has joined #archiveteam-bs [11:12] *** me has joined #archiveteam-bs [11:17] *** BlueMaxim has quit IRC (Read error: Operation timed out) [11:18] *** BlueMaxim has joined #archiveteam-bs [11:28] *** GE has joined #archiveteam-bs [11:35] *** JAA has quit IRC (Quit: Page closed) [11:56] *** BlueMaxim has quit IRC (Read error: Connection reset by peer) [12:14] *** pizzaiolo has joined #archiveteam-bs [12:16] *** JAA has joined #archiveteam-bs [12:22] *** wabu has quit IRC (Read error: Operation timed out) [12:22] *** wabu has joined #archiveteam-bs [12:40] *** GE has quit IRC (Remote host closed the connection) [12:44] *** godane has quit IRC (Ping timeout: 633 seconds) [12:53] https://www.reddit.com/r/DataHoarder/comments/5zbk40/westwoods_ftp_ist_still_up_maybe_you_want_to/ [13:00] *** j08nY has joined #archiveteam-bs [13:23] *** compu has joined #archiveteam-bs [13:23] *** antomati_ has joined #archiveteam-bs [13:23] *** swebb sets mode: +o antomati_ [13:26] *** espes___ has joined #archiveteam-bs [13:26] *** HP_ has joined #archiveteam-bs [13:26] *** w0rp has quit IRC (Read error: Operation timed out) [13:26] *** antomatic has quit IRC (Read error: Operation timed out) [13:26] *** wolfpld has quit IRC (Read error: Operation timed out) [13:26] *** HP has quit IRC (Read error: Operation timed out) [13:26] *** espes__ has quit IRC (Read error: Operation timed out) [13:26] *** luckcolor has quit IRC (Read error: Operation timed out) [13:26] *** TC01 has quit IRC (Read error: Operation timed out) [13:26] *** yuitimoth has quit IRC (Read error: Connection reset by peer) [13:26] *** kevinr has quit IRC (Ping timeout: 381 seconds) [13:26] *** kevinr has joined #archiveteam-bs [13:26] *** compu_85 has quit IRC (Ping timeout: 246 seconds) [13:26] *** w0rp_ has joined #archiveteam-bs [13:26] *** w0rp_ is now known as w0rp [13:27] *** TC01 has joined #archiveteam-bs [13:27] *** wolfpld has joined #archiveteam-bs [13:27] *** luckcolor has joined #archiveteam-bs [13:28] *** yuitimoth has joined #archiveteam-bs [13:39] *** godane has joined #archiveteam-bs [14:29] *** kyounko has joined #archiveteam-bs [14:35] *** kyounko|2 has quit IRC (Read error: Operation timed out) [14:54] *** kyounko|2 has joined #archiveteam-bs [14:56] *** kyounko has quit IRC (Read error: Operation timed out) [15:04] *** masterX24 has quit IRC (Ping timeout: 268 seconds) [15:18] *** masterX24 has joined #archiveteam-bs [15:18] and another firefox crash got me [15:20] *** ndiddy has quit IRC () [15:23] 84k atm, looks like its a close race atm on DMOZ [15:34] Wait a second... The "we're going down" notice has disappeared from https://www.dmoz.org/ ?! [15:36] Ah no, there it is again... weird [15:37] Every few page refreshes, instead of the "Important Notice" it just says "Welcome to DMOZ! It's the Web, Organized." [15:51] 77k atm, disappearing for a bit now, goin home from work, resuming monitoring then [15:51] *** masterX24 has quit IRC (Quit: Page closed) [16:00] JAA: Probably just caching issues on their end [16:19] *** passerby has quit IRC (Read error: Operation timed out) [16:19] *** passerby has joined #archiveteam-bs [16:45] In a strike of irony, this page: http://recordkeepingroundtable.org/2011/06/25/where-do-old-websites-go-to-die-with-jason-scott-of-archive-team-podcast/ [16:45] Linked from: http://archiveteam.org/index.php?title=User:Jscott [16:45] is 404. [16:50] *** GE has joined #archiveteam-bs [16:53] *** odemg has joined #archiveteam-bs [16:53] *** odemg has quit IRC (Remote host closed the connection) [16:55] *** odemg has joined #archiveteam-bs [17:10] IA chrome sends you to https://web.archive.org/web/20121204052508/http://recordkeepingroundtable.org:80/2011/06/25/where-do-old-websites-go-to-die-with-jason-scott-of-archive-team-podcast/ [17:10] :D [17:11] I know personally a lot of the links on JScott's twitter backlog are dead. [17:36] *** Smiley has quit IRC (Read error: Operation timed out) [17:37] *** ndiddy has joined #archiveteam-bs [17:37] *** JAA has quit IRC (Quit: Page closed) [18:23] *** JAA has joined #archiveteam-bs [18:36] *** icedice has joined #archiveteam-bs [18:42] *** masterX24 has joined #archiveteam-bs [18:42] back, also dmoz 30k to go [19:02] rocode: I happen to have a copy of that mp3 in my archives, if the IA does not.. [19:11] *** Aranje has joined #archiveteam-bs [19:14] *** GE has quit IRC (Remote host closed the connection) [19:14] closure: It does. Just listened to it. [19:15] *** Smiley has joined #archiveteam-bs [19:16] I really really hope Self Destructing Cookies gets fixed properly somehow. [19:16] Wrong channel. [19:29] 16k on DMOZ [19:50] less than 10k, seems like that one gets saved... [19:57] *** odemg has quit IRC (Remote host closed the connection) [20:00] *** odemg has joined #archiveteam-bs [20:04] dmoz is a manual project? [20:04] I think I missed the memo. [20:18] single crawler from me running on it [20:18] archivebot was running, too but a config error there slowed it down massively [20:19] (it hunted external links, too and had a few ignores missing to skip the abuse and flag links) [20:20] hm [20:21] any reason they couldn't be added to the archivebot job? [20:21] 2k to go [20:21] idk, i posted it on the channel shortly after noticing it [20:26] 600... [20:31] *** cf has quit IRC (Quit: Bye.) [20:38] saved [20:38] now to get the files to archive.org [20:43] *** GE has joined #archiveteam-bs [21:04] Thanks masterX24! [21:06] damnit... upload doesnt have any progress indicator (having to use coommandline tools from my webserver due to home internet being slow) [21:52] took a while but first warc got in, how to get the files linked to archiveteam collection? SketchCow xmc [21:55] storm is really windy today [21:57] logging out, sleeptime [21:57] *** masterX24 has quit IRC (Quit: Page closed) [22:34] *** amiiboh has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [22:50] *** JAA has quit IRC (Quit: Page closed) [22:50] *** JAA has joined #archiveteam-bs [22:55] looks like bbc took down there xml data [22:56] https://github.com/get-iplayer/get_iplayer/issues [22:56] i breaks downloading streams [22:58] plus side is i was able to grab the audio jason talked about on twitter 12 hours ago [23:20] *** icedice has quit IRC (Ping timeout: 245 seconds) [23:23] *** odemg has quit IRC (Remote host closed the connection) [23:30] *** odemg has joined #archiveteam-bs [23:30] *** odemg has quit IRC (Remote host closed the connection) [23:36] *** mls has joined #archiveteam-bs [23:54] I just finished adding the "Frostbyte Collection" to Archive. [23:54] 4,250 items. [23:54] Now I get to do QA Checks until forever