[00:21] *** DFJustin has joined #archiveteam [00:21] *** swebb sets mode: +o DFJustin [00:26] dashcloud: SketchCow: Here you are! http://www.archiveteam.org/index.php?title=F-Droid [00:26] thanks! [00:26] took the time to comment the code as well. [00:57] *** kristian_ has quit IRC (Leaving) [01:19] *** schbirid2 has joined #archiveteam [01:22] *** BlueMaxim has joined #archiveteam [01:23] *** schbirid has quit IRC (Read error: Operation timed out) [02:14] *** tomwsmf has quit IRC (Read error: Operation timed out) [02:41] *** dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) [02:43] *** dashcloud has joined #archiveteam [03:09] *** G4JC has quit IRC (Are you a good person? Find out! www.NeedGod.com) [03:14] *** ndiddy has joined #archiveteam [03:35] *** ndiddy has quit IRC (Read error: Connection reset by peer) [03:44] *** JesseW has joined #archiveteam [03:58] *** DopefishJ has joined #archiveteam [03:58] *** swebb sets mode: +o DopefishJ [04:01] *** DFJustin has quit IRC (Ping timeout: 260 seconds) [04:11] *** Atom__ has quit IRC (Read error: Connection reset by peer) [04:14] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [04:18] *** DopefishJ is now known as DFJustin [04:22] *** Sk1d has joined #archiveteam [04:27] *** alembic has quit IRC (Read error: Connection reset by peer) [04:33] *** TC02 has quit IRC (Read error: Operation timed out) [04:45] *** DopefishJ has joined #archiveteam [04:45] *** swebb sets mode: +o DopefishJ [04:47] *** DFJustin has quit IRC (Ping timeout: 260 seconds) [04:48] *** TC02 has joined #archiveteam [04:50] *** dashcloud has quit IRC (Read error: Operation timed out) [04:54] *** TC02 has quit IRC (Read error: Operation timed out) [04:54] *** dashcloud has joined #archiveteam [05:01] *** TC02 has joined #archiveteam [05:17] *** DopefishJ is now known as DFJustin [05:28] *** oli has quit IRC (Ping timeout: 260 seconds) [05:30] *** oli has joined #archiveteam [05:52] *** signius has quit IRC (Ping timeout: 260 seconds) [05:54] *** wp494 has quit IRC (Read error: Operation timed out) [05:55] *** wp494 has joined #archiveteam [05:59] *** Rye has quit IRC (Ping timeout: 244 seconds) [06:03] *** Rye has joined #archiveteam [06:04] *** signius has joined #archiveteam [06:28] *** JesseW has quit IRC (Ping timeout: 370 seconds) [06:45] *** Honno has joined #archiveteam [06:51] *** antonizoo has quit IRC (Read error: Connection reset by peer) [06:52] *** antonizoo has joined #archiveteam [07:27] *** midas1 is now known as midas [07:39] *** Laverne has joined #archiveteam [08:09] *** RichardG has joined #archiveteam [08:47] *** ravetcofx has quit IRC (Leaving) [09:16] *** i0npulse has quit IRC (Ping timeout: 244 seconds) [09:31] *** i0npulse has joined #archiveteam [10:04] *** kristian_ has joined #archiveteam [10:07] *** RichardG has quit IRC (Read error: Operation timed out) [10:11] *** RichardG has joined #archiveteam [11:23] *** Stiletto has quit IRC (Read error: Operation timed out) [11:39] *** BlueMaxim has quit IRC (Quit: Leaving) [11:51] *** lesderid has quit IRC (Ping timeout: 260 seconds) [11:52] *** lesderid has joined #archiveteam [11:56] arkiver: can you requeue livejournal discovery? [12:04] ^ I was going to say the same thing [12:05] requeued! [12:06] brb turning my crawler on again [12:07] :) [12:07] *** fie_ has joined #archiveteam [12:09] *** fie has quit IRC (Read error: Operation timed out) [12:19] *** Stiletto has joined #archiveteam [12:20] *** WinterFox has joined #archiveteam [12:23] *** xhdr has quit IRC (Ping timeout: 194 seconds) [12:27] *** xhdr has joined #archiveteam [12:29] *** Froggypwn has quit IRC (Ping timeout: 244 seconds) [12:29] *** Froggypwn has joined #archiveteam [13:21] *** WinterFox has quit IRC (Read error: Operation timed out) [13:36] *** dashcloud has quit IRC (Read error: Operation timed out) [13:40] *** dashcloud has joined #archiveteam [13:56] *** dashcloud has quit IRC (Ping timeout: 250 seconds) [14:00] *** ndiddy has joined #archiveteam [14:00] *** dashcloud has joined #archiveteam [14:32] *** tuankiet has quit IRC (Ping timeout: 244 seconds) [14:38] *** tuankiet has joined #archiveteam [14:40] *** z00nx has quit IRC (Ping timeout: 244 seconds) [14:44] *** z00nx has joined #archiveteam [15:22] *** tomwsmf has joined #archiveteam [15:30] 19 of the 45 newspapers and/or newssites, that Erdogan closed about a month ago, are now archived or lost. I could use some help on the research, time is currently limited for me... http://archiveteam.org/index.php?title=Turkey_Media_Crackdown [15:35] I'll help you out Medowar [15:44] *** JesseW has joined #archiveteam [15:46] *** Aranje has joined #archiveteam [16:02] *** JesseW has quit IRC (Ping timeout: 370 seconds) [16:19] *** Fake-Nam1 has joined #archiveteam [16:19] *** midas1 has joined #archiveteam [16:19] *** swebb sets mode: +o midas1 [16:19] *** _acridAxd has joined #archiveteam [16:19] *** MMovie1 has quit IRC (Read error: Operation timed out) [16:20] *** godane1 has joined #archiveteam [16:20] *** ndizzle has joined #archiveteam [16:20] *** Froggypwn has quit IRC (Read error: Operation timed out) [16:20] *** aMunster has quit IRC (Write error: Broken pipe) [16:20] *** beardicus has quit IRC (Read error: Operation timed out) [16:20] *** midas has quit IRC (Read error: Operation timed out) [16:20] *** SmileyG has quit IRC (Read error: Operation timed out) [16:21] *** acridAxid has quit IRC (Read error: Operation timed out) [16:21] *** _acridAxd is now known as acridAxid [16:22] *** Lord_Nigh has quit IRC (Read error: Operation timed out) [16:24] *** Lord_Nigh has joined #archiveteam [16:25] *** godane has quit IRC (Read error: Operation timed out) [16:25] *** ndiddy has quit IRC (Read error: Operation timed out) [16:26] *** Fake-Name has quit IRC (Read error: Operation timed out) [16:38] *** Smiley has joined #archiveteam [16:53] *** Honno_ has joined #archiveteam [16:53] *** RichardG_ has joined #archiveteam [16:53] *** chazchaz has quit IRC (Read error: Operation timed out) [16:53] *** antomati_ has joined #archiveteam [16:54] *** chazchaz has joined #archiveteam [16:54] *** max has quit IRC (Read error: Operation timed out) [16:55] *** ErkDog_ has joined #archiveteam [16:55] *** is-_ has joined #archiveteam [16:55] *** ErkDog_ has quit IRC (Remote host closed the connection!) [16:56] *** aschmitz has quit IRC (Read error: Operation timed out) [16:56] *** MrRadar has quit IRC (Read error: Operation timed out) [16:56] *** ranma has quit IRC (Read error: Operation timed out) [16:56] *** ErkDog has quit IRC (Read error: Operation timed out) [16:56] *** is- has quit IRC (Read error: Operation timed out) [16:56] *** antomatic has quit IRC (Read error: Operation timed out) [16:56] *** dserodio has quit IRC (Read error: Operation timed out) [16:56] *** mistym- has quit IRC (Ping timeout: 370 seconds) [16:56] *** filippo__ has quit IRC (Ping timeout: 272 seconds) [16:56] *** RichardG has quit IRC (Read error: Connection reset by peer) [16:56] *** nertzy2 has joined #archiveteam [16:56] *** khaoohs has joined #archiveteam [16:56] *** swebb sets mode: +o antomati_ [16:56] *** swebb sets mode: +o chazchaz [16:56] *** macks has joined #archiveteam [16:56] *** macks is now known as max [16:56] *** ErkDog has joined #archiveteam [16:56] *** dcmorton has quit IRC (Excess Flood) [16:57] *** aschmitz has joined #archiveteam [16:57] *** ranma has joined #archiveteam [16:57] *** dserodio has joined #archiveteam [16:58] *** superkuh has quit IRC (Excess Flood) [16:58] *** superkuh has joined #archiveteam [16:58] *** dcmorton has joined #archiveteam [16:58] *** swebb sets mode: +o dcmorton [17:01] *** mistym has joined #archiveteam [17:01] *** swebb sets mode: +o mistym [17:02] *** nertzy has quit IRC (Read error: Operation timed out) [17:02] *** MrRadar has joined #archiveteam [17:05] *** Honno has quit IRC (Read error: Operation timed out) [17:05] *** Aranje has quit IRC (Quit: Three sheets to the wind) [17:07] *** khaoohs_ has quit IRC (Read error: Operation timed out) [17:09] *** Aranje has joined #archiveteam [17:10] *** TC02 has quit IRC (Read error: Operation timed out) [17:10] *** jspiros has quit IRC (Read error: Operation timed out) [17:10] *** ranma has quit IRC (Read error: Operation timed out) [17:10] *** aschmitz has quit IRC (Read error: Operation timed out) [17:10] *** Mayonaise has quit IRC (Read error: Operation timed out) [17:10] *** yakfish has quit IRC (Read error: Operation timed out) [17:10] *** SadDM has quit IRC (Read error: Operation timed out) [17:10] *** yipdw_ has quit IRC (Read error: Operation timed out) [17:11] *** TC02 has joined #archiveteam [17:11] *** acridAxid has quit IRC (Read error: Operation timed out) [17:11] *** aschmitz has joined #archiveteam [17:11] *** Stilett0 has joined #archiveteam [17:11] *** marvinw has quit IRC (Read error: Operation timed out) [17:11] *** kristian_ has quit IRC (Read error: Operation timed out) [17:11] *** REiN^ has quit IRC (Read error: Operation timed out) [17:12] *** Stiletto has quit IRC (Ping timeout: 246 seconds) [17:12] *** Honno has joined #archiveteam [17:12] *** balrog has quit IRC (Read error: Operation timed out) [17:12] *** rduser has quit IRC (Read error: Operation timed out) [17:12] *** HCross has quit IRC (Ping timeout: 246 seconds) [17:12] *** zenguy has quit IRC (Read error: Operation timed out) [17:12] *** remsen has quit IRC (Read error: Operation timed out) [17:12] *** REiN^ has joined #archiveteam [17:13] *** matthusb- has quit IRC (Read error: Operation timed out) [17:14] *** balrog has joined #archiveteam [17:14] *** swebb sets mode: +o balrog [17:15] *** marvinw has joined #archiveteam [17:16] *** Honno_ has quit IRC (Ping timeout: 492 seconds) [17:17] *** filippo__ has joined #archiveteam [17:18] we're going to start the NUjij grab [17:18] joepie91 ^ [17:19] *** remsen has joined #archiveteam [17:19] *** rduser has joined #archiveteam [17:21] *** yipdw has joined #archiveteam [17:21] *** Frogging sets mode: +o yipdw [17:22] *** zenguy has joined #archiveteam [17:25] *** beardicus has joined #archiveteam [17:25] *** swebb sets mode: +o beardicus [17:26] arkiver: noted [17:26] * joepie91 is currently busy reconfiguring his desktop [17:26] (thanks) [17:27] *** acridAxid has joined #archiveteam [17:28] *** ranma has joined #archiveteam [17:28] *** aMunster has joined #archiveteam [17:29] *** nekomune has quit IRC (Ping timeout: 244 seconds) [17:29] *** MMovie has joined #archiveteam [17:30] *** nekomune has joined #archiveteam [17:31] *** HCross has joined #archiveteam [17:31] *** JW_work has quit IRC (Read error: Connection reset by peer) [17:31] *** JW_work has joined #archiveteam [17:32] *** kristian_ has joined #archiveteam [17:32] joepie91: :) [17:32] we'll be doing 50 IDs per item [17:33] *** espes__ has quit IRC (Ping timeout: 244 seconds) [17:56] So thanks to http://fos.textfiles.com/pipeline.html you can see how orkut continues to choke it. [18:04] *** espes__ has joined #archiveteam [18:28] cool SketchCow :-D [18:29] best pipeline name :D [18:29] orkut *sounds* like a pipeline choking [18:30] SketchCow, a total at the bottom would be a nice touch please [18:30] It'd be meaningless. [18:30] Hey HCross a tracker for newsbuddy would be a nice touch please ;) [18:30] Like counting how many cars are in a city limit [18:31] I'll make noise if there's a space problem. [18:31] I could see putting in thresholds. [18:31] Like, "300gb+" [18:31] graphs [18:31] Orkut is almost done though, only 80,000 items left [18:33] It's choking the pipeline, althought we're not in Danger Zone [18:37] Yahoo Answers should probably get a mover/upload soon, because like it's lots of stuff 80 Gigs and only 2300 out of 500,000 items [18:38] 80gb isn't a big deal. 300gb is a big deal [18:38] *** dominic has joined #archiveteam [18:39] Hello, does anyone know how to open a 50 gb warc.gz file ? [18:42] You can extract it with warcat. [18:43] https://pypi.python.org/pypi/Warcat/ [18:51] *** ravetcofx has joined #archiveteam [18:59] *** Selavi has quit IRC (Quit: verb. to stop or discontinue) [19:00] *** kristian_ has quit IRC (Quit: Leaving) [19:07] godane1 you d/led like all of gawker?!? [19:14] all of http://gawker.com is downloaded [19:15] i'm uploading kotaku.com right now cause its very big [19:16] also IA is being slow with the derive [19:16] so i have like 30+ items [19:18] 30 items waiting to be derived [19:19] Ok i will give warcat a try [19:22] *** RichardG_ is now known as RichardG [19:24] *** ravetcofx has quit IRC (Ping timeout: 370 seconds) [19:27] godane1: I'm still working on a heritrix crawl of the gawker properties. It's been going for like 50+ days and still going. [19:27] *** Start has quit IRC (Quit: Disconnected.) [19:30] *** is-_ is now known as is- [19:30] ok [19:32] *** Start has joined #archiveteam [19:33] *** atrocity has joined #archiveteam [19:33] *** Start has quit IRC (Client Quit) [19:33] *** ravetcofx has joined #archiveteam [19:36] *** AlexLehm has joined #archiveteam [19:51] *** AlexLehm has quit IRC (Ping timeout: 260 seconds) [20:05] *** Mayonaise has joined #archiveteam [20:22] https://archive.org/details/forum.openstreetmap.org_20160816/ uploaded [20:23] HCross arkiver: Will be gone for the weekend(and maybe some time after that). If something breaks, well... ¯\_(ツ)_/¯ [20:23] ok, enjoy [20:24] i'm starting to upload reuters videos again [20:24] i have like over 100gb more videos that collection [20:28] *** r3c0d3x has quit IRC (Quit: Leaving) [20:28] *** r3c0d3x has joined #archiveteam [20:35] *** ravetcofx has quit IRC (Ping timeout: 370 seconds) [20:53] *** AlexLehm has joined #archiveteam [20:58] Can someone reset the out on Orkut so they'll finish up? [20:58] http://tracker.archiveteam.org/orkut/ [20:58] *** ravetcofx has joined #archiveteam [21:11] *** Stilett0 has quit IRC () [21:27] *** dominic has quit IRC (Quit: Page closed) [21:28] *** schbirid2 has quit IRC (Quit: Leaving) [21:30] *** G4JC has joined #archiveteam [21:32] SketchCow: FYI, the script now backs up sources as well. And I implemented some integrity checking, which found a few corrupt files that are re-downloaded. So the script should be complete now... [21:33] (FDroid) [21:33] :) [22:09] fixing support for python 3 momentarily.. [22:30] *** AlexLehm has quit IRC (Ping timeout: 260 seconds) [22:34] Nevermind, Python2 works4me. [22:34] Python3 is stupid, all the streams aren't encoded properly for it. [22:40] alright, should be 100% for python2. :) [22:41] Binaries: 10942 items, totalling 10.3 GB [22:41] Sources: 4091 items, totall [22:41] 4091 items, totalling 19.7 GB * [23:12] *** Start has joined #archiveteam