[00:40] *** j08nY has quit IRC (Quit: Leaving) [00:43] *** Stilett0 has quit IRC () [01:05] HCross2: I'd be willing to run a chunk, I'm still set up from the last one [01:43] *** godane has quit IRC (Ping timeout: 255 seconds) [01:46] *** Stilett0 has joined #archiveteam-bs [01:47] *** Stilett0 is now known as Stiletto [01:50] HCross2: iamine will speed it up a LOT. [01:50] Also, taking bwn's offer is probably a good idea. :-) [01:50] OTOH, taking another month is fine too. :-) [01:58] *** Stiletto has quit IRC (Read error: Connection reset by peer) [02:12] *** pnJay has quit IRC (Leaving) [02:20] *** Stilett0 has joined #archiveteam-bs [02:50] *** godane has joined #archiveteam-bs [03:16] *** pizzaiolo has quit IRC (Remote host closed the connection) [03:24] *** Honno_ has joined #archiveteam-bs [03:26] *** BlueMaxim has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** odemg has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** fie has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** TC01 has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** Honno has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** yipdw has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** luckcolor has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** zino has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** w0rp has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** MrRadar has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** chazchaz has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** eprillios has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** Cameron_D has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** Somebody2 has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** leftyfb has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** superkuh has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** xmc has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** zenguy has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** dcmorton has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** swebb has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** atlogbot has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** slyphic has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** lainu has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** antonizoo has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** Baljem has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** dxrt has quit IRC (hub.efnet.us irc.servercentral.net) [03:26] *** Phil21 has quit IRC (hub.efnet.us irc.servercentral.net) [03:28] *** swebb_ has joined #archiveteam-bs [03:29] *** dxrt_ has joined #archiveteam-bs [03:30] *** w0rp_ has joined #archiveteam-bs [03:31] *** TC04 has joined #archiveteam-bs [03:32] *** zenguy_pc has joined #archiveteam-bs [03:40] *** fie_ has joined #archiveteam-bs [03:40] *** chazchaz1 has joined #archiveteam-bs [03:41] *** superkuh has joined #archiveteam-bs [03:41] *** luckcolor has joined #archiveteam-bs [03:41] *** zino has joined #archiveteam-bs [03:41] *** odemg has joined #archiveteam-bs [03:41] *** w0rp_ is now known as w0rp [03:41] *** zenguy_pc is now known as zenguy [03:42] *** swebb_ is now known as swebb [03:46] *** eprillios has joined #archiveteam-bs [04:03] *** Baljem has joined #archiveteam-bs [04:11] *** ndiddy has quit IRC () [04:12] *** lainu has joined #archiveteam-bs [04:13] *** MrRadar has joined #archiveteam-bs [04:14] *** xmc has joined #archiveteam-bs [04:16] *** slyphic has joined #archiveteam-bs [04:23] *** antonizoo has joined #archiveteam-bs [05:02] *** Somebody2 has joined #archiveteam-bs [05:32] Somebody2: have you got any idea how to make iamine go please [05:34] HCross2: maybe... also ask bwn [05:34] It runs for about 5 seconds then stops [05:36] HCross2: so, what exact command are you running? [05:36] I'm kicking off the IA mine script [05:43] hm, let me look up that script [05:43] https://archive.org/download/ia_census_201604/do_census.sh -- this, right? [05:46] Yep [05:46] *** zenguy has quit IRC (Read error: Operation timed out) [05:47] and passing the list of identifiers as the first argument [05:48] Does ./ia-mine-0.5-py3.4.pex --help [05:48] give you the expected help message? [05:49] and does echo nasa | ./ia-mine-0.5-py3.4.pex [05:49] give you output? [05:49] HCross2: [05:52] *** zenguy has joined #archiveteam-bs [05:57] SketchCow: i'm grabbing a youtube channel called NewsActive3 [05:57] it has tons of full local news programs [06:03] https://www.irccloud.com/pastebin/rpbozOHC/ [06:03] Somebody2: ^ [06:03] it tries... then stops [06:04] looking now [06:06] ive edited the script slightly as I have iamine from pip [06:06] HCross2: I don't understand that paste. [06:06] what happens when you a single item into iamine? [06:06] i.e. the nasa example above [06:08] it prints a whole load of json for the item [06:08] cool, that's it working [06:08] but its not liking the entire census hmm [06:09] it just does one iteration of something I think, then stops [06:09] try a very small itemlist, with like 5 items, but running it through the do_census.sh script [06:09] could it be something with the pip ia-mine? [06:09] do you have anything in jq_errors maybe? [06:09] it certainly could have to do with the version of iamine from pip, yeah [06:13] HCross2: here's a good mini-itemlist: http://termbin.com/69h5 [06:13] It download about 4mb this time [06:14] HCross2: please pastebin the full contents of your modified script somewhere [06:14] I suspect the problem is elswhere in the script, not iamine [06:17] http://termbin.com/fkbf [06:18] Somebody2: ^ its that [06:19] looking now [06:23] HCross2: try ./ia-mine --workers 600 --retries 30 2> error_messages | [06:23] er [06:23] HCross2: try ./ia-mine - --workers 600 --retries 30 2> error_messages | [06:24] when I ran the mini-itemlist as follows: http://termbin.com/clz2 [06:25] the output file (blah) contained: http://termbin.com/smby [06:25] I take it when you run it, it never finishes? [06:27] Somebody2: the pex seems to default to grabbing from stdin, the ia-mine from pip doesn't seem to behave the same way [06:27] *AH* -- that could be a problem, yes! [06:27] in the script [06:30] HCross2: ping [06:31] Yea. I'll switch back to pex [06:31] i think we need a archive of old pre-loaded pc hard drive with junk software installed [06:32] You don't have to -- just switch to passign the itemlist as an argument -- all you lose is one of the progress bars [06:33] my idea is so have a collection of hard drive images of pc fresh out of box [06:33] so people can release setting up a old 2000 gateway pc with all the junk on it [06:34] *** pikhq has quit IRC (Ping timeout: 244 seconds) [06:34] godane: good idea [06:35] Somebody2: thanks, will do [06:35] Now all I need is a decent LTE connection.. damn thing [06:35] OK, cool. I'm going to head to sleep then. Let us know how it goes! [06:36] some of this may not be possible cause it would need un-open pcs [06:39] *** SketchCow has quit IRC (Read error: Operation timed out) [06:39] *** SketchCow has joined #archiveteam-bs [06:40] *** eprillios has quit IRC (Read error: Operation timed out) [06:41] *** Baljem has quit IRC (Read error: Operation timed out) [06:42] *** mundus20- has quit IRC (Read error: Operation timed out) [06:43] *** mundus201 has joined #archiveteam-bs [06:43] *** brayden has quit IRC (Read error: Operation timed out) [06:44] *** Ravenloft has quit IRC (Read error: Connection reset by peer) [06:44] *** dashcloud has quit IRC (Read error: Operation timed out) [06:44] *** eprillios has joined #archiveteam-bs [06:45] *** pikhq has joined #archiveteam-bs [06:47] *** dashcloud has joined #archiveteam-bs [06:57] *** Baljem has joined #archiveteam-bs [07:02] *** schbirid has joined #archiveteam-bs [07:03] Somebody2: eta of 10 minutes now LOL [07:27] *** schbirid2 has joined #archiveteam-bs [07:30] *** schbirid has quit IRC (Read error: Operation timed out) [07:32] *** username1 has joined #archiveteam-bs [07:34] *** schbirid2 has quit IRC (Read error: Operation timed out) [07:36] bwm: is it normal to see it do stuff like this please? https://usercontent.irccloud-cdn.com/file/tf7rexSG/ [08:05] bwn: you around? [08:07] *** schbirid2 has joined #archiveteam-bs [08:10] *** username1 has quit IRC (Read error: Operation timed out) [08:24] *** yipdw has joined #archiveteam-bs [08:54] *** odemg has quit IRC (Remote host closed the connection) [09:00] *** GE has joined #archiveteam-bs [09:13] Somebody2: - so it got to 70% and was using all of the RAM on the server - its now hung [09:28] *** username1 has joined #archiveteam-bs [09:31] *** schbirid2 has quit IRC (Read error: Operation timed out) [09:59] *** odemg has joined #archiveteam-bs [10:00] *** schbirid2 has joined #archiveteam-bs [10:03] *** username1 has quit IRC (Read error: Operation timed out) [10:10] *** brayden has joined #archiveteam-bs [10:13] *** JensRex has quit IRC (Remote host closed the connection) [10:13] *** JensRex has joined #archiveteam-bs [10:18] *** Jonison has joined #archiveteam-bs [10:22] *** GE has quit IRC (Remote host closed the connection) [10:31] *** username1 has joined #archiveteam-bs [10:35] *** schbirid2 has quit IRC (Read error: Operation timed out) [10:39] *** odemg has quit IRC (Remote host closed the connection) [10:40] *** odemg has joined #archiveteam-bs [10:58] *** schbirid2 has joined #archiveteam-bs [11:02] *** username1 has quit IRC (Read error: Operation timed out) [11:09] *** jspiros has quit IRC (leaving) [11:10] *** jspiros has joined #archiveteam-bs [11:16] *** Jonison has quit IRC (Read error: Connection reset by peer) [11:20] *** username1 has joined #archiveteam-bs [11:22] *** odemg2 has joined #archiveteam-bs [11:22] *** odemg has quit IRC (Read error: Operation timed out) [11:23] *** odemg2 has quit IRC (Remote host closed the connection) [11:23] *** odemg2 has joined #archiveteam-bs [11:24] *** schbirid2 has quit IRC (Read error: Operation timed out) [12:01] *** odemg2 has quit IRC (Remote host closed the connection) [12:26] *** username1 is now known as schbirid [12:46] *** GE has joined #archiveteam-bs [13:34] *** pizzaiolo has joined #archiveteam-bs [13:42] *** odemg has joined #archiveteam-bs [13:45] *** pnJay has joined #archiveteam-bs [13:45] *** RichardG has quit IRC (Read error: Operation timed out) [13:54] *** dxrt_ is now known as dxrt [14:01] *** odemg has quit IRC (Remote host closed the connection) [14:07] *** kyounko has quit IRC (Max SendQ exceeded) [14:07] *** REiN^ has quit IRC (Max SendQ exceeded) [14:07] *** REiN^ has joined #archiveteam-bs [14:08] *** kyounko has joined #archiveteam-bs [14:32] any idea how i get all the "licenses, musicinfo, stats, lyrics" in a query via https://developer.jamendo.com/v3.0/tracks ? [14:35] *** pnJay has quit IRC (Read error: Connection reset by peer) [14:35] is it not just a case of using include? [14:35] *** pnJay has joined #archiveteam-bs [14:37] its not really clear if you can do more than one at a time though [14:45] thats what i mean =) [15:02] *** schbirid2 has joined #archiveteam-bs [15:04] *** schbirid has quit IRC (Read error: Operation timed out) [15:06] *** Frogging has quit IRC (Read error: Operation timed out) [15:07] *** Frogging has joined #archiveteam-bs [15:22] *** username1 has joined #archiveteam-bs [15:25] *** schbirid2 has quit IRC (Read error: Operation timed out) [15:47] what happens if you use mutliple include= ? [15:49] maybe include=licenses&include=musicinfo or include[]=licenses&include[]=musicinfo [15:54] *** schbirid2 has joined #archiveteam-bs [15:54] HCross2: this is for the *mini* census, really? [15:54] Or were you trying with the pex version on the full list? [15:57] *** username1 has quit IRC (Read error: Operation timed out) [15:59] Somebody2: full list [15:59] It just sort of slowly used all the resources the server could provide [16:05] didnt notice you werent here schbirid2 [16:05] what happens if you use mutliple include= ? [16:05] maybe include=licenses&include=musicinfo or include[]=licenses&include[]=musicinfo [16:28] *** odemg has joined #archiveteam-bs [16:30] *** TC04 has quit IRC (Read error: Connection reset by peer) [16:36] *** TC01 has joined #archiveteam-bs [16:37] alternatively include=musicinfo,licenses [16:46] *** JAA has joined #archiveteam-bs [16:54] include[]=licenses&include[]=musicinfo did the trick, thanks SpaffGarg ! [16:56] sweet [16:56] really isnt clear in that documentation [17:02] *** zino has quit IRC (Remote host closed the connection) [17:05] WunderBlogs grab is basically done, 1.26M URLs fetched, now retrying some 38k errors [17:05] Contained some beautiful traps though, e.g. https://www.wunderground.com/blog/N2jaiCapeMayNJ/www.facebook.com/www.facebook.com/www.twitter.com/www.facebook.com/www.twitter.com/www.twitter.com/archive.html?MR=1 [17:06] Also %3C/%3C/%3C/%3C/... [17:06] Mininova is at 482k done, 127k left, 4k per hour [17:07] *** fie_ has quit IRC (Ping timeout: 246 seconds) [17:09] was there ever an update on what the plan is for mlkshk? [17:10] I think there's a channel (not the one mentioned in the Wiki though) [17:10] yeah the channel is just me and one other person, i guess i thought maybe i missed something [17:12] * JAA shrugs [17:16] Interesting status codes I saw in the WunderBlogs grab: 420, 477, 530, 561. No idea what any of these mean. Sadly no 418. [17:20] *** fie has joined #archiveteam-bs [17:22] *** Simpbrain has quit IRC (Read error: Operation timed out) [17:35] *** Simpbrain has joined #archiveteam-bs [17:37] *** RichardG has joined #archiveteam-bs [17:41] gotta love jamendo, the api stops giving responses after ~25000 tracks [17:41] or maybe they really got this small? [17:41] >:D [17:45] *** username1 has joined #archiveteam-bs [17:49] *** schbirid2 has quit IRC (Read error: Operation timed out) [17:56] mlkshk should be fairly easy to grab. Ids are sequential and encoded with base36(?) [18:23] *** odemg has quit IRC (Remote host closed the connection) [18:45] *** icedice has joined #archiveteam-bs [18:53] *** GE has quit IRC (Remote host closed the connection) [19:23] *** icedice has quit IRC (Ping timeout: 245 seconds) [19:24] *** icedice has joined #archiveteam-bs [19:25] *** schbirid2 has joined #archiveteam-bs [19:28] *** username1 has quit IRC (Read error: Operation timed out) [19:33] *** bwn has quit IRC (Ping timeout: 244 seconds) [19:39] *** Ravenloft has joined #archiveteam-bs [19:41] *** icedice2 has joined #archiveteam-bs [19:44] *** icedice has quit IRC (Ping timeout: 250 seconds) [20:02] *** bwn has joined #archiveteam-bs [20:06] *** bwn has quit IRC (Client Quit) [20:13] *** ndiddy has joined #archiveteam-bs [20:15] *** pnJay has quit IRC (Quit: Leaving) [20:18] *** GE has joined #archiveteam-bs [20:23] *** bwn has joined #archiveteam-bs [20:46] *** dcmorton has joined #archiveteam-bs [20:53] *** username1 has joined #archiveteam-bs [20:58] *** schbirid2 has quit IRC (Read error: Operation timed out) [21:01] So, Mininova torrents -- how can we pull this off? There are about 70k torrents, 3.5 TiB total, and roughly 90% have only 1 seeder (most likely Mininova). They're all going to die on 4th of April. [21:01] are you really trying to download all the torrents? [21:01] bold [21:01] For those who only know Mininova from way back (pre-2009): those torrents are content distribution uploaded by the publisher, not TPB-like torrents. [21:01] *** pnJay has joined #archiveteam-bs [21:02] ah interesting [21:02] Well, no. I don't have the resources for that. :-/ [21:13] *** Zebranky has quit IRC (Ping timeout: 633 seconds) [21:24] *** Zebranky has joined #archiveteam-bs [21:35] curious that codeplex is shutting down at the same time that google opened its own PR campa--uh I mean free software project repository [21:36] what, google code? [21:36] *** schbirid2 has joined #archiveteam-bs [21:37] xmc: no, opensource.google.com [21:37] another one?? [21:37] Microsoft's [21:38] ¯\_(ツ)_/¯ [21:38] Ugh [21:38] they're going to have a readonly archive and a Move To Github button so it doesn't look like a big deal [21:39] as well as a "project moved" banner [21:39] codeplex is doing the right thing [21:41] *** username1 has quit IRC (Read error: Operation timed out) [21:47] yipdw: ideally archive.org should campaign or make it easy for projects to donate their stuff to archive.org before they die [21:47] it would save AT a lot of trouble [21:48] I guess, though the solution here seems practical [21:48] pizzaiolo: you can upload any kind of file to the IA [21:48] nothing stopping website owners from doing that [21:48] I'm aware [21:49] but you know, you gotta make it dumb-proof and easy enough that lazy admins would do it [21:49] there's also the practical problem of shipping gigabytes over the Internet [21:50] also, a campaign like that needs $ [21:50] both to run the campaign and the expectation of being paid as the custodian [21:50] nobody expects the latter, for some reason [21:53] JAA: upload the .torrent files to IA ;) [22:02] *** username1 has joined #archiveteam-bs [22:06] *** schbirid2 has quit IRC (Read error: Operation timed out) [22:09] *** BlueMaxim has joined #archiveteam-bs [22:14] *** db420 is now known as dboard [22:15] *** username1 has quit IRC (Quit: Leaving) [22:50] Yeah, I don't think so either. ;-) [23:10] looks like Arirang delete alot of stuff hosted on there site [23:21] *** odemg has joined #archiveteam-bs [23:23] *** db420 has joined #archiveteam-bs [23:23] *** dboard has quit IRC (Read error: Operation timed out) [23:23] *** balrog_ has joined #archiveteam-bs [23:23] *** balrog has quit IRC (Read error: Operation timed out) [23:23] *** Jay_ has joined #archiveteam-bs [23:24] *** whydomain has quit IRC (Write error: Broken pipe) [23:24] *** REiN^ has quit IRC (Read error: Operation timed out) [23:25] *** phuzion has quit IRC (Read error: Operation timed out) [23:25] *** marvinw has quit IRC (Read error: Operation timed out) [23:25] *** jtn2 has quit IRC (Write error: Broken pipe) [23:25] *** joepie91 has quit IRC (Read error: Operation timed out) [23:25] *** dxrt has quit IRC (Read error: Operation timed out) [23:25] *** antomati_ has joined #archiveteam-bs [23:25] *** zerkalo has quit IRC (Read error: Operation timed out) [23:25] *** balrog_ has quit IRC (Read error: Operation timed out) [23:25] *** db420 has quit IRC (Read error: Operation timed out) [23:25] *** ranma has quit IRC (Read error: Operation timed out) [23:26] *** dxrt has joined #archiveteam-bs [23:26] *** fenn has quit IRC (Read error: Operation timed out) [23:26] *** fenn has joined #archiveteam-bs [23:26] *** RKenshin has joined #archiveteam-bs [23:26] *** SadDM has quit IRC (Read error: Operation timed out) [23:26] *** chfoo has quit IRC (Read error: Operation timed out) [23:26] *** acridAxid has quit IRC (Read error: Operation timed out) [23:26] *** tephra has quit IRC (Read error: Operation timed out) [23:26] *** Dark_Star has quit IRC (Read error: Operation timed out) [23:27] *** tfgbd_znc has quit IRC (Read error: Connection reset by peer) [23:27] *** Kenshin has quit IRC (Write error: Broken pipe) [23:27] *** arkiver has quit IRC (Ping timeout: 601 seconds) [23:27] *** RKenshin is now known as Kenshin [23:27] *** antomatic has quit IRC (Read error: Operation timed out) [23:27] *** mistym has quit IRC (Ping timeout: 246 seconds) [23:27] *** trs80 has quit IRC (Ping timeout: 246 seconds) [23:27] so i'm grabbing stuff off of Arirang TV youtube channel [23:27] i'm doning it by collection playlist [23:27] *** jbroome has quit IRC (Read error: Connection reset by peer) [23:27] one at a time [23:27] first one is going to be simply k-pop [23:27] *** pnJay has quit IRC (Write error: Broken pipe) [23:27] *** wabu has quit IRC (Write error: Broken pipe) [23:27] since we maybe able to get a full run of that [23:27] good news is i maybe to grab a youtube set of Korea Today [23:28] *** swebb has quit IRC (Read error: Operation timed out) [23:28] *** robink has quit IRC (Read error: Operation timed out) [23:28] *** balrog has joined #archiveteam-bs [23:28] *** jspiros has quit IRC (Read error: Operation timed out) [23:28] *** kyounko has quit IRC (Read error: Operation timed out) [23:28] *** robink has joined #archiveteam-bs [23:28] *** phuzion has joined #archiveteam-bs [23:28] *** tfgbd_znc has joined #archiveteam-bs [23:29] *** whydomain has joined #archiveteam-bs [23:29] *** joepie91 has joined #archiveteam-bs [23:29] *** tephra has joined #archiveteam-bs [23:29] *** marvinw has joined #archiveteam-bs [23:30] *** mistym has joined #archiveteam-bs [23:31] *** arkiver has joined #archiveteam-bs [23:31] *** jtn2 has joined #archiveteam-bs [23:32] *** jbroome has joined #archiveteam-bs [23:32] *** wabu has joined #archiveteam-bs [23:32] *** JAA has quit IRC (Quit: Page closed) [23:34] *** swebb has joined #archiveteam-bs [23:35] *** chfoo has joined #archiveteam-bs [23:36] *** ranma has joined #archiveteam-bs [23:36] *** db420 has joined #archiveteam-bs [23:36] *** Dark_Star has joined #archiveteam-bs [23:37] *** btfo has quit IRC (Ping timeout: 600 seconds) [23:39] *** zerkalo has joined #archiveteam-bs [23:40] *** acridAxid has joined #archiveteam-bs [23:59] *** GE has quit IRC (Remote host closed the connection)