[00:29] *** JesseW has joined #archiveteam [00:34] I notice that the "PEOPLE ALSO FOUND" section below the detail body does not exclude the currently viewed collection [00:35] it puts the fear of duping in the uploader [00:36] oh, sorry. this is not IA [00:36] Zandro: feel free to discuss this in #internetarchive , if you'd like [00:43] *** jspiros has quit IRC (Ping timeout: 186 seconds) [00:56] *** jspiros has joined #archiveteam [00:59] *** xk_id has quit IRC (Remote host closed the connection) [00:59] *** wyatt8740 has quit IRC (Remote host closed the connection) [01:03] *** wyatt8740 has joined #archiveteam [01:09] *** aaaaaaaa_ has joined #archiveteam [01:09] *** aaaaaaaaa has quit IRC (Read error: Connection reset by peer) [01:17] *** aaaaaaaa_ is now known as aaaaaaaaa [01:17] *** xk_id has joined #archiveteam [01:21] *** primus104 has quit IRC (Leaving.) [01:30] *** JesseW has quit IRC (Leaving.) [01:30] *** JesseW has joined #archiveteam [01:33] *** oli has quit IRC (Read error: Operation timed out) [01:39] *** oli has joined #archiveteam [01:46] *** JesseW has quit IRC (Read error: Operation timed out) [01:47] *** c_b2 has joined #archiveteam [01:47] *** JesseW has joined #archiveteam [01:51] *** c_b has quit IRC (Read error: Operation timed out) [01:53] *** c_b2 is now known as c_b [02:11] *** c_b has quit IRC (c_b) [02:12] *** JesseW has quit IRC (Read error: Operation timed out) [02:18] looking into my upload, it will not complete the derive step due to its attempt to download from the dead swarm. [02:22] *** khaoohs has joined #archiveteam [02:24] transmission 8h41s idle... hope it times out before the 3 day limit for the next derive task [02:25] *** khaoohs__ has quit IRC (Ping timeout: 240 seconds) [03:00] *** RichardG_ has joined #archiveteam [03:00] *** RichardG has quit IRC (Read error: Connection reset by peer) [03:01] *** khaoohs has quit IRC (Read error: Operation timed out) [03:01] *** slyphic_ has quit IRC (Read error: Operation timed out) [03:01] *** atlogbot has quit IRC (Read error: Operation timed out) [03:02] *** no2pencil has quit IRC (Read error: Operation timed out) [03:03] *** no2pencil has joined #archiveteam [03:03] *** swebb has quit IRC (Ping timeout: 369 seconds) [03:04] *** vOYtEC has quit IRC (Ping timeout: 369 seconds) [03:04] *** ohhdemgir has quit IRC (Read error: Operation timed out) [03:05] *** vOYtEC has joined #archiveteam [03:05] *** dserodio has quit IRC (Read error: Operation timed out) [03:05] *** chazchaz has quit IRC (Read error: Operation timed out) [03:07] *** mistym has quit IRC (Ping timeout: 369 seconds) [03:07] *** slyphic has joined #archiveteam [03:07] *** atlogbot has joined #archiveteam [03:10] *** Laverne has quit IRC (Ping timeout: 369 seconds) [03:12] *** khaoohs has joined #archiveteam [03:13] *** robink has quit IRC (Ping timeout: 492 seconds) [03:13] *** slyphic_ has joined #archiveteam [03:13] *** atlogbot has quit IRC (Ping timeout: 369 seconds) [03:14] *** Laverne has joined #archiveteam [03:14] *** dserodio has joined #archiveteam [03:14] *** robink has joined #archiveteam [03:15] *** atlogbot has joined #archiveteam [03:16] *** swebb has joined #archiveteam [03:17] *** mistym has joined #archiveteam [03:18] *** chazchaz has joined #archiveteam [03:27] *** slyphic has quit IRC (Read error: Operation timed out) [03:32] *** wp494_ has joined #archiveteam [03:38] *** wp494 has quit IRC (Ping timeout: 483 seconds) [03:39] *** slyphic_ has quit IRC (Read error: Operation timed out) [03:39] *** chazchaz has quit IRC (Read error: Operation timed out) [03:39] *** Laverne has quit IRC (Read error: Operation timed out) [03:40] *** mistym has quit IRC (Ping timeout: 369 seconds) [03:40] *** slyphic has joined #archiveteam [03:40] *** chazchaz has joined #archiveteam [03:41] *** Laverne has joined #archiveteam [03:42] *** mistym has joined #archiveteam [04:07] *** wutno has quit IRC (Read error: Operation timed out) [04:10] *** Wyatts has quit IRC (Remote host closed the connection) [04:14] *** Wyatts has joined #archiveteam [04:27] *** aaaaaaaaa has quit IRC (Leaving) [04:42] *** JesseW has joined #archiveteam [04:48] *** khaoohs_ has joined #archiveteam [04:54] *** khaoohs has quit IRC (Ping timeout: 483 seconds) [05:12] *** oli has quit IRC (Read error: Operation timed out) [05:15] *** oli has joined #archiveteam [05:18] *** scyther has joined #archiveteam [05:48] *** wyatt8740 has quit IRC (Read error: Operation timed out) [05:56] *** scyther has quit IRC (Read error: Connection reset by peer) [06:02] *** wyatt8740 has joined #archiveteam [06:03] *** primus104 has joined #archiveteam [06:23] *** habi has joined #archiveteam [06:23] *** habi has left [06:39] *** JesseW has quit IRC (Read error: Operation timed out) [07:10] *** xk_id has quit IRC (Read error: Operation timed out) [07:11] *** PurpleSym has joined #archiveteam [07:13] *** trs80 has quit IRC (Ping timeout: 186 seconds) [07:14] *** trs80 has joined #archiveteam [07:18] *** schbirid has joined #archiveteam [07:30] *** trs80 has quit IRC (Ping timeout: 186 seconds) [07:30] *** trs80 has joined #archiveteam [07:41] *** atomotic has joined #archiveteam [07:52] *** jspiros has quit IRC (Ping timeout: 186 seconds) [07:59] *** wp494_ is now known as wp494 [08:31] *** jspiros has joined #archiveteam [08:32] *** xk_id has joined #archiveteam [08:33] *** primus104 has quit IRC (Leaving.) [08:41] *** RedType has quit IRC (Ping timeout: 252 seconds) [08:43] *** Elegance_ has quit IRC (Read error: Operation timed out) [08:46] *** zenguy_pc has quit IRC (Read error: Connection reset by peer) [08:48] *** brayden has joined #archiveteam [08:49] *** Elegance has joined #archiveteam [08:58] *** MMovie has joined #archiveteam [09:01] *** MMovie2 has quit IRC (Ping timeout: 306 seconds) [09:02] *** zenguy_pc has joined #archiveteam [09:05] *** brayden has quit IRC (Quit: Leaving) [09:20] *** brayden has joined #archiveteam [09:35] *** RedType has joined #archiveteam [09:43] *** wp494 has quit IRC (Read error: Connection reset by peer) [09:56] *** wp494 has joined #archiveteam [09:57] *** RedType has quit IRC (Ping timeout: 483 seconds) [10:15] *** RedType has joined #archiveteam [10:33] *** anomie has quit IRC (Read error: Connection reset by peer) [10:39] *** Wyatts has quit IRC (Remote host closed the connection) [10:41] *** anomie has joined #archiveteam [10:43] *** Wyatts has joined #archiveteam [10:55] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [11:02] *** primus104 has joined #archiveteam [11:16] *** dashcloud has quit IRC (Ping timeout: 483 seconds) [11:18] *** dashcloud has joined #archiveteam [11:25] *** primus104 has quit IRC (Leaving.) [11:42] *** Boppen has quit IRC (Read error: Connection reset by peer) [11:43] *** Boppen has joined #archiveteam [11:44] *** arkiver2 has joined #archiveteam [11:48] *** atomotic has joined #archiveteam [12:18] *** primus104 has joined #archiveteam [12:25] *** vitzli has joined #archiveteam [12:27] *** BlueMaxim has quit IRC (Quit: Leaving) [12:31] *** scyther has joined #archiveteam [12:35] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [13:11] *** vitzli has quit IRC (Quit: Leaving) [13:18] *** primus104 has quit IRC (Leaving.) [13:18] *** arkiver2 has joined #archiveteam [13:27] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [13:30] *** arkiver2 has joined #archiveteam [13:37] *** PurpleSym has quit IRC (Remote host closed the connection) [13:45] *** RichardG_ is now known as RichardG [13:46] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [13:48] *** habi has joined #archiveteam [13:51] *** K4k has joined #archiveteam [13:57] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [14:03] *** PurpleSym has joined #archiveteam [14:15] *** habi has left [14:50] *** scyther has quit IRC (Read error: Connection reset by peer) [15:09] *** vitzli has joined #archiveteam [15:11] *** Stilett0 has joined #archiveteam [15:13] *** Stiletto has quit IRC (Ping timeout: 306 seconds) [15:13] cfarence: only http://market.envato.com/ is closing or other parts too? [15:14] cfarence: and all these 8 sites (http://themeforest.net/, etc) are closing too and are part of the envato market? [15:26] arkiver: just the Flash site [15:26] "activeden" [15:27] so only http://activeden.net/ [15:34] *** Alice_ has joined #archiveteam [15:52] *** Alice_ has quit IRC (Ping timeout: 240 seconds) [16:08] *** JesseW has joined #archiveteam [16:10] wow [16:10] my archiveteam stopped pushing [16:26] *** wutno has joined #archiveteam [16:34] *** JesseW has quit IRC (Read error: Operation timed out) [16:42] *** primus104 has joined #archiveteam [17:11] *** SimpBrain has joined #archiveteam [17:49] *** vitzli has quit IRC (Quit: Leaving) [17:52] *** nertzy has joined #archiveteam [18:00] *** mksplg has joined #archiveteam [18:08] *** nertzy has quit IRC (Quit: This computer has gone to sleep) [18:08] Minor thing: We're freaking IA a little. [18:08] Just a little. [18:21] *** winr4r has joined #archiveteam [18:22] SketchCow: why's that? [18:26] *** RichardG has quit IRC (Remote host closed the connection) [18:28] Doubled intake [18:29] I think we're one of the factors, not the factor. [18:34] --------------------------------------------------- [18:34] GRAB THINGIVERSE IMMEDIATELY [18:34] --------------------------------------------------- [18:34] I M M E D I A T E L Y [18:34] I know godane has done it, I know others can. Move. Top priority. [18:37] *** habi has joined #archiveteam [18:38] OK! [18:39] SketchCow: what's happenning over at thingiverse? [18:39] I don't entirely know [18:39] https://www.thingiverse.com/thing:192937 [18:39] I think they're going to kill it [18:40] arkiver: need any help with grabbing script work? [18:40] it's just empty [18:40] got a backend fetch fail on first attempt to load dashboard, not sure if related [18:40] *** RichardG has joined #archiveteam [18:40] joepie91: I think I'll be fine, I'll ping you if I need any help [18:40] uhh [18:40] HTTP 500 error [18:40] https://www.thingiverse.com/thing:1008750 [18:40] Will first find out if we can actually get the stuff still [18:41] Looking at the site I should be able to have the scripts ready very fast if there's still anything to download [18:41] arkiver: I see 500 everywhere... [18:41] yeah [18:41] arkiver: We have a thingiverse downloader. [18:41] maybe that's just the html pages [18:41] it worked seconds ago [18:41] err minutes [18:42] *** habi1 has joined #archiveteam [18:42] SketchCow: where is it? [18:42] arkiver: it's slow and eventually 500s. I'm guessing a backend service is unavailable [18:42] *** habi has quit IRC (Read error: Connection reset by peer) [18:43] Luckily I'm buddies with Bre Pettis. Hitting him up for info. [18:43] (Bre is no longer associated with the company) [18:44] stuff is still available https://thingiverse-production-new.s3.amazonaws.com/zipfiles/5f/47/93/7a/4f/SCH_13mm.zip [18:49] *** beardicus has quit IRC (Quit: bye now) [18:49] SketchCow: I know how to get the stuff [18:49] well not the html, the files [18:50] getting scripts ready now [18:52] *** beardicus has joined #archiveteam [18:54] tested a bit more [18:55] we can't get the zip files [18:55] we can get the individual models [18:55] arkiver: why's that? [18:56] well, the zip are redirected to through a link that is not available (500) [18:56] the individual models are still downloadable through an other link [18:56] that's still workin [18:56] working* [18:56] arkiver: but you just linked a zip? [18:57] yeah, I found it through godane's grabs [18:57] just to check if their files weren't deleted yet [18:57] ah [18:57] arkiver: how are you getting them? [18:57] http://www.thingiverse.com/download:183243 [18:57] for now we're getting the inidividual models, the other things will be looked at later [18:57] most important info first [18:58] might be worth checking which files are already at IA [18:58] arkiver: and what did you use to try and get at the ZIPs? [18:58] (just wanting to explore a bit) [18:58] http://www.thingiverse.com/thing:79692/zip [18:58] that zip should contain http://www.thingiverse.com/download:183243 and http://www.thingiverse.com/download:183244 [18:58] from https://webcache.googleusercontent.com/search?q=cache:www.thingiverse.com/thing:79692 [18:59] I see [18:59] thanks, going to have a poke around [18:59] chfoo: can you please add thingiverse to the project.json and add a FOS rsync? [19:00] joepie91: thanks! [19:04] uh oh [19:04] arkiver: I think they just went down? [19:04] what? [19:05] they work for me [19:05] getting cloudflare errors [19:05] even on the download: link [19:05] scripts almost ready [19:05] arkiver: so account for HTTP 522 errors [19:05] yes [19:05] I'l continue the grab on 522 error [19:05] errors [19:07] actually [19:07] what shall I do continue or abort on 522 [19:07] I think I'll do one file per item [19:07] arkiver: well, they are supposedly temporary. pause and retry? [19:07] so I can jsut do abort [19:08] yeah [19:08] (was first doing 10) [19:08] it's back for me [19:10] ok [19:10] arkiver: and down again, now a HTTP 521 [19:10] Origin Down [19:11] *** habi1 has left [19:12] ok, scripts working [19:12] arkiver: want me to spin up a script? [19:12] wait [19:14] *** aaaaaaaaa has joined #archiveteam [19:14] I need an rsync target [19:14] arkiver: how much data are we talking? [19:14] ballpark [19:15] No idea [19:15] I don't think more then 1T [19:15] because I just got a new box with 4TB disk and 10TB traffic @ 1gbps [19:15] but it's single-disk [19:15] ok, please PM me the rsync target [19:15] and it's a new disk [19:15] :) [19:15] so it's not _guaranteed_ that it won't fail [19:15] hmm [19:15] (disks usually fail either very early or very late) [19:16] arkiver: do we have any alternative RAID-backed rsync targets? [19:16] or is it either this one or none? [19:16] (cc Kenshin ) [19:16] ------------------------------ [19:16] IMPORTANT: [19:16] We need a reliable rsync target ASAP [19:16] ------------------------------ [19:16] hmm [19:16] I'll use the gamefront rsync [19:16] SketchCow ^ [19:17] and joepie91 ^ [19:17] arkiver: I'll set up an rsync target on my box anyway - if you end up not having anything else, you at least have a fallback that hopefully won't fail [19:18] ok! [19:18] preparing items now [19:21] *** scyther has joined #archiveteam [19:22] items being added [19:22] I'll pause blingee [19:23] we have started! [19:23] it's not yet in the warrior, but it can be run manually [19:23] arkiver: link to repo? [19:23] http://tracker.archiveteam.org/blingee/ [19:23] https://github.com/ArchiveTeam/thingiverse-grab [19:23] wrong tracker [19:23] :P [19:23] oops [19:23] http://tracker.archiveteam.org/thingiverse/ [19:23] arkiver: you're missing the README btw [19:23] yeah [19:24] will be added now [19:24] wasn't vital [19:25] wow [19:25] need grabber [19:25] I SETUP [19:26] not linked to wiki [19:26] shame [19:26] SketchCow: grab of thiingiverse started [19:26] arkiver: let me know when you've added the README and checked that the list of deps is correct etc [19:26] Only files currently [19:27] joepie91: I'll just use the template [19:27] what does actually happen [19:27] if i run my googlecode grab [19:27] and it isnt started? [19:27] does it start when itstarts? [19:29] limebyte: most likely [19:29] so the warrior idles [19:29] unti you press play or what? [19:30] limebyte: not sure what you mean, but probably better suited for -bs [19:30] readme added. [19:30] we have 1.6 million files to go [19:30] give it all you have I'd say [19:30] okay [19:33] arkiver: you typoed the repo name [19:33] missing -grab [19:33] arkiver: hold, PR incoming [19:34] ya [19:34] didnt worked [19:34] shame on you [19:34] * arkiver is afk for 20 minutes [19:34] arkiver: https://github.com/ArchiveTeam/thingiverse-grab/pull/1 [19:34] accept pls [19:34] :P [19:35] danke [19:35] 429 is the rate limited status I think [19:35] bitte [19:35] hmm, joepie91: maybe we should go with the 10 or 100 files/item [19:36] not sure if the tracker can hold [19:36] hmmm [19:36] currently doing 1200/min [19:37] wow [19:37] poop, another error [19:37] arkiver: wait 1 min please [19:37] ok [19:38] Getting a decent amount of 503s [19:38] arkiver: new PR [19:38] arkiver: hmm. won't it interfere with the gamefront data? [19:38] gamefront didn't start yet [19:38] ahh [19:38] right [19:39] and all thiingiverse warc's have prefix thingiverse- [19:39] right :P [19:39] arkiver: filesizes look right to you? [19:39] yes [19:39] ok [19:40] i'm awake [19:41] joepie91: most are very small and compress very well, example: https://www.thingiverse.com/download:220747 [19:41] i will tell you that thingiverse has about 90k things between 2009 to 2013 [19:42] arkiver: makes sense [19:47] wikipedia: There were 25,000 designs uploaded to Thingiverse as of November 2012[4] and more than 100,000 in June 2013.[5] The 400000th Thing was published on the 19 July 2014[6] [19:48] exponential growth [19:48] probably safe to assume a few million by now [19:48] lol [19:51] they may have been losing money on each upload, but made up for it in volume [19:52] 'Tracker returned status code 500. The tracker has probably malfunctioned.' on thingiverse-grab - or is it just me? [19:52] ah, off it goes again. sorry! [19:58] we break the tracker again? [19:58] lol [19:58] 1.6 million [20:00] *** SimpBrain has quit IRC (Quit: Leaving) [20:02] is it --context-value shared:rsync_threads=N to change the number of rsync jobs at once? I seem to be stuck on 1 which looks like a bit of a bottleneck [20:02] *** RichardG has quit IRC (Remote host closed the connection) [20:07] IIRC that is ignored for some reason. Plus, overloading the rsync target causes other problems. [20:09] *** RichardG has joined #archiveteam [20:10] *** wyatt8760 has joined #archiveteam [20:10] *** wyatt8760 is now known as wyatt|cla [20:10] *** wyatt|cla is now known as wyattclas [20:10] *** wyattclas is now known as wyatt8760 [20:15] *** RedType has quit IRC (Ping timeout: 1730 seconds) [20:20] *** SmileyG has quit IRC (Ping timeout: 240 seconds) [20:25] what is the proper way to make a m3u for uploading to archive.org? [20:26] because i just noticed the m3u I uploaded had hard paths to files in it -_- [20:28] *** Smiley has joined #archiveteam [20:31] *** RichardG has quit IRC (Remote host closed the connection) [20:46] *** Morbus has quit IRC (Quit: http://www.disobey.com/) [20:47] just remove the paths in a text editor ;) [20:47] *** schbirid has quit IRC (Quit: Leaving) [20:49] *** RichardG has joined #archiveteam [20:50] *** Morbus has joined #archiveteam [20:53] *** zenguy_pc has quit IRC (Read error: Connection reset by peer) [20:55] *** wyatt8760 has quit IRC (Read error: Operation timed out) [20:57] *** wyatt8760 has joined #archiveteam [21:10] *** zenguy_pc has joined #archiveteam [21:12] https://twitter.com/thingiverse/status/646769701285175297 [21:15] *** scyther has quit IRC (Read error: Connection reset by peer) [21:16] *** PurpleSym has quit IRC (Remote host closed the connection) [21:16] *** ersi has quit IRC (Read error: Operation timed out) [21:17] wow [21:17] such Archiveteam DDOS [21:17] nah [21:17] i hope i dont get abuse [21:17] this one is not our fault [21:17] :) [21:17] sure? :D [21:17] yeah [21:17] it was breaking before we even had a script [21:18] is there Archiveteam 2.0 outside? [21:18] maybe [21:18] lol [21:18] but, -bs [21:18] :p [21:19] strict rules [21:19] eh [21:22] *** K4k has quit IRC (Ping timeout: 186 seconds) [21:26] jumping in on thingiverse now [21:26] yay, my 2.5mbit u/l line can finally be put to good use aside from personal backups! [21:32] *** ersi has joined #archiveteam [21:54] *** wyatt8760 has quit IRC (Read error: Operation timed out) [22:01] schbirid: do you mean make them look in the current folder (e.g. './filename.flac')? [22:02] or would it lack the / [22:02] *./ [22:03] I believe that if they are in the same directory, you can just do the file name [22:15] Looks like tracker isn't pleased with us [22:18] matthusby: that has a tendency of happening :P [22:18] matthusby: kind of amusing - the one part of archiveteam infra that's prone to failure, is the Ruby thing [22:18] (iirc it was Ruby anyway) [22:20] hm, what seems to be wrong with the tracker [22:21] xmc: been seeing some slowdowns. we're probably just hammering it [22:21] lots of tiny tasks [22:21] hm ok [22:21] oh [22:21] it's actually Really Down now [22:21] @ xmc [22:22] The Phusion Passenger application server encountered an error while starting your web application. [22:22] hahaha [22:23] it is being updated [22:23] https://github.com/ArchiveTeam/universal-tracker/commit/4770a65e12091cb4b3c1921efe3b823241abf728 [22:24] *** vegbrasil has joined #archiveteam [22:24] aaaaaaaaa: oh heh [22:25] in other news, looks like global admins can now update the projects.json [22:25] err, in related news [22:26] oh [22:26] good news everybody! :P [22:44] *** RichardG has quit IRC (Remote host closed the connection) [22:46] *** RichardG has joined #archiveteam [22:53] *** RichardG has quit IRC (Ping timeout: 255 seconds) [22:55] *** RichardG has joined #archiveteam [23:03] *** RichardG has quit IRC (Ping timeout: 362 seconds) [23:04] typically when you have a tracker failure it's not the app, it's the datastore [23:04] but this is pointless demarcation [23:04] *** RichardG has joined #archiveteam [23:07] *** RichardG has quit IRC (Remote host closed the connection) [23:15] *** cfarence has quit IRC (Quit: Page closed) [23:16] Thiniverse is back [23:32] <100K items to go on thingiverse tracker [23:37] so you guys are grabbing it now? [23:37] i hope it grabs images and the zip file [23:37] We are, but I'm concerned we're not doing the full thing. [23:37] But let's emergency grab right now [23:37] ok [23:37] i will still do my grabs [23:38] my code: wget -x -i index.txt -p --content-disposition --no-check-certificate --warc-file=$website-thing-id-$start-to-$end-$(date +%Y%m%d) --warc-cdx -H --domains=thingiverse.com,s3.amazonaws.com --reject-regex="(target_thumb|/icon-flag.png|/mb-images/|/img/|/fonts/)" -w 0.3 -E --warc-max-size=1G -o wget.log [23:39] you have to have a wait or you will get block [23:39] but the block is short luckly [23:40] *** RedType has joined #archiveteam [23:43] wp494: not sure it's all the items [23:43] seems only 500k [23:43] I suspect there's more [23:55] so 10 lost episodes of JIBS News General 2008-02 is saved [23:55] 2008-02-10 episode couldn't be saved with vlc [23:56] it was dead dead [23:56] the lost episodes are only dead when grabbing in mplayer