[00:01] Rebooting things is a mistake we all make. [00:01] I'll check out Warrior this afternoon. [00:02] I was already rather impressed with the interface.. if it isn't better though they're fired. [00:03] DoubleJ: using that method, how would you prevent any further pipeline downloads? [00:06] Wait, nevermind. [00:20] GLaDOS: Glad you figured it out, 'cause I don't really know what you were asking :) [00:20] if you meant "touch STOP" that works like always. [00:52] root@teamarchive-1:/1# rm -rf PICPLZ [00:52] yay [00:52] done [00:52] * BlueMax pops the champagne [01:05] awesome [01:06] thnx to alard and doublej instructions got my cinch running on non warrior [01:06] might want to jsut remove the https from git clone and pip otherwise it complains about the cert [01:16] 18G . [01:16] root@teamarchive-1:/1/CINCH# du -sh . [01:22] i'm getting the last episode of making the most of micro [01:22] i'm also not banned from thebox.wz [01:22] We're likely to kill the entire list of cinch sites we have so far in one night. [01:22] That'd be nice. [01:27] Just messing with the archiveteam warrior image now for the first time. It says that there are no active projects. Is that expected? [01:33] i dont think it is swebb but there might be some update command you need to run to get it to download the latest projects [01:36] git must not be updated because I'm following the menu.sh script and it's going a git pull successfully, so I'm guessing that the git repo for the archiveteam warrior hasn't been updated yet for this cinch pull. [01:45] im not sure but im pretty sure people here are running the warrior but yeah maybe they had to do something manual [01:48] I'm going to get it working manually using the archiveteam warrior iso as a base. [01:48] We'll see if I can get it running before the whole job is done. :) [01:50] Yea! I'm grabbing! [01:54] :D [01:54] nice [01:55] swebb: were you using the older warrior, or the newer python-based one? [01:56] I just downloaded the warrior image today. The menu says that it was updated 12 days ago. [01:56] I had to install a bunch of stuff to get the python stuff working. Most notably lua. [01:57] swebb: this one? Warrior v2 beta version: http://archive.org/download/archiveteam-warrior/archiveteam-warrior-v2-20120813.ova [01:57] nope [01:57] I'm using v1. [02:02] Anyone know anything about the pipeline.py script? I wasn't able to start two of them on the same machine due to a "port in use" error. Is there a way to get two running at once? [02:05] pass --help [02:05] it has a parameter for how many concurrent downloads to run [02:05] also, check out http://localhost:8080/ [02:05] Ahh, thanks. [02:06] er [02:06] make that port 8001 [02:06] Yup. Just saw that. Thanks! [02:06] (looking at http://archive.org/details/archiveteam-warrior which also links to the most recent v2) [02:36] swebb [02:36] if you kill your single one and then run it again with the concurrent it will work [02:40] Yea, I've used some of the older ones and I know about the STOP file, so I stopped and restarted with more workers. [02:50] Received item 'cumiscuteonyrfce' from tracker [02:51] SAVED [02:51] Wow! underscore just started like 100 workers. :) [02:52] 16 :D [02:52] what's going on, guys? [02:52] Nintendud: cinch.fm is closing [02:52] idk what that is, but it sounds bad [02:53] is this one of those things that I can use the archiveteam warrior on? [02:53] yes [02:53] cool cool. I'll see if I can finally get that set up. [02:55] the warrior is so awesome [02:55] I'm using the v1 version of the archiveteam warrior and I didn't see it on the list of active projects. Do I need v2 to see it in the menu? [02:55] the only sad part is that it's going to be running as a VM in a VM [02:56] Nintendud: it doesn't take hardly any CPU. [02:56] it's not CPU [02:56] disk IO is slow already :) [02:56] it's going to have even slower disk io [02:56] I'm too lazy to move it closer to the machine [02:57] underscore doubled my efforts in just 5 minutes. :) [02:57] Nintendud: Why double-vm'd? [02:58] swebb: maybe? dunno if the old one still works. should though [02:58] underscor: I don't feel like repartitioning the disk / memory / cpu among three VMs [02:59] I share this server with a friend [02:59] why not run it on the host? [02:59] the host machine is VMWare ESXi. Doesn't have any hard drive space allocated to it [02:59] well, a bare minimum :) [03:00] no biggie though [03:00] should be interesting to kick off this VM from the command line. usually I use a GUI. [03:00] do the workers wait while an upload (sync) is running? It seems like no downloads are going on during the sync when using concurrentcy. [03:01] concurrency [03:01] Nintendud: oic [03:01] swebb: each thread is synchronous [03:01] so if all 6 threads are "uploading" (or however many you're running), then yeah, nothing will download [03:02] ok, but each thread is independent, so it doesn't deadlock or something during an rsync. [03:09] underscor / swebb: Is there a dashboard showing current projects / activity? [03:09] wondering how you knew that underscor spun up workers [03:09] http://tracker.archiveteam.org/cinch/ for cinch [03:09] Nintendud: http://warriorhq.archiveteam.org/ to see what we're saving and who's participating [03:09] and http://tracker.archiveteam.org/cinch/ to see people in action [03:09] http://warriorhq.archiveteam.org/ [03:09] awesome. why is this not more prominent on archiveteam.org? [03:09] s/who's participating/where people are participating from/ [03:10] it's still in beta-ish, iirc [03:10] nice font [03:10] is that droid? [03:10] also the front page is superprotected afaik [03:11] no idea [03:11] I think it's a google web font, if that helps [03:11] yeah, I think it's droid [03:11] Firefox just fubared trying to inspect element :3 [03:12] so cool to watch warriorhq [03:12] lots of eastcoasters [03:12] id be on there in dallas if i was running warrior [03:12] maybe ill load it up here at home [03:14] For cinch, is there more discovery crawling needed to be done? [03:14] i'm uploading all my remainer gamesmaster season 5 to the root (/) of my items-uploads.archive.org account [03:15] this way i can just move them the item id folder when i need to [03:15] i just want to know will this cause problems? [03:15] importing vm :3 [03:16] swebb i believe everything in the tracker is what cinch publishes on their sitemap which is *supposed* to be everything on the site [03:20] damn; looks like I'll have to do this through the vbox gui [03:23] dang the warrior is nice [03:29] think i might have had the biggest one so far. 1.3gb? [03:30] i think the concurrent does deadlock the other threads during an rsync [03:31] at least not in warrior [03:38] underscor, you big fat cheater. :| [03:39] howso? [03:39] :D [03:41] woot. got the warrior working [03:42] why is the maximum number of downloads 6? :3 [03:44] underscor, because you have 2600 items because your e-peen is huge [03:44] I'm predicting that underscore will grab > 50% of the whole site. [03:50] lawl [03:51] trying to get stuff setup on a centos box but stupid lib errors all over the place [03:52] Nintendud i wonder if 6 is a limit on the ajax interface that or prob since its only running with 400mb of ram [03:54] tracker just died for me. [03:55] really? works fine for me. [03:55] back [03:55] yeah [03:55] it doesn't seem to be uploading [03:55] or accepting uploads* [03:55] My uploads have stopped and the tracker page comes up, but no traffic on it. [03:56] yeah [03:56] same here [03:56] something died somewhere. [03:56] odd. [03:56] there's no 'single box' that we're uploading to, right? [03:56] the tracker is a single box, but I'm logged in to it right now [03:57] Yea, we're rsyncing to a single box. [03:57] ah [03:57] that doesn't seem like it would scale [03:57] My downloads are still flowing for existing jobs [03:57] but uploads are hung. [03:57] ok [03:57] my downloads are hung because the one upload thread is blocked [03:58] if I could spawn more download threads, I could be making progress :> [03:58] (with that said, the warrior is one fine piece of software) [03:58] I got an rsync timeout. [03:59] failed connect to fos.textfiles.com: Connection times out. [03:59] yeah me 2. foss is different from the tracker [04:01] i imagine SketchCow must already know :P [04:01] I'm seeing life again on the tracker now. [04:01] I'm guessing that it has recovered. I'm uploading again. [04:01] yup [04:03] underscor needs to stop downloading and let everyone catch up [04:04] What sort of bloody pipe is he working with [04:04] I don't know, but it's *way* bigger than what I've got. [04:05] I've got a big one... 175 urls so far.. [04:05] he has access to archive.org boxen [04:05] Ahh. [04:06] archive.org has a crap-ton of hardware and bandwidth. [04:06] 250... [04:08] ok, we're past the 1/2-way mark. [04:08] 50... [04:08] 350* even [04:14] this guy really loves using cinch... http://cinch.fm/travisdecker [04:14] think he has some mental illness..not joking or being funny seems like he used it to send some messages [04:15] I'm in public with no headphones [04:15] so I can't check it out yet [04:15] 625 urls so far [04:15] I'm downloading scobleizer now! :) [04:15] up to 1500 files so-far [04:15] or wow [04:15] oh wow* [04:16] http://cinch.fm/scobleizer [04:19] long tail is loooooooooong [04:20] 925... I'm really curious how large his file is going to be [04:21] too bad I'll have to head home soon. I'll miss that upload. [04:23] what's the name Nintendud [04:23] oh wait, travis decker [04:23] I'll keep my eyes open for it [04:23] cool cool [04:23] 1050 [04:24] considering that he has something like 100 files over scobleizer, I assume it'll be 1500 or more files? [04:28] scoble had 1700+ urls on cinch. [04:28] Any way to estimate from the user summary? [04:29] I dunno what it counts as 'pages' [04:29] er, URLs [04:29] 1.3GB for just his stuff. :) [04:29] 1350 so far and counting for Mr. travis decker [04:29] OH it's uploading [04:30] looks like ~90 MB [04:30] disappointing [04:30] Yup. 84MB. [04:32] when this is over, I'm going to want stats on stats [04:34] welp, going home. see you guys later [04:35] cya [04:35] the annoying thing about munin is you have to wait a long time to see interesting graphs. [04:36] http://176.58.114.30/localdomain/localhost.localdomain/index.html#archiveteam [04:37] (spoiler: come back in a few hours.) [04:44] cacti is the same way. [04:48] yeah, I would think so [04:53] DFJustin: these are all on my home network and on various free OVH vks's [04:53] http://www.ovh.com/us/vks/index.xml [04:54] hahahaha [04:54] \o/ [04:54] first place [04:55] Dude, you trounced us all! [04:56] http://maelstrom.foxfag.com/munin/foxfag.com/ponder.foxfag.com/fw_conntrack.html hahaha [04:59] My stats are much wimpier: https://skitch.com/scumola/epa9g/cacti [05:04] that's nice [05:05] underscor watch your back [05:05] you pulled one over on me though :( [05:07] oh. now that I'm home... yeah, S[h]O[r]T... travisdecker.. :( [05:08] probably has a mental illness [05:08] good on him tho using the service for whatever his needs were [05:09] yeah [05:09] he has 9 followers. I wonder if he was leaving those for a loved one? [05:10] * Daevien watches the counter spin [05:14] Lord_Nigh: I misread that as Lord_High [05:14] keep on truckin [05:14] wheeeee [05:14] we're almost done [05:14] how many boxes you got going underscor? :p [05:14] only 8 :( [05:15] heh. i've got two, one is in london. and then my crappy home connection [05:15] aha, you're one of the londoners [05:16] http://warriorhq.archiveteam.org/ [05:16] i'm not in london, but have a vps there yeah [05:16] (the points are purposefully obfuscated somewhat) [05:16] sorry, that's what I meant [05:16] the london one is a highly bored linode 512mb vps, figured i may as well do something with it :p [05:17] cool :D [05:17] yeah, most of mine are OVH vKSes that have mostly just been idling [05:17] er, i also appear to be the only one for sure in canada as well lol [05:18] I mean, they do stuff like static file hosting and irc client [05:18] but nothing taxing [05:18] :o cool! [05:18] :D [05:18] i'm trying to smack darkstar and get another position before it's done :p [05:19] I think my box is in france [05:20] somewhere in the cinch datacenter someone is cursing us probably [05:20] THe IP I'm behind is in Ireland though [05:20] hahaha [05:20] they're on ec2 [05:20] so they likely won't notice until they get their usage summary [05:21] :D [05:21] heh yeah and then will go wtf happened? [05:21] lul [05:21] ec2++ [05:21] where is jason's pic of the rubber glove that he's used for his avatar? :p [05:22] I'm kinda amazed we'll have this done inside of 6 hours [05:22] I dunno [05:22] I was looking for it before [05:22] shodul see if we can get a number of users to change their avatar to that :) [05:22] t SketchCow where's your rubberglove picture? [05:22] yeah i looked for it to send to a friend that was liek wtf are you doing? :p [05:22] lol [05:23] since then he's been checkign the counter while he programs lol [05:24] has SketchCow noticed how fast it's going yet or will he be surprised when he gets back to a connection? :p [05:25] he doesn't know [05:25] but he probably won't be too surprised [05:25] heh it's still a ton of data in this time period [05:25] archiveteam tends to do things like this :D [05:25] i usually miss the stuff when its going on or too busy [05:26] Can we have a bat signal [05:26] batsignal@archiveteam.org mailing list? [05:26] we have a mailing list now! [05:26] :O [05:26] see topic [05:26] o_o [05:26] oh right [05:26] <-- blind [05:29] https://groups.google.com/d/msg/archive-team/VloK-lYGA9Q/5GvMs9khL1gJ [05:29] better late than never [05:33] nearing 500 left [05:34] lame, I had to reset the munin bytes & items graphs [05:38] hmm, crap. guess i'm about out of thigns to do and shodul go to bed here in a sec when this finishes :p [05:39] er, negative counter. interesting [05:39] ?? [05:39] oh [05:40] yeah, that happens [05:40] and it's a cinch [05:40] ha [05:40] didn't even have time to get a full gigabyte :( [05:41] Daevien: the negative is because one counter is "number unclaimed" and the other is "number finished", but a finished event decrements the unclaimed counter in the ui [05:41] got one goign with 500mb and still at 70% lol [05:44] underscor :( [05:44] i thought i could make a come back and beat u [05:44] think again! [05:44] underscor generally wins [05:44] heh you'd have had to open some big pipes, he was cruising [05:44] :D [05:44] <3 you all though [05:44] i did open big pipes [05:45] i just didnt notice and get everything going until too late [05:45] had 30 on one server, 100 on another, 200 on another, and 500 on another :P [05:45] underscor is a big fat cheat :| [05:45] S[h]O[r]T: the problem with that is that you get cpustarved [05:46] :P [05:46] because you have 450 runnable threads [05:46] not really [05:46] not running on a vps [05:46] all corequads except 1 machine [05:46] what limit do you hit then? [05:46] nothing [05:46] well, probably upstream to fos actually [05:46] the limit of rsyncing to foss i guess since i dont get more than 1-3MB/s [05:47] but its not because pipe is full [05:47] sinec you can only have one running at a time [05:47] since* [05:47] yup [05:47] 689mb file landed heh. someone had a lot to talk about [05:47] so if i hit a big file with this it stops [05:47] you can only have one rsync across all workthreads at a time [05:47] yep [05:47] :( [05:47] not a huge deal its done :p but yeah [05:47] i beat you on picplz and tabloo i think though [05:48] haha [05:49] probably [05:49] but I didn't have a lot of spare boxen for them [05:49] :) [05:49] this was just good timing [05:53] underscor: you flipped over to 50gb :p [05:53] :D [05:53] * BlueMax throws a ball at underscor [05:53] * underscor catches, returns [05:56] hmm... still >2400 to do (according to the tracker) but I keep getting "no item received"... [05:56] ah. [05:57] now the tracker shows 0 left. [05:57] I have a HUGE upload going [05:57] this is like over 1.5GB [05:57] yow [05:57] cool [05:57] Dark_Star: it seems to be peopel with a lot of processes sending the last bits in [05:57] wwwpubliceyeimagingcom [05:57] after a refresh. I think something in the tracker's code is broken :) [05:57] it's not broken, it works fine [05:57] it has weirdness though [05:58] woot, i managed to just push above you in gb Dark_Star heh [05:58] well I had the window open since yesterday and it kept showing 2300 items left, still counting down. when I pressed F5, it now shows 0 items and no progress [05:58] @Davien :-P [06:00] well that was an easy project... what's next? :) [06:00] once it got going and i noticed where i was int he list, i wanted to stay in the top 10 hehe, stupid things i do late at night when bored :p [06:01] yeah, well, I couldn't do much against it, my upload speed is only about 5% of my download speed... while I could have grabbed more, it would have taken a few days to upload it all :) [06:02] Same here, I upload at about 30kb/s tops :( [06:02] yeah my own connection is usually about 1mbit max even though isp claims 2 [06:02] I'm using a 100mbit box, but... 6 threads :( [06:02] I used to get 1mbit, I now really miss that [06:02] i had a ton of bandwidth quota on two vps doing nothing that rolls over in a weekish so put it to some use :p [06:03] I have 100mbit downstream as well, and I ran 2 VMs with 6 threads each, but it doesn't help when all the downloads are constantly finished and waiting for the uploads :-( [06:03] I want something more symmetrical. FTTH would be nice [06:03] this account I'm uploading is like.. 2GB [06:04] yeah [06:04] i can get 100 down, 30 up if i paid the money with my isp.. but all packages besides my 20/2 (20/1 really) are with bandwidth caps... my 20 is not. so i stick with it ;) [06:04] symmetrical fibre would be nice [06:05] thats another option in my area, but i haven't heard that great of feedback about it here [06:05] Well, anything that is more than .3mbps up would be nice [06:05] there's no fiber here, sadly. at least no affordable fiber [06:05] Australia's national fibre network is kinda promising [06:06] plus the company that ffers teh fibre is one of the countries biggest pro-bandwidth capping supporters so i don't like them [06:06] But the data caps, not so much [06:06] I think the best plan at the moment is 100/40mbps with a 1tb cap [06:08] I can apparently only upload at 1MB/s to the box that;s hosting the cinch.fm stuff [06:08] in one connection [06:09] gah, my isp nuked their big package.. 100 down, 5 up, 250gb cap. bleh. [06:09] and "only" $165 canadian a month :p [06:10] other company offering 250 down, 30 up, with supposedly no cap but liek i said they are in favor of them so makes me leery.. $250/month [06:15] 1 box left finishing up [06:17] https://twitter.com/textfiles/status/237790179090374657 [06:19] that's me [06:19] 2GB+ upload [06:20] 13 mins or so remaining [06:20] Nintendud: slowass :p [06:21] 1MB/s :3 [06:21] from france or something [06:22] heh not sure where your server is even? :p [06:24] hey, it happens [06:25] Well, I'm billed from France, IP is Ireland [06:25] I think the main IP is french, [06:25] :3 [06:26] cheapest dedicated server I could get. works as a minecraft server. [06:26] ovh or one of the resellers? [06:27] Daevien: http://www.qdb.us/5273 [06:27] ovh [06:28] yeahhh 2443MB [06:28] chronomex: heh i've had that happen before. stuffed an old box in the closet and then stuff got piled onto it over time.. it was my router at the time, hd finally blew up one day and took me a bit to find it [06:28] yep [06:28] i finally gave up and chased it down via network cable :p [06:29] Bahahahaha [06:29] http://i.imgur.com/9zVkp.png [06:29] haha [06:38] looks like bsmith094 got the last one, hasn't moved in a bit [06:43] dammit [06:43] Guess it's going in -bs [06:45] heh [07:43] i've heard stories of a long-running server that wound up lost in some hidden room where the door had been plastered/drywalled over in a remodel [07:43] classic story, tat [07:43] *that [07:46] yeah, urban legend [07:52] here's one instance: http://www.theregister.co.uk/2001/04/12/missing_novell_server_discovered_after/ [08:19] uploaded: http://archive.org/details/filesharefreak.com-20120820-mirror [08:31] can my services be of assistance with cinch.fm? [08:47] kennethre: You're too late (as am I). They've eaten everything already. [08:47] hehe, a good problem to have :) [08:47] maybe there should be a mailing list? [08:48]