#archiveteam 2012-08-21,Tue

↑back Search

Time Nickname Message
00:01 🔗 brayden Rebooting things is a mistake we all make.
00:01 🔗 brayden I'll check out Warrior this afternoon.
00:02 🔗 brayden I was already rather impressed with the interface.. if it isn't better though they're fired.
00:03 🔗 GLaDOS DoubleJ: using that method, how would you prevent any further pipeline downloads?
00:06 🔗 GLaDOS Wait, nevermind.
00:20 🔗 DoubleJ GLaDOS: Glad you figured it out, 'cause I don't really know what you were asking :)
00:20 🔗 DoubleJ if you meant "touch STOP" that works like always.
00:52 🔗 SketchCow root@teamarchive-1:/1# rm -rf PICPLZ
00:52 🔗 SketchCow yay
00:52 🔗 SketchCow done
00:52 🔗 * BlueMax pops the champagne
01:05 🔗 S[h]O[r]T awesome
01:06 🔗 S[h]O[r]T thnx to alard and doublej instructions got my cinch running on non warrior
01:06 🔗 S[h]O[r]T might want to jsut remove the https from git clone and pip otherwise it complains about the cert
01:16 🔗 SketchCow 18G .
01:16 🔗 SketchCow root@teamarchive-1:/1/CINCH# du -sh .
01:22 🔗 godane i'm getting the last episode of making the most of micro
01:22 🔗 godane i'm also not banned from thebox.wz
01:22 🔗 SketchCow We're likely to kill the entire list of cinch sites we have so far in one night.
01:22 🔗 SketchCow That'd be nice.
01:27 🔗 swebb Just messing with the archiveteam warrior image now for the first time. It says that there are no active projects. Is that expected?
01:33 🔗 S[h]O[r]T i dont think it is swebb but there might be some update command you need to run to get it to download the latest projects
01:36 🔗 swebb git must not be updated because I'm following the menu.sh script and it's going a git pull successfully, so I'm guessing that the git repo for the archiveteam warrior hasn't been updated yet for this cinch pull.
01:45 🔗 S[h]O[r]T im not sure but im pretty sure people here are running the warrior but yeah maybe they had to do something manual
01:48 🔗 swebb I'm going to get it working manually using the archiveteam warrior iso as a base.
01:48 🔗 swebb We'll see if I can get it running before the whole job is done. :)
01:50 🔗 swebb Yea! I'm grabbing!
01:54 🔗 S[h]O[r]T :D
01:54 🔗 S[h]O[r]T nice
01:55 🔗 Coderjoe swebb: were you using the older warrior, or the newer python-based one?
01:56 🔗 swebb I just downloaded the warrior image today. The menu says that it was updated 12 days ago.
01:56 🔗 swebb I had to install a bunch of stuff to get the python stuff working. Most notably lua.
01:57 🔗 Coderjoe swebb: this one? Warrior v2 beta version: http://archive.org/download/archiveteam-warrior/archiveteam-warrior-v2-20120813.ova
01:57 🔗 swebb nope
01:57 🔗 swebb I'm using v1.
02:02 🔗 swebb Anyone know anything about the pipeline.py script? I wasn't able to start two of them on the same machine due to a "port in use" error. Is there a way to get two running at once?
02:05 🔗 Coderjoe pass --help
02:05 🔗 Coderjoe it has a parameter for how many concurrent downloads to run
02:05 🔗 Coderjoe also, check out http://localhost:8080/
02:05 🔗 swebb Ahh, thanks.
02:06 🔗 Coderjoe er
02:06 🔗 Coderjoe make that port 8001
02:06 🔗 swebb Yup. Just saw that. Thanks!
02:06 🔗 Coderjoe (looking at http://archive.org/details/archiveteam-warrior which also links to the most recent v2)
02:36 🔗 S[h]O[r]T swebb
02:36 🔗 S[h]O[r]T if you kill your single one and then run it again with the concurrent it will work
02:40 🔗 swebb Yea, I've used some of the older ones and I know about the STOP file, so I stopped and restarted with more workers.
02:50 🔗 underscor Received item 'cumiscuteonyrfce' from tracker
02:51 🔗 underscor SAVED
02:51 🔗 swebb Wow! underscore just started like 100 workers. :)
02:52 🔗 underscor 16 :D
02:52 🔗 Nintendud what's going on, guys?
02:52 🔗 underscor Nintendud: cinch.fm is closing
02:52 🔗 Nintendud idk what that is, but it sounds bad
02:53 🔗 Nintendud is this one of those things that I can use the archiveteam warrior on?
02:53 🔗 SketchCow yes
02:53 🔗 Nintendud cool cool. I'll see if I can finally get that set up.
02:55 🔗 underscor the warrior is so awesome
02:55 🔗 swebb I'm using the v1 version of the archiveteam warrior and I didn't see it on the list of active projects. Do I need v2 to see it in the menu?
02:55 🔗 Nintendud the only sad part is that it's going to be running as a VM in a VM
02:56 🔗 swebb Nintendud: it doesn't take hardly any CPU.
02:56 🔗 Nintendud it's not CPU
02:56 🔗 Nintendud disk IO is slow already :)
02:56 🔗 Nintendud it's going to have even slower disk io
02:56 🔗 Nintendud I'm too lazy to move it closer to the machine
02:57 🔗 swebb underscore doubled my efforts in just 5 minutes. :)
02:57 🔗 underscor Nintendud: Why double-vm'd?
02:58 🔗 underscor swebb: maybe? dunno if the old one still works. should though
02:58 🔗 Nintendud underscor: I don't feel like repartitioning the disk / memory / cpu among three VMs
02:59 🔗 Nintendud I share this server with a friend
02:59 🔗 underscor why not run it on the host?
02:59 🔗 Nintendud the host machine is VMWare ESXi. Doesn't have any hard drive space allocated to it
02:59 🔗 Nintendud well, a bare minimum :)
03:00 🔗 Nintendud no biggie though
03:00 🔗 Nintendud should be interesting to kick off this VM from the command line. usually I use a GUI.
03:00 🔗 swebb do the workers wait while an upload (sync) is running? It seems like no downloads are going on during the sync when using concurrentcy.
03:01 🔗 swebb concurrency
03:01 🔗 underscor Nintendud: oic
03:01 🔗 underscor swebb: each thread is synchronous
03:01 🔗 underscor so if all 6 threads are "uploading" (or however many you're running), then yeah, nothing will download
03:02 🔗 swebb ok, but each thread is independent, so it doesn't deadlock or something during an rsync.
03:09 🔗 Nintendud underscor / swebb: Is there a dashboard showing current projects / activity?
03:09 🔗 Nintendud wondering how you knew that underscor spun up workers
03:09 🔗 S[h]O[r]T http://tracker.archiveteam.org/cinch/ for cinch
03:09 🔗 underscor Nintendud: http://warriorhq.archiveteam.org/ to see what we're saving and who's participating
03:09 🔗 underscor and http://tracker.archiveteam.org/cinch/ to see people in action
03:09 🔗 S[h]O[r]T http://warriorhq.archiveteam.org/
03:09 🔗 Nintendud awesome. why is this not more prominent on archiveteam.org?
03:09 🔗 underscor s/who's participating/where people are participating from/
03:10 🔗 underscor it's still in beta-ish, iirc
03:10 🔗 Nintendud nice font
03:10 🔗 Nintendud is that droid?
03:10 🔗 underscor also the front page is superprotected afaik
03:11 🔗 underscor no idea
03:11 🔗 underscor I think it's a google web font, if that helps
03:11 🔗 Nintendud yeah, I think it's droid
03:11 🔗 Nintendud Firefox just fubared trying to inspect element :3
03:12 🔗 underscor so cool to watch warriorhq
03:12 🔗 underscor lots of eastcoasters
03:12 🔗 S[h]O[r]T id be on there in dallas if i was running warrior
03:12 🔗 S[h]O[r]T maybe ill load it up here at home
03:14 🔗 swebb For cinch, is there more discovery crawling needed to be done?
03:14 🔗 godane i'm uploading all my remainer gamesmaster season 5 to the root (/) of my items-uploads.archive.org account
03:15 🔗 godane this way i can just move them the item id folder when i need to
03:15 🔗 godane i just want to know will this cause problems?
03:15 🔗 Nintendud importing vm :3
03:16 🔗 S[h]O[r]T swebb i believe everything in the tracker is what cinch publishes on their sitemap which is *supposed* to be everything on the site
03:20 🔗 Nintendud damn; looks like I'll have to do this through the vbox gui
03:23 🔗 S[h]O[r]T dang the warrior is nice
03:29 🔗 S[h]O[r]T think i might have had the biggest one so far. 1.3gb?
03:30 🔗 S[h]O[r]T i think the concurrent does deadlock the other threads during an rsync
03:31 🔗 S[h]O[r]T at least not in warrior
03:38 🔗 BlueMax underscor, you big fat cheater. :|
03:39 🔗 underscor howso?
03:39 🔗 underscor :D
03:41 🔗 Nintendud woot. got the warrior working
03:42 🔗 Nintendud why is the maximum number of downloads 6? :3
03:44 🔗 BlueMax underscor, because you have 2600 items because your e-peen is huge
03:44 🔗 swebb I'm predicting that underscore will grab > 50% of the whole site.
03:50 🔗 BlueMax lawl
03:51 🔗 S[h]O[r]T trying to get stuff setup on a centos box but stupid lib errors all over the place
03:52 🔗 S[h]O[r]T Nintendud i wonder if 6 is a limit on the ajax interface that or prob since its only running with 400mb of ram
03:54 🔗 swebb tracker just died for me.
03:55 🔗 chronomex really? works fine for me.
03:55 🔗 Nintendud back
03:55 🔗 Nintendud yeah
03:55 🔗 Nintendud it doesn't seem to be uploading
03:55 🔗 Nintendud or accepting uploads*
03:55 🔗 swebb My uploads have stopped and the tracker page comes up, but no traffic on it.
03:56 🔗 Nintendud yeah
03:56 🔗 Nintendud same here
03:56 🔗 swebb something died somewhere.
03:56 🔗 chronomex odd.
03:56 🔗 Nintendud there's no 'single box' that we're uploading to, right?
03:56 🔗 chronomex the tracker is a single box, but I'm logged in to it right now
03:57 🔗 swebb Yea, we're rsyncing to a single box.
03:57 🔗 Nintendud ah
03:57 🔗 Nintendud that doesn't seem like it would scale
03:57 🔗 swebb My downloads are still flowing for existing jobs
03:57 🔗 swebb but uploads are hung.
03:57 🔗 chronomex ok
03:57 🔗 Nintendud my downloads are hung because the one upload thread is blocked
03:58 🔗 Nintendud if I could spawn more download threads, I could be making progress :>
03:58 🔗 Nintendud (with that said, the warrior is one fine piece of software)
03:58 🔗 swebb I got an rsync timeout.
03:59 🔗 swebb failed connect to fos.textfiles.com: Connection times out.
03:59 🔗 S[h]O[r]T yeah me 2. foss is different from the tracker
04:01 🔗 S[h]O[r]T i imagine SketchCow must already know :P
04:01 🔗 swebb I'm seeing life again on the tracker now.
04:01 🔗 swebb I'm guessing that it has recovered. I'm uploading again.
04:01 🔗 Nintendud yup
04:03 🔗 BlueMax underscor needs to stop downloading and let everyone catch up
04:04 🔗 BlueMax What sort of bloody pipe is he working with
04:04 🔗 swebb I don't know, but it's *way* bigger than what I've got.
04:05 🔗 Nintendud I've got a big one... 175 urls so far..
04:05 🔗 DFJustin he has access to archive.org boxen
04:05 🔗 swebb Ahh.
04:06 🔗 swebb archive.org has a crap-ton of hardware and bandwidth.
04:06 🔗 Nintendud 250...
04:08 🔗 swebb ok, we're past the 1/2-way mark.
04:08 🔗 Nintendud 50...
04:08 🔗 Nintendud 350* even
04:14 🔗 Nintendud this guy really loves using cinch... http://cinch.fm/travisdecker
04:14 🔗 S[h]O[r]T think he has some mental illness..not joking or being funny seems like he used it to send some messages
04:15 🔗 Nintendud I'm in public with no headphones
04:15 🔗 Nintendud so I can't check it out yet
04:15 🔗 Nintendud 625 urls so far
04:15 🔗 swebb I'm downloading scobleizer now! :)
04:15 🔗 swebb up to 1500 files so-far
04:15 🔗 Nintendud or wow
04:15 🔗 Nintendud oh wow*
04:16 🔗 swebb http://cinch.fm/scobleizer
04:19 🔗 Nintendud long tail is loooooooooong
04:20 🔗 Nintendud 925... I'm really curious how large his file is going to be
04:21 🔗 Nintendud too bad I'll have to head home soon. I'll miss that upload.
04:23 🔗 BlueMax what's the name Nintendud
04:23 🔗 BlueMax oh wait, travis decker
04:23 🔗 BlueMax I'll keep my eyes open for it
04:23 🔗 Nintendud cool cool
04:23 🔗 Nintendud 1050
04:24 🔗 Nintendud considering that he has something like 100 files over scobleizer, I assume it'll be 1500 or more files?
04:28 🔗 swebb scoble had 1700+ urls on cinch.
04:28 🔗 Nintendud Any way to estimate from the user summary?
04:29 🔗 Nintendud I dunno what it counts as 'pages'
04:29 🔗 Nintendud er, URLs
04:29 🔗 swebb 1.3GB for just his stuff. :)
04:29 🔗 Nintendud 1350 so far and counting for Mr. travis decker
04:29 🔗 Nintendud OH it's uploading
04:30 🔗 Nintendud looks like ~90 MB
04:30 🔗 Nintendud disappointing
04:30 🔗 Nintendud Yup. 84MB.
04:32 🔗 Nintendud when this is over, I'm going to want stats on stats
04:34 🔗 Nintendud welp, going home. see you guys later
04:35 🔗 chronomex cya
04:35 🔗 chronomex the annoying thing about munin is you have to wait a long time to see interesting graphs.
04:36 🔗 chronomex http://176.58.114.30/localdomain/localhost.localdomain/index.html#archiveteam
04:37 🔗 chronomex (spoiler: come back in a few hours.)
04:44 🔗 swebb cacti is the same way.
04:48 🔗 chronomex yeah, I would think so
04:53 🔗 underscor DFJustin: these are all on my home network and on various free OVH vks's
04:53 🔗 underscor http://www.ovh.com/us/vks/index.xml
04:54 🔗 underscor hahahaha
04:54 🔗 underscor \o/
04:54 🔗 underscor first place
04:55 🔗 swebb Dude, you trounced us all!
04:56 🔗 underscor http://maelstrom.foxfag.com/munin/foxfag.com/ponder.foxfag.com/fw_conntrack.html hahaha
04:59 🔗 swebb My stats are much wimpier: https://skitch.com/scumola/epa9g/cacti
05:04 🔗 chronomex that's nice
05:05 🔗 S[h]O[r]T underscor watch your back
05:05 🔗 S[h]O[r]T you pulled one over on me though :(
05:07 🔗 Nintendud oh. now that I'm home... yeah, S[h]O[r]T... travisdecker.. :(
05:08 🔗 Nintendud probably has a mental illness
05:08 🔗 S[h]O[r]T good on him tho using the service for whatever his needs were
05:09 🔗 Nintendud yeah
05:09 🔗 Nintendud he has 9 followers. I wonder if he was leaving those for a loved one?
05:10 🔗 * Daevien watches the counter spin
05:14 🔗 chronomex Lord_Nigh: I misread that as Lord_High
05:14 🔗 chronomex keep on truckin
05:14 🔗 underscor wheeeee
05:14 🔗 underscor we're almost done
05:14 🔗 Daevien how many boxes you got going underscor? :p
05:14 🔗 underscor only 8 :(
05:15 🔗 Daevien heh. i've got two, one is in london. and then my crappy home connection
05:15 🔗 underscor aha, you're one of the londoners
05:16 🔗 underscor http://warriorhq.archiveteam.org/
05:16 🔗 Daevien i'm not in london, but have a vps there yeah
05:16 🔗 underscor (the points are purposefully obfuscated somewhat)
05:16 🔗 underscor sorry, that's what I meant
05:16 🔗 Daevien the london one is a highly bored linode 512mb vps, figured i may as well do something with it :p
05:17 🔗 underscor cool :D
05:17 🔗 underscor yeah, most of mine are OVH vKSes that have mostly just been idling
05:17 🔗 Daevien er, i also appear to be the only one for sure in canada as well lol
05:18 🔗 underscor I mean, they do stuff like static file hosting and irc client
05:18 🔗 underscor but nothing taxing
05:18 🔗 underscor :o cool!
05:18 🔗 underscor :D
05:18 🔗 Daevien i'm trying to smack darkstar and get another position before it's done :p
05:19 🔗 Nintendud I think my box is in france
05:20 🔗 Daevien somewhere in the cinch datacenter someone is cursing us probably
05:20 🔗 Nintendud THe IP I'm behind is in Ireland though
05:20 🔗 underscor hahaha
05:20 🔗 underscor they're on ec2
05:20 🔗 underscor so they likely won't notice until they get their usage summary
05:21 🔗 underscor :D
05:21 🔗 Daevien heh yeah and then will go wtf happened?
05:21 🔗 chronomex lul
05:21 🔗 chronomex ec2++
05:21 🔗 Daevien where is jason's pic of the rubber glove that he's used for his avatar? :p
05:22 🔗 underscor I'm kinda amazed we'll have this done inside of 6 hours
05:22 🔗 underscor I dunno
05:22 🔗 underscor I was looking for it before
05:22 🔗 Daevien shodul see if we can get a number of users to change their avatar to that :)
05:22 🔗 underscor t SketchCow where's your rubberglove picture?
05:22 🔗 Daevien yeah i looked for it to send to a friend that was liek wtf are you doing? :p
05:22 🔗 underscor lol
05:23 🔗 Daevien since then he's been checkign the counter while he programs lol
05:24 🔗 Daevien has SketchCow noticed how fast it's going yet or will he be surprised when he gets back to a connection? :p
05:25 🔗 underscor he doesn't know
05:25 🔗 underscor but he probably won't be too surprised
05:25 🔗 Daevien heh it's still a ton of data in this time period
05:25 🔗 underscor archiveteam tends to do things like this :D
05:25 🔗 Daevien i usually miss the stuff when its going on or too busy
05:26 🔗 Nintendud Can we have a bat signal
05:26 🔗 Nintendud batsignal@archiveteam.org mailing list?
05:26 🔗 underscor we have a mailing list now!
05:26 🔗 Nintendud :O
05:26 🔗 underscor see topic
05:26 🔗 chronomex o_o
05:26 🔗 Nintendud oh right
05:26 🔗 Nintendud <-- blind
05:29 🔗 underscor https://groups.google.com/d/msg/archive-team/VloK-lYGA9Q/5GvMs9khL1gJ
05:29 🔗 underscor better late than never
05:33 🔗 Daevien nearing 500 left
05:34 🔗 chronomex lame, I had to reset the munin bytes & items graphs
05:38 🔗 Daevien hmm, crap. guess i'm about out of thigns to do and shodul go to bed here in a sec when this finishes :p
05:39 🔗 Daevien er, negative counter. interesting
05:39 🔗 chronomex ??
05:39 🔗 chronomex oh
05:40 🔗 chronomex yeah, that happens
05:40 🔗 DFJustin and it's a cinch
05:40 🔗 chronomex ha
05:40 🔗 DFJustin didn't even have time to get a full gigabyte :(
05:41 🔗 chronomex Daevien: the negative is because one counter is "number unclaimed" and the other is "number finished", but a finished event decrements the unclaimed counter in the ui
05:41 🔗 Daevien got one goign with 500mb and still at 70% lol
05:44 🔗 S[h]O[r]T underscor :(
05:44 🔗 S[h]O[r]T i thought i could make a come back and beat u
05:44 🔗 chronomex think again!
05:44 🔗 chronomex underscor generally wins
05:44 🔗 Daevien heh you'd have had to open some big pipes, he was cruising
05:44 🔗 underscor :D
05:44 🔗 underscor <3 you all though
05:44 🔗 S[h]O[r]T i did open big pipes
05:45 🔗 S[h]O[r]T i just didnt notice and get everything going until too late
05:45 🔗 S[h]O[r]T had 30 on one server, 100 on another, 200 on another, and 500 on another :P
05:45 🔗 BlueMax underscor is a big fat cheat :|
05:45 🔗 underscor S[h]O[r]T: the problem with that is that you get cpustarved
05:46 🔗 underscor :P
05:46 🔗 underscor because you have 450 runnable threads
05:46 🔗 S[h]O[r]T not really
05:46 🔗 S[h]O[r]T not running on a vps
05:46 🔗 S[h]O[r]T all corequads except 1 machine
05:46 🔗 underscor what limit do you hit then?
05:46 🔗 S[h]O[r]T nothing
05:46 🔗 underscor well, probably upstream to fos actually
05:46 🔗 S[h]O[r]T the limit of rsyncing to foss i guess since i dont get more than 1-3MB/s
05:47 🔗 S[h]O[r]T but its not because pipe is full
05:47 🔗 underscor sinec you can only have one running at a time
05:47 🔗 underscor since*
05:47 🔗 S[h]O[r]T yup
05:47 🔗 Daevien 689mb file landed heh. someone had a lot to talk about
05:47 🔗 S[h]O[r]T so if i hit a big file with this it stops
05:47 🔗 underscor you can only have one rsync across all workthreads at a time
05:47 🔗 underscor yep
05:47 🔗 underscor :(
05:47 🔗 S[h]O[r]T not a huge deal its done :p but yeah
05:47 🔗 S[h]O[r]T i beat you on picplz and tabloo i think though
05:48 🔗 chronomex haha
05:49 🔗 underscor probably
05:49 🔗 underscor but I didn't have a lot of spare boxen for them
05:49 🔗 underscor :)
05:49 🔗 underscor this was just good timing
05:53 🔗 Daevien underscor: you flipped over to 50gb :p
05:53 🔗 underscor :D
05:53 🔗 * BlueMax throws a ball at underscor
05:53 🔗 * underscor catches, returns
05:56 🔗 Dark_Star hmm... still >2400 to do (according to the tracker) but I keep getting "no item received"...
05:56 🔗 Dark_Star ah.
05:57 🔗 Dark_Star now the tracker shows 0 left.
05:57 🔗 Nintendud I have a HUGE upload going
05:57 🔗 Nintendud this is like over 1.5GB
05:57 🔗 chronomex yow
05:57 🔗 chronomex cool
05:57 🔗 Daevien Dark_Star: it seems to be peopel with a lot of processes sending the last bits in
05:57 🔗 Nintendud wwwpubliceyeimagingcom
05:57 🔗 Dark_Star after a refresh. I think something in the tracker's code is broken :)
05:57 🔗 chronomex it's not broken, it works fine
05:57 🔗 chronomex it has weirdness though
05:58 🔗 Daevien woot, i managed to just push above you in gb Dark_Star heh
05:58 🔗 Dark_Star well I had the window open since yesterday and it kept showing 2300 items left, still counting down. when I pressed F5, it now shows 0 items and no progress
05:58 🔗 Dark_Star @Davien :-P
06:00 🔗 Dark_Star well that was an easy project... what's next? :)
06:00 🔗 Daevien once it got going and i noticed where i was int he list, i wanted to stay in the top 10 hehe, stupid things i do late at night when bored :p
06:01 🔗 Dark_Star yeah, well, I couldn't do much against it, my upload speed is only about 5% of my download speed... while I could have grabbed more, it would have taken a few days to upload it all :)
06:02 🔗 Cameron_D Same here, I upload at about 30kb/s tops :(
06:02 🔗 Daevien yeah my own connection is usually about 1mbit max even though isp claims 2
06:02 🔗 Nintendud I'm using a 100mbit box, but... 6 threads :(
06:02 🔗 Cameron_D I used to get 1mbit, I now really miss that
06:02 🔗 Daevien i had a ton of bandwidth quota on two vps doing nothing that rolls over in a weekish so put it to some use :p
06:03 🔗 Dark_Star I have 100mbit downstream as well, and I ran 2 VMs with 6 threads each, but it doesn't help when all the downloads are constantly finished and waiting for the uploads :-(
06:03 🔗 Dark_Star I want something more symmetrical. FTTH would be nice
06:03 🔗 Nintendud this account I'm uploading is like.. 2GB
06:04 🔗 Nintendud yeah
06:04 🔗 Daevien i can get 100 down, 30 up if i paid the money with my isp.. but all packages besides my 20/2 (20/1 really) are with bandwidth caps... my 20 is not. so i stick with it ;)
06:04 🔗 Cameron_D symmetrical fibre would be nice
06:05 🔗 Daevien thats another option in my area, but i haven't heard that great of feedback about it here
06:05 🔗 Cameron_D Well, anything that is more than .3mbps up would be nice
06:05 🔗 Dark_Star there's no fiber here, sadly. at least no affordable fiber
06:05 🔗 Cameron_D Australia's national fibre network is kinda promising
06:06 🔗 Daevien plus the company that ffers teh fibre is one of the countries biggest pro-bandwidth capping supporters so i don't like them
06:06 🔗 Cameron_D But the data caps, not so much
06:06 🔗 Cameron_D I think the best plan at the moment is 100/40mbps with a 1tb cap
06:08 🔗 Nintendud I can apparently only upload at 1MB/s to the box that;s hosting the cinch.fm stuff
06:08 🔗 Nintendud in one connection
06:09 🔗 Daevien gah, my isp nuked their big package.. 100 down, 5 up, 250gb cap. bleh.
06:09 🔗 Daevien and "only" $165 canadian a month :p
06:10 🔗 Daevien other company offering 250 down, 30 up, with supposedly no cap but liek i said they are in favor of them so makes me leery.. $250/month
06:15 🔗 S[h]O[r]T 1 box left finishing up
06:17 🔗 Daevien https://twitter.com/textfiles/status/237790179090374657
06:19 🔗 Nintendud that's me
06:19 🔗 Nintendud 2GB+ upload
06:20 🔗 Nintendud 13 mins or so remaining
06:20 🔗 Daevien Nintendud: slowass :p
06:21 🔗 Nintendud 1MB/s :3
06:21 🔗 Nintendud from france or something
06:22 🔗 Daevien heh not sure where your server is even? :p
06:24 🔗 chronomex hey, it happens
06:25 🔗 Nintendud Well, I'm billed from France, IP is Ireland
06:25 🔗 Nintendud I think the main IP is french,
06:25 🔗 Nintendud :3
06:26 🔗 Nintendud cheapest dedicated server I could get. works as a minecraft server.
06:26 🔗 Daevien ovh or one of the resellers?
06:27 🔗 chronomex Daevien: http://www.qdb.us/5273
06:27 🔗 Nintendud ovh
06:28 🔗 Nintendud yeahhh 2443MB
06:28 🔗 Daevien chronomex: heh i've had that happen before. stuffed an old box in the closet and then stuff got piled onto it over time.. it was my router at the time, hd finally blew up one day and took me a bit to find it
06:28 🔗 chronomex yep
06:28 🔗 Daevien i finally gave up and chased it down via network cable :p
06:29 🔗 BlueMax Bahahahaha
06:29 🔗 BlueMax http://i.imgur.com/9zVkp.png
06:29 🔗 Daevien haha
06:38 🔗 Daevien looks like bsmith094 got the last one, hasn't moved in a bit
06:43 🔗 underscor dammit
06:43 🔗 underscor Guess it's going in -bs
06:45 🔗 chronomex heh
07:43 🔗 Coderjoe i've heard stories of a long-running server that wound up lost in some hidden room where the door had been plastered/drywalled over in a remodel
07:43 🔗 chronomex classic story, tat
07:43 🔗 chronomex *that
07:46 🔗 ersi yeah, urban legend
07:52 🔗 Coderjoe here's one instance: http://www.theregister.co.uk/2001/04/12/missing_novell_server_discovered_after/
08:19 🔗 godane uploaded: http://archive.org/details/filesharefreak.com-20120820-mirror
08:31 🔗 kennethre can my services be of assistance with cinch.fm?
08:47 🔗 alard kennethre: You're too late (as am I). They've eaten everything already.
08:47 🔗 kennethre hehe, a good problem to have :)
08:47 🔗 kennethre maybe there should be a mailing list?
08:48 🔗 alard There's a little bit of cleanup left, but that's all.
08:48 🔗 chronomex more work units to come tomorrow, apparently
08:48 🔗 alard Oh, where from?
08:48 🔗 chronomex idk, I think SketchCow said something
08:55 🔗 chronomex I could be wrong thoug
08:55 🔗 alard He may have meant the requeue-what's-not-returned phase.
08:56 🔗 chronomex yeah
08:56 🔗 alard And we could always do a second run, of course.
08:57 🔗 alard By the way, I've started made notes of some of the warrior feature requests. If there's anything missing, please add it here: https://github.com/ArchiveTeam/seesaw-kit/issues
09:23 🔗 GLaDOS alard: did you reset the users that I horribly failed at downloading?
09:24 🔗 alard GLaDOS: You had quite a few, yes. :) I've requeued everything, they're almost gone now.
09:24 🔗 GLaDOS Yeah, turns out I forgot to install liblua on that VPS.
09:40 🔗 SmileyG Awesome, I was right about using TC :)
10:47 🔗 chronomex alard: I made up a unified munin thing for several of my hosts today
10:47 🔗 chronomex new home of graphs for shilling is at http://zeppelin.xrtc.net/corp.xrtc.net/shilling.corp.xrtc.net/index.html
10:49 🔗 chronomex the old data is still cascading into the new graphs, so things are not completely settled yet
10:49 🔗 chronomex also goddamnit today I was not going to stay up to 4am
10:49 🔗 chronomex next week, perhaps I will get some kind of vpn system up.
11:47 🔗 ersi chronomex: #archiveteam-bs man
11:47 🔗 ersi default there
12:11 🔗 alard We have completed the first phase of Cinch.fm: the queue is empty, all outstanding tasks are finished.
12:33 🔗 SmileyG :O
12:33 🔗 SmileyG was that via warrior?
12:36 🔗 SketchCow Just wanted to say......
12:36 🔗 SketchCow So we have that warrior, that consistent warrior
12:37 🔗 SketchCow I'd prefer people not cobble together six pieces of rebar and a side of ham and some twist ties and join in the fun
12:37 🔗 SketchCow And then we have to redo it because the environment isn't consistent.
12:37 🔗 SketchCow That's no different than when we had the Windows do-gooders with their uppercase and their broken date strings
12:40 🔗 SketchCow Just saying.
12:40 🔗 SketchCow Now, as for phases, I mostly assumed we had the usual situation of there being low-hanging fruit, and then efforts to find the deeper, not-obvious-though-the previous-means users.
12:41 🔗 SketchCow But are you really convinced cinch had 27,000 users?
12:41 🔗 alard What we have now is the list of all users who had at least one recording in the sitemap.
12:41 🔗 SketchCow OK.
12:42 🔗 SketchCow So there's a chance more are out there.
12:43 🔗 alard Yes. There may be users without recordings (they're less interesting to archive, of course) and there may be users who are not in the sitemap.
12:54 🔗 SketchCow Could the whole site be 27,000 users and 178gb? That'd be sad.
12:55 🔗 SketchCow Maybe do some google searches?
12:56 🔗 alard 79792 usernames-20120821.txt
12:56 🔗 alard That's all the users in the sitemaps.
12:56 🔗 SketchCow Huh.
12:56 🔗 SketchCow The thing last night looked like it was looking at 27,000 things to download.
12:56 🔗 SketchCow Was it not?
12:56 🔗 SketchCow Was it 79,000?
12:57 🔗 alard Yes, no. I made the first list by listing the recording urls, then extracting the usernames from those urls. So the first batch included only users with content.
12:57 🔗 alard This second list is a list of every user.
12:57 🔗 alard I've checked a few, every user we haven't done has zero recordings.
12:58 🔗 alard I'll add the list, but I expect we won't find more recordings.
12:58 🔗 SketchCow But, but the Scobelizer said Cinch was as important as twitter
12:59 🔗 alard Did he? Okay, there we go: 50382 items in the queue.
13:02 🔗 balrog_ I wish there was a way to participate without having to run a VM :/
13:02 🔗 alard There are only 561412 recordings/messages on Cinch. (Assuming they started counting at 1.)
13:02 🔗 balrog_ (since while the VM is nice, it won't run on low power hardware that isn't x86)
13:09 🔗 altlabel guys, I'm trying to upload some MobileMe data with the upload-finished script, but it's returning me an error:
13:09 🔗 altlabel Uploading anitafranken
13:09 🔗 altlabel An rsync error. Scary!
13:09 🔗 altlabel @ERROR: Unknown module 'mobileme'
13:09 🔗 altlabel rsync error: error starting client-server protocol (code 5) at main.c(1524) [sender=3.0.7]
13:10 🔗 altlabel ...what can it be?
13:11 🔗 alard That's probably because the upload space was closed down.
13:15 🔗 altlabel ah, ops. ..will it be re-opened?
13:18 🔗 alard No, I don't think so. It's probably easiest if you can make a tar file (don't gzip) of the files you have, then ask SketchCow where to upload that tar file.
13:19 🔗 altlabel ok, thanks alard
13:19 🔗 alard We're more or less done mobileme, so a lot of the automated things have stopped. But every bit helps.
13:22 🔗 alard Okay, there is at least one unlisted recording on Cinch.
13:23 🔗 alard This http://cinch.fm/jeremy-millers/100013 is not in the sitemap, and not listed on the user's page (the user is in the sitemap).
13:24 🔗 alard The sitemap lists 182841 recordings. We know there are less than 561412.
13:29 🔗 alard Another one: http://cinch.fm/lingonaut/123sesame/240001 with an apparently unlisted album http://cinch.fm/lingonaut/123sesame
13:36 🔗 swebb I made an archiveteam-batsignal mailing list if people want to sign up: http://badcheese.com/cgi-bin/mailman/listinfo/archiveteam-batsignal
14:09 🔗 SketchCow OK, I'm out in the cube for a while this morning.
14:21 🔗 alard We should probably do a second run of Cinch.fm, number-based this time.
14:28 🔗 Schbirid Nintendud: you gonna up "The [INCOMPLETE] PC Gamer Demo/Sampler Collection [Demo/Sampler/2001]" to archive.org?
14:34 🔗 DFJustin it's all (or almost all) up already isn't it
14:39 🔗 SmileyG Wooo, it shows activity in the webpage now!
14:40 🔗 SmileyG zomg how many uploads is it gonna try and do?
14:40 🔗 SmileyG Holy cow this is cool :) I'd not seen it working on anything but tumblr which obviously took forever...
14:42 🔗 DFJustin all these new users seem to be 1mb
14:48 🔗 SmileyG Always 25 urls...
14:53 🔗 swebb How do I get to a console on the archiveteam warrior v2?
15:02 🔗 SmileyG Ooo I got a 50url one.
15:02 🔗 SmileyG swebb: why do you want a console?
15:03 🔗 swebb Just curious. I wanted to know the IP that the machine was running on. An ssh connection would also be good. I don't like not being able to see what's running on my machine.
15:03 🔗 swebb Also, it seems to be crashing every now and then.
15:04 🔗 alard swebb: Give focus to the virtual machine, then press Alt+F3.
15:04 🔗 swebb I tried that.
15:05 🔗 alard And?
15:05 🔗 swebb (rebooting now)
15:05 🔗 SmileyG lol
15:05 🔗 swebb I'll try again in a sec.
15:05 🔗 swebb The VM crashed, so I had to reboot it.
15:05 🔗 SmileyG swebb: Reason I asked btw was to check you'd read the instructions.
15:05 🔗 alard You should see a normal login screen, saying "the root password is archiveteam"
15:05 🔗 swebb Ahh, I got it now.
15:06 🔗 swebb and yes, I read the instructions
15:06 🔗 swebb thanks for belittling me though.
15:06 🔗 alard Alt+F2 should show output from the system.
15:06 🔗 alard Any error messages might pop up there.
15:06 🔗 alard The thing doesn't have an ssh server.
15:07 🔗 alard (But you can apt-get install that, of course.)
15:07 🔗 SmileyG swebb: Ok, I do lots of support, lots of users don't read instructions and say they've done things when they haven't. Don't hate for trying to make both mine and your life easier :(
15:08 🔗 swebb Hung again. No error on tty #2.
15:08 🔗 alard What happens on tty #1?
15:09 🔗 alard Does it get to the splash screen image?
15:09 🔗 swebb Yea, it got there, and it ran 51 fetches (I'm running 4 concurrent processes).
15:10 🔗 alard But when does it hang, exactly?
15:10 🔗 swebb It started again.
15:10 🔗 swebb The web browser says "lost connection to the warrior" in a red bar along the top of the web page.
15:11 🔗 swebb The console still works but the web guy has that red bar after crawling for a short while.
15:11 🔗 swebb s/guy/gui/g
15:11 🔗 alard That happens as soon as you restart the warrior. It may not reconnect immediately when it comes back up. The guy checks every now and then, but you should refresh the page if you want it to go faster.
15:12 🔗 alard Maybe this is something for #warrior, if there's more.
15:13 🔗 swebb Screenshot: https://skitch.com/scumola/ep4in/skitch
15:13 🔗 swebb I'm seeing errors now on tty #2
15:14 🔗 swebb File "/usr/local/lib/python2.6/dist-packages/tornado/websocket.py", line 258, in wrapper ...
15:14 🔗 swebb HTTPError(401, "Invalid Session")
15:34 🔗 DoubleJ SketchCow: re 8:37, I diasgree. At some point we'll have another large project that will require those of us with Linux boxes but who can't/won't run the warrior. I'd rather shake out any problems with things like Cinch, where we can re-download everything in udner a day, than lose time on a project that has none.
15:40 🔗 SketchCow RE 8:37 is pretty oof
15:41 🔗 SketchCow I'd rather just recruit more warriors and let the linux people be doing other things than just straight up/download
15:42 🔗 swebb I made an archiveteam-batsignal mailing list if people want to sign up: http://badcheese.com/cgi-bin/mailman/listinfo/archiveteam-batsignal
15:43 🔗 SketchCow What is its purpose
15:43 🔗 SketchCow That the IRC channel doesn't solve
15:43 🔗 swebb Send an email to the mailing list as a "call to action" to notify people if/when they're not on the irc channel. People were talking about it, so I made one.
15:43 🔗 SketchCow OK.
15:44 🔗 swebb aka, "new project - start your downloaders"
15:44 🔗 SketchCow Let's go further on the linux vs. warrior
15:44 🔗 SketchCow It's the windows argument ALL over again
15:44 🔗 SketchCow it's MUCH better to have a consistent environment. We can have one. We can just get more people involved on that side.
15:44 🔗 SketchCow The side itself is the most unthinking part of archiveteam - running a program that uploads.
15:45 🔗 SketchCow It's better it be produced consistently, so down the line we don't discover someone forgot to install lksdhsiodufhsdaioufhasdfhdf.h and it produces stuff that's "just a little different"
15:46 🔗 * SketchCow is firing up the warrior before heading out to do more infocube
15:46 🔗 SketchCow Wh aren't any of you fuckers near Fishkill NY
15:46 🔗 SketchCow Totally need help
15:46 🔗 * SketchCow waves fist
15:47 🔗 SketchCow I'm so old, and here I am lifting boxes
15:47 🔗 * SketchCow calls amnesy international
15:47 🔗 SketchCow amnesy
15:49 🔗 SketchCow Loving the warrior
15:49 🔗 SketchCow By the way
15:49 🔗 SketchCow It's tv/movie quality
15:56 🔗 ersi The only thing that's "bad" with the warrior is that it's hard to scale. It's really great for distributing ONE machine to everyone - but I can't spin one up everywhere. Scripts I can fire in a screen anywhere
16:02 🔗 swebb ersi: agreed
16:02 🔗 swebb screen is the shit.
16:04 🔗 SmileyG can't you grab the collection of scripts which make up warrior and use them?
16:04 🔗 SmileyG but don't tell anyone?
16:04 🔗 SmileyG Oh wait, that kilsl the clean system idea.
16:04 🔗 * SmileyG goes off to think
16:04 🔗 swebb having the scripts in github is a good start and seems to work for most people who don't run the warrior image.
16:05 🔗 swebb Most people run ubuntu anyway.
16:05 🔗 SmileyG yeah but it negates the whole point.
16:05 🔗 swebb I understand the idea of the black box. It's a great idea.
16:05 🔗 SmileyG Not just being a blackbox though, its a _Stable_ blackbox.
16:06 🔗 swebb but for people with available resources, it's easy to pull a script onto 10 machines here and there and just start them all up. With the warrior image, I need a VM to run it on somewhere.
16:07 🔗 SmileyG yeah
16:07 🔗 SmileyG make a binary... that way theres no "zomg wrong lib" issues...
16:07 🔗 swebb Yea, a single binary would be a good idea.
16:07 🔗 swebb statically linked
16:08 🔗 SmileyG indeed.
16:08 🔗 swebb The whole linux/warc/rsync combination should be a standard-enough platform though.
16:10 🔗 SmileyG well warc is only in some versions of wget...
16:10 🔗 swebb Yea, I know.
16:11 🔗 swebb Don't get me wrong, I think that the tracker and warrior stuff is really well done and I admire the work, but personally, I can only run it on a few machines with VirtualBox on them. I can't run it on any of my servers.
16:15 🔗 balrog_ SmileyG: warc is in all latest versions of wget
16:15 🔗 balrog_ it's a standard part of wget 1.14, which was released
16:17 🔗 Deewiant Two weeks ago, so most distributions probably don't have it yet(?)
16:17 🔗 balrog_ most do, though some in unstable repos
16:18 🔗 Deewiant Well yeah, I guess I should have said "most users"
16:18 🔗 balrog_ ubuntu doesn't though, but of course there's probably a PPA
16:30 🔗 alard So you'd need a statically linked Wget+Lua (in this case), plus the Python stuff in an easy to handle form.
16:32 🔗 swebb Perhaps a package (dpkg or rpm) that will handle all of the dependancies to ensure a compatible environment?
16:33 🔗 alard Yes. (Although that still gives you a lot of different distributions and versions to deal with.)
17:37 🔗 nitro2k01 http://aws.amazon.com/glacier/
17:39 🔗 nitro2k01 Long term storage for $0.01/GB/mo. Access time, hours. Amazon are probably trying tapes handled by robots.
17:40 🔗 Ymgve eh, $10 per terabyte per month is still a bit
17:45 🔗 swebb and there's a penalty if you delete the data in less than 3 months.
17:46 🔗 swebb "Amazon is encouraging customers to use Glacier for archival, rather than short-term, storage. If information is deleted within three months of being uploaded, there is an "early deletion fee", it said."
17:47 🔗 swebb upload is free. storage is $0.01/GB/mo. download is $0.12/GB
17:50 🔗 Ymgve literally holding your data prisoner
18:01 🔗 yipdw Ymgve: only if they force you to delete your other copies :P
18:02 🔗 * yipdw will probably be using Glacier as off-site backup for a photo archive
18:02 🔗 yipdw it's not easy to efficiently store 200 gigabytes of images
18:03 🔗 yipdw well
18:03 🔗 yipdw it is easy. it's not *cheap* :P
18:04 🔗 yipdw and $2/month is an order of magnitude less than what I'm paying now for an off-site backup for that
18:05 🔗 SketchCow Hug hug
18:09 🔗 Ymgve yipdw: yeah, then it's not too bad
18:10 🔗 yipdw I guess the other solution is "prune your damn archive"
18:10 🔗 yipdw which works, but is really a short stopgap solution
18:10 🔗 yipdw frickin 5D mkII generates huge outputs no matter what you do, unless you shoot in sRAW/small JPEG or something
18:11 🔗 swebb Flicker pro account is $30/yr which is just a little more than $2/mo
18:12 🔗 SmileyG gah, now your making me tempted to write a "warrior-script" ebuild for gentoo ¬_¬
18:12 🔗 yipdw Flickr's a possibility -- I just have more trust in Amazon than Yahoo, which is important whilst I only have one off-site backup
19:19 🔗 swebb SmileyG: re: Gentoo, geek points +1! I've not built a Gentoo box for over 5 years. Is Gentoo still maintained and releasing new stuff still?
19:19 🔗 ersi Fuck, how hard is it to remember #archiveteam-bs?
19:19 🔗 * ersi gives nitro2k01 the cold stare
19:33 🔗 SmileyG swebb: oh yes, and if you want to discusss this futher lets go to #archiveteam-bs :D
19:44 🔗 underscor it would be cool if there was an openphoto connector for glacier
20:00 🔗 chronomex woop woop woop off-topic siren
20:03 🔗 underscor D:
20:41 🔗 swebb I'm getting "Tracker returned status code 455" now from my client.
20:43 🔗 alard Yes. I've just started the second round of Cinch.fm. That requires a new version of the scripts. If you're running a warrior, it should update itself (it probably has done that by now).
20:44 🔗 alard If you're not running a warrior, you should do the update yourself:
20:44 🔗 alard 1. install the latest version of the seesaw-kit with pip
20:44 🔗 alard 2. git pull on the cinch-grab repository
20:46 🔗 alard 455 is code for "you're too old, please upgrade". (The new version of the seesaw-kit has a better error message.)
20:46 🔗 chronomex is an auto-updating seesaw-kit in the roadmap?
20:48 🔗 alard No, but feel free to look into it. It's auto-updating on the warrior, but not standalone. It probably requires extensive sudo permissions, for instance, and it depends on how you've installed the thing.
20:48 🔗 chronomex ah, an auto-updating warrior is all I was asking about
20:48 🔗 alard (Of the two upgrade steps #2, the project-specific upgrade, is probably easier and more frequent than #1.)
20:49 🔗 alard Your warrior is fine. It might take up to an hour for the warrior to realize it's out of date, but then it will reboot.
20:56 🔗 case_ anyone have experience using BagIt? http://www.dcc.ac.uk/resources/external/bagit-library
21:30 🔗 godane http://arstechnica.com/gaming/2012/08/source-nintendo-power-magazine-to-cease-publication/
21:30 🔗 godane we maybe able to get all of nintendo power pdfs up in a year
21:42 🔗 Nemo_bis does the deriver digest even WebEx arf format?
22:04 🔗 Nemo_bis no it doesn't
22:05 🔗 chronomex aw fuck I just got a google result pointing to splinder
22:05 🔗 chronomex why do these fucks keep breaking the internet
22:38 🔗 underscor alard: http://p.defau.lt/?nkK_A5hDzvu_VGi7KrfVLg
22:39 🔗 underscor oh, seesaw-kit needs updating too
22:39 🔗 underscor damn
22:39 🔗 underscor nevermind
22:42 🔗 * Daevien ducks for cover, underscor has started up again :p
22:42 🔗 underscor :D
22:49 🔗 Daevien gah, my mind has converted fully to linux.. command prompt on windows and it took me 2 tries to realize why ls wasn't working
22:53 🔗 balrog_ lol
22:53 🔗 balrog_ happens to me all the time
22:54 🔗 Daevien doesn't help that with my laptop having a ssd, i can flip completely from one to another within seconds and forget which i'm in until i pull up the taskbar :p
22:56 🔗 underscor haha
22:56 🔗 underscor solution is to not use windows at all
22:56 🔗 underscor :D
22:57 🔗 Daevien i mostly have windows for a game or two but sometimes it's useful
22:58 🔗 chronomex woop woop woop off-topic siren
22:59 🔗 Daevien :p

irclogger-viewer