#archiveteam 2011-11-12,Sat

↑back Search

Time Nickname Message
00:04 🔗 underscor This wget has been hanging for +12 hours now
00:04 🔗 underscor and the last few lines in the log are
00:04 🔗 underscor 2011-11-09 08:24:47 URL:http://web.me.com/bobfarmer/20110726Web%20Cards/ps01/ps01_446.htm [2326/2326] -> "data/b/bo/bob/bobfarmer/web.me.com/files/web.me.com/bobfarmer/20110726Web Cards/ps01/ps01_446.htm" [1]
00:04 🔗 underscor 2011-11-09 12:54:27 ERROR 404: Not Found.
00:04 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps20/:
00:04 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps20/feed.xml:
00:04 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps21/:
00:04 🔗 underscor 2011-11-11 12:10:48 ERROR 404: Not Found.
00:04 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps21/feed.xml:
00:04 🔗 underscor 2011-11-11 12:10:49 ERROR 404: Not Found.
00:04 🔗 underscor It is now 00:04 server time
00:04 🔗 underscor so it's been nearly 12 hours since the last update
00:04 🔗 underscor alard: Do you think it's dead or something?
00:13 🔗 alard underscor: Maybe, maybe it's trying to download a very big file. Have you looked at the url list?
00:16 🔗 underscor alard: There'd be an entry for every file, right?
00:16 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps21
00:16 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps21/feed.xml
00:16 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps21/ps21_001.htm
00:16 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps21/ps21_002.htm
00:16 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps21/ps21_003.htm
00:16 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps21/ps21_004.htm
00:16 🔗 underscor http://web.me.com/bobfarmer/20110726Web%20Cards/ps21/ps21_005.htm
00:17 🔗 underscor None of those are particularly large
00:17 🔗 alard No, so it's probably hanging.
00:17 🔗 underscor k
00:17 🔗 underscor ctrl-c, dld-single?
00:17 🔗 alard Yes.
00:18 🔗 underscor k :)
00:26 🔗 DFJustin I can't read chinese no
01:45 🔗 DrFaustus hola
01:46 🔗 DrFaustus there is an old friend of mine that has been running a mailing list for a long time. I'm not sure how much longer he's going to do it.
01:47 🔗 DrFaustus He just got the server back up, but it's pretty old
01:47 🔗 DrFaustus http://www.team.net/archive/
01:47 🔗 DrFaustus he may have even older email archives from before the "new" mailer
01:48 🔗 DrFaustus mjb@autox.team.net
01:48 🔗 DrFaustus can anyone help grab that stuff before mark peters out?
01:53 🔗 db48x DrFaustus: very probably
01:53 🔗 db48x I believe someone here already has a mailman archiver
01:55 🔗 db48x ah, it's even set up to let us download mbox files
01:55 🔗 DrFaustus indeed
01:56 🔗 DrFaustus you'd have to contact mark to see if he still has even older archive files around
01:56 🔗 DrFaustus he's an old fart of a unix admin from university of utah
01:56 🔗 DrFaustus so it'd likely be in an easy formar
01:57 🔗 db48x well, mbox is never as easy as it should be
01:57 🔗 db48x but it's easy enough :)
01:58 🔗 DrFaustus fair enough
01:59 🔗 DrFaustus those lists hold about fifteen years of technical discussions on just about every classic british sports car made
02:31 🔗 DrFaustus thanks guys
03:47 🔗 yipdw whoa, when did the splinder grab go from 990,000 users to 1.3 million?
03:47 🔗 yipdw oh, splinder.us
03:47 🔗 yipdw I see
04:05 🔗 db48x where are the splinder scripts? github?
04:41 🔗 yipdw db48x: https://github.com/ArchiveTeam/splinder-grab
05:52 🔗 underscor Wow, the splinder todo went way up!
05:53 🔗 underscor Luckily, splinder looks to only be about a terabyte and a half
06:53 🔗 yipdw heh
06:53 🔗 yipdw http://memac.heroku.com/ is reporting 666 MB/user
06:53 🔗 yipdw I always knew Mac users were satanic
06:54 🔗 underscor hahaha
09:55 🔗 chronomex damn. wget-warc is eating all my ram.
10:10 🔗 chronomex shit, that fucker ran for a day and a half before it got OOM killed
11:55 🔗 db48x alard: I notice that when I visit memac.heroku.com, it's getting log messages about splinder :)
11:56 🔗 alard db48x: It's the same source. (But it's not showing them, I hope?)
12:11 🔗 Wyatt|Wor So I just realised, I have a sizable ball of google groups to upload still.
12:11 🔗 Wyatt|Wor Also a few chunks of berlios
12:25 🔗 Wyatt|Wor I'm off to a Doc appointment now, but feel free to /msg me a place to rsync to; I'll get to it tonight. Sorry to be so late about this.
13:12 🔗 db48x alard: yea, it's not showing them
13:13 🔗 db48x alard: you should set up seperate streams for them
13:13 🔗 alard Why? It works?
13:15 🔗 db48x it's just extra work every time a message comes in
13:16 🔗 db48x anyway, the real reason I'm looking is that the splinder tracker kills my browser when it's open
13:29 🔗 db48x hmm. updating the chart is super expensive
13:31 🔗 alard Yes, I'm looking at a way to have fewer points in the graph, that should help somewhat.
13:35 🔗 db48x it could just update less frequently
14:21 🔗 alard db48x: Should be faster now.
14:29 🔗 db48x that it is
15:24 🔗 db48x alard: are you also alart? :)
15:46 🔗 alard Yes, typo. :)
15:46 🔗 closure Wyatt|Wor: you need to ask SketchCow for an rsync on batcave
19:03 🔗 Schbirid i think i will start logging http://store.steampowered.com/stats/
19:03 🔗 Schbirid might be interesting to make a 365 day graph
19:10 🔗 dnova Recipient: Bovine Ignition Systems
19:10 🔗 dnova Amount: $100.00
19:10 🔗 dnova lol
19:30 🔗 underscor haha
20:19 🔗 yipdw huh, this is kind of weird
20:19 🔗 yipdw https://gist.github.com/7723903aa5ff2c0fbeb3
20:20 🔗 Paradoks I got an error on the malacarne profile, when it attempted to download the blog from porno.splinder.com.
20:21 🔗 yipdw oh, the docum profile has been made unavailable
20:21 🔗 yipdw ok
20:32 🔗 underscor Paradoks: haha, nice name
21:04 🔗 yipdw so it looks like we're download about 21.8k Splinder users/day
21:04 🔗 yipdw somehow, that doesn't seem fast enough
21:05 🔗 yipdw to the Amazon
21:05 🔗 yipdw 'course, that's moot if we're maxing their pipe already :P
21:06 🔗 underscor when do we have til again?
21:11 🔗 db48x it was 14 days, right?
21:11 🔗 db48x that's only 300k
21:12 🔗 underscor uh oh
21:12 🔗 Schbirid how much data/bandwidth is it roughly if i joined as downloader?
21:12 🔗 db48x Schbirid: tiny
21:12 🔗 underscor Yeah
21:12 🔗 underscor Like 0.8 mb/user
21:13 🔗 db48x http://splinder.heroku.com/
21:13 🔗 underscor db48x: Download moar!
21:13 🔗 db48x I'm limited by iops
21:13 🔗 db48x I guess I can leave off sorting poetry for a while
21:13 🔗 underscor aww :(
21:13 🔗 Schbirid • :D
21:14 🔗 underscor I'm pulling 4mbps right now
21:15 🔗 * db48x cancels three other tasks
21:15 🔗 Schbirid ok, how do i join in?
21:16 🔗 db48x pull from the git repository
21:16 🔗 underscor I'm running 96 clients right now
21:16 🔗 underscor <3
21:17 🔗 db48x Schbirid: https://github.com/ArchiveTeam/splinder-grab
21:17 🔗 underscor 18-25% iowait, so that's probably just about perfectly balanced
21:18 🔗 underscor RX bytes:10762759777147 (10.7 TB) TX bytes:12421364281615 (12.4 TB)
21:20 🔗 Schbirid db48x: done! how interruptable is it?
21:20 🔗 Schbirid i switch off my pc at night
21:20 🔗 db48x touch STOP and it'll stop cleanly
21:21 🔗 Schbirid nice
21:21 🔗 underscor Schbirid: BLASPHEMY
21:21 🔗 underscor NO SHUTTING OFF IN HERE
21:21 🔗 underscor 21:21:52 up 18 days, 14:46, 1 user,
21:21 🔗 underscor 16:22:03 up 7 days, 23:35, 1 user,
21:21 🔗 underscor 13:22:22 up 26 days, 14:59, 2 users
21:22 🔗 Schbirid :)
21:22 🔗 underscor 15:24:01 up 18 days, 2:17, 2 users,
21:22 🔗 yipdw 21:22:55 up 41 days, 14:43, 1 user, load average: 0.32, 0.13, 0.25
21:22 🔗 yipdw me@avatar:~$ uptime
21:22 🔗 underscor yipdw: :(
21:22 🔗 Schbirid ouch, seems to want python2 or something?
21:23 🔗 Schbirid db48x: http://pastebin.com/vV9fu51i
21:23 🔗 yipdw of course, what that really means is "41 days since last kernel upgrade"
21:23 🔗 underscor haha
21:23 🔗 yipdw since who the hell uses ksplice etc
21:23 🔗 underscor ofc
21:23 🔗 Schbirid my python is 3.2.2 by default, 2 would be python2
21:23 🔗 underscor Yeah, it uses 2.[5-7].x, iirc
21:24 🔗 underscor uses/needs
21:24 🔗 Schbirid do i just add
21:25 🔗 Schbirid #!/usr/bin/python2
21:25 🔗 Schbirid to the soup py file?
21:25 🔗 underscor Oh, I see, you have it installed aready
21:25 🔗 underscor Yeah, change it to wherever python 2.x lives
21:25 🔗 yipdw Schbirid: substitute python2 for python at dld-profile.sh:88
21:26 🔗 underscor 0 1:24PM:abuie@teamarchive-0:/2/TBAG/mobileme-grab 3944 π du -sh data
21:26 🔗 underscor 1.3T data
21:26 🔗 Schbirid totally missed that, cheers
21:26 🔗 yipdw ha
21:26 🔗 Schbirid underscor: good for you, my python is bigger though
21:26 🔗 underscor lol
21:26 🔗 underscor Just a *little* mobileme data
21:27 🔗 Schbirid mobileme is a name i never heard anyone call IT
21:27 🔗 Schbirid okok, i will stop ;D
21:27 🔗 underscor :P
21:27 🔗 yipdw first time I've ever used the EU West EC2 region
21:27 🔗 underscor yipdw: Work well?
21:27 🔗 yipdw dunno yet
21:27 🔗 yipdw we'll see
21:28 🔗 yipdw I wonder if a micro will be good enough
21:28 🔗 yipdw yeah, probably
21:29 🔗 Schbirid working well now, thanks
21:29 🔗 underscor 102 hour tar?
21:29 🔗 underscor :(:(:(:(:(:(:(:(:(:(:(:(:(:(:(
21:32 🔗 yipdw underscor: what do you use to manage downloader instances? GNU parallel or something?
21:32 🔗 yipdw I figure if I'm going to get raped by Amazon EC2, I might as well deserve it
21:33 🔗 underscor yipdw: tmux panes
21:33 🔗 underscor Lemme take a screenshot
21:33 🔗 yipdw oh
21:34 🔗 yipdw hmm
21:34 🔗 yipdw https://gist.github.com/3018d5389a62de4d2caa
21:34 🔗 yipdw could be worse, I guess
21:34 🔗 underscor http://i.imgur.com/MpNcW.png
21:35 🔗 yipdw yikes
21:36 🔗 underscor :D
21:36 🔗 underscor I like that I can still keep an eye on them though
21:36 🔗 yipdw I guess
21:36 🔗 yipdw I'm not likely to invest that much effort though :P
21:36 🔗 yipdw hmm
21:36 🔗 yipdw I guess I could have monit monitor them for me
21:36 🔗 underscor haha
21:37 🔗 yipdw and periodically run dld-single on failed ones
21:37 🔗 yipdw ABSTRACTION SOLVES LAZINESS
21:38 🔗 underscor Is monit good for this?
21:38 🔗 yipdw it's overkill
21:38 🔗 yipdw IMO
21:38 🔗 underscor I've never used it, but heard of it before
21:39 🔗 yipdw I just want something to automatically restart clients that stop due to errors
21:39 🔗 underscor oh
21:39 🔗 yipdw but a loop in bash does that just as wel
21:39 🔗 yipdw l
21:39 🔗 underscor :P
21:39 🔗 underscor while true; ./dld-client yipdw ;done
21:39 🔗 underscor Yeah
21:39 🔗 underscor hahah
21:39 🔗 yipdw yeah, more or less
21:39 🔗 yipdw it'll just screw up badly when we're done
21:39 🔗 yipdw or, more precisely, when the tracker has nothing left
21:42 🔗 underscor yeah
21:42 🔗 underscor but hopefully you'll be around when we get closeish
21:43 🔗 underscor :D
21:43 🔗 Schbirid that heroku page totally needs a users/timeunit per participant :)
21:43 🔗 yipdw yeah
21:43 🔗 * underscor winds
21:43 🔗 underscor wins*
21:43 🔗 underscor hahah
21:45 🔗 yipdw "Quadruple Extra Large Hi-Memory On-Demand Instance"
21:45 🔗 yipdw jeez
21:45 🔗 underscor ha
21:45 🔗 yipdw just call it "Super Size Bigass Instance With Extra Fries"
21:45 🔗 yipdw "Now With More Molecules"
21:45 🔗 underscor hahahaha, the akamai edge servers I'm downloading from are 2 hops away
21:45 🔗 underscor It's basically "peering point"->
21:46 🔗 yipdw Splinder uses Akamai?
21:46 🔗 underscor "akamai's router"
21:46 🔗 underscor No, mobile-me does
21:46 🔗 yipdw oh
21:46 🔗 yipdw I was like "damn, I've been hitting the wrong thing"
21:46 🔗 underscor haha
21:47 🔗 underscor http://tracker.archive.org/tracker.png
21:47 🔗 underscor You can see where I stopped splinder overnight, haha
21:47 🔗 underscor And then just started up mobile me
21:47 🔗 yipdw good job sir
21:47 🔗 yipdw wow, 40 MB of Splinder data for henrymusica
21:47 🔗 underscor http://tracker.archive.org/batcave.png
21:47 🔗 yipdw that's the biggest I've seen yet
21:47 🔗 underscor Mobileme's data goes straight to batcave
21:48 🔗 underscor Nice little peak where it's started
21:48 🔗 underscor Schbirid: I DON'T SEE YOU ON THE TRACKER YET.........;..........
21:49 🔗 Schbirid i see me and i am just passing db48x
21:49 🔗 underscor Oh, are you spirit?
21:49 🔗 Schbirid haha
21:49 🔗 Schbirid yes
21:49 🔗 underscor oh
21:49 🔗 underscor grr :P
21:49 🔗 Schbirid =(
21:49 🔗 underscor Use your irc nick!
21:49 🔗 underscor hehe
21:50 🔗 Schbirid only germans get it :\
21:50 🔗 Schbirid no idea why
21:50 🔗 underscor Google translate says nothing
21:51 🔗 yipdw that is a more profound statement than you know
21:51 🔗 underscor :D
21:51 🔗 Schbirid its just spirit pronounshed like shad
21:51 🔗 underscor I like how the "Users downloaded" line on splinder pretty much follows my line at the beginning
21:52 🔗 underscor A global "users/hour" counter would be nice
21:52 🔗 * underscor loves making all these feature requests for alard_
21:54 🔗 underscor !!!!!!
21:54 🔗 underscor I officially have the highest bandwidth-used port at IA
21:56 🔗 db48x heh
21:57 🔗 Schbirid for splinder, how many parallel instances should i run? bandwidth is tiny but maybe saturation is elsewhere?
21:58 🔗 underscor Saturation is disk io
21:59 🔗 underscor Run ~10 and see if your iowait shoots up
22:15 🔗 yipdw hmm
22:15 🔗 yipdw [ec2-user@ip-10-227-178-174 it]$ sudo iostat
22:15 🔗 yipdw Linux 2.6.35.14-97.44.amzn1.x86_64 (ip-10-227-178-174) 11/12/2011 _x86_64_ (1 CPU)
22:15 🔗 yipdw avg-cpu: %user %nice %system %iowait %steal %idle
22:15 🔗 yipdw 2.63 0.00 2.91 2.19 19.40 72.87
22:16 🔗 yipdw that's with 6 dld-clients on a t1.micro
22:16 🔗 yipdw I guess I can double that
22:19 🔗 alard_ underscor: http://splinder.heroku.com/
22:20 🔗 underscor alard_: I love you
22:20 🔗 underscor Remind me to buy you a beer when I turn 21
22:20 🔗 chronomex underscor: I think you can do alcohol mail-order, the only person who needs to be over 21 is the recipient iiuc
22:21 🔗 underscor haha
22:21 🔗 yipdw wow, we're only pulling 500 kB/s?
22:21 🔗 underscor I don't know how well international alcohol mail-order would go over
22:21 🔗 chronomex yipdw: it uses a linear interpolation of reported data
22:22 🔗 yipdw ahh, ok
22:22 🔗 chronomex f.e. I've been downloading this one user for 4 days
22:22 🔗 yipdw so I guess down clients will
22:22 🔗 yipdw yeah, and that
22:22 🔗 yipdw jeez
22:22 🔗 alard yipdw: And I'm not even sure it's completely correct, so it may be helpful to check the numbers.
22:22 🔗 yipdw how big is the WARC for that user?
22:22 🔗 chronomex yipdw: huge. wget-warc died the first time around thanks to my OOM killer.
22:23 🔗 yipdw damn
22:23 🔗 chronomex 18G and growing
22:23 🔗 yipdw one of splinder's top users
22:23 🔗 chronomex web.me.com is "at least 23879 files"
22:23 🔗 yipdw oh
22:23 🔗 yipdw mobileme
22:23 🔗 chronomex yeah, mobileme
22:23 🔗 yipdw I was looking at the splinder dash
22:24 🔗 * chronomex not doing splinder
22:27 🔗 yipdw bwahaha
22:27 🔗 yipdw underscor's monopoly on the splinder board is broken
22:28 🔗 yipdw well, was
22:28 🔗 underscor What happened?
22:28 🔗 yipdw a bunch of other download clients finished
22:28 🔗 underscor oic
22:30 🔗 chronomex heh
23:06 🔗 yipdw heh, oops
23:06 🔗 yipdw just realized this about the micro EC2 instance I was running for splinder:
23:06 🔗 yipdw Mem: 611252k total, 537012k used, 74240k free, 27696k buffers
23:06 🔗 yipdw Swap: 0k total, 0k used, 0k free, 423180k cached
23:35 🔗 alard Good news for anyone not underscor: you can click a name to hide that downloader from the graph, so you can see yourself a little better. http://splinder.heroku.com/
23:35 🔗 Wyatt|Wor Oh yeah, I haven't offered my congratulations yet. alard, great work on getting a patch accepted to wget!
23:35 🔗 alard Thanks!
23:35 🔗 underscor alard: That's awesome
23:35 🔗 underscor Feels good to remove everyone else
23:35 🔗 underscor ;D
23:36 🔗 alard underscor: I thought you already did?
23:36 🔗 underscor huh?
23:36 🔗 Wyatt|Wor Wait, what are we graphing here? Is there some large-scale fetch task I missed in the hurlyburly of moving?
23:37 🔗 underscor 's funny how the graph changes when I remove myself
23:37 🔗 underscor Wyatt|Wor: splinder.com is shutting down in like 13 days
23:42 🔗 Wyatt|Wor Oh, okay. The wiki page hasn't been updated. I take it I have to make an account first, then point these github scripts at my account and let it run?
23:44 🔗 underscor Don't need an account
23:44 🔗 underscor Clone the repo, ./get-wget-warc.sh, ./dld-client.sh Wyatt
23:44 🔗 underscor (run a few of the clients if your io can take it)
23:48 🔗 Wyatt|Wor Understood; I'll get on that then.
23:51 🔗 alard underscor: Maybe it's time for some ops?
23:52 🔗 underscor :)
23:55 🔗 ndurner1 is there a reason why I am not a member of github.com/archiveteam anymore?
irclogger-viewer