#internetarchive.bak 2015-03-26,Thu

↑back Search

Time Nickname Message
00:04 🔗 GLaDOS has quit (Read error: Operation timed out)
00:05 🔗 GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak
00:05 🔗 svchfoo2 gives channel operator status to GLaDOS
00:35 🔗 svchfoo1 has quit (Read error: Operation timed out)
00:36 🔗 wp494_ (~wickedpla@[redacted]) has joined #internetarchive.bak
00:37 🔗 svchfoo1 (~chfoo1@[redacted]) has joined #internetarchive.bak
00:37 🔗 svchfoo2 gives channel operator status to svchfoo1
00:39 🔗 wp494 has quit (Read error: Operation timed out)
00:54 🔗 patricko- is now known as patrickod
01:04 🔗 wp494_ is now known as wp494
01:05 🔗 SketchCow closure: Yo, bro
01:06 🔗 SketchCow Turns out, flying to Sweden overnight got me a little time-messed.
01:23 🔗 patrickod is now known as patricko-
01:26 🔗 patricko- is now known as patrickod
01:32 🔗 patrickod is now known as patricko-
01:59 🔗 GLaDOS has quit (Read error: Operation timed out)
02:00 🔗 GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak
02:01 🔗 svchfoo2 gives channel operator status to GLaDOS
02:10 🔗 GLaDOS has quit (Ping timeout: 260 seconds)
02:11 🔗 GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak
02:11 🔗 svchfoo2 gives channel operator status to GLaDOS
02:26 🔗 GLaDOS has quit (Read error: Operation timed out)
02:26 🔗 GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak
02:27 🔗 svchfoo1 gives channel operator status to GLaDOS
02:39 🔗 GLaDOS has quit (Read error: Operation timed out)
02:42 🔗 GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak
02:42 🔗 svchfoo2 gives channel operator status to GLaDOS
02:52 🔗 GLaDOS has quit (Ping timeout: 260 seconds)
02:55 🔗 GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak
02:55 🔗 svchfoo2 gives channel operator status to GLaDOS
05:36 🔗 zottelbey (~zottelbey@[redacted]) has joined #internetarchive.bak
06:01 🔗 bzc6p has quit (bzc6p)
06:10 🔗 GLaDOS has quit (Read error: Operation timed out)
06:11 🔗 GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak
06:11 🔗 svchfoo2 gives channel operator status to GLaDOS
06:28 🔗 wp494 has quit (Ping timeout: 740 seconds)
06:53 🔗 wp494 (~wickedpla@[redacted]) has joined #internetarchive.bak
08:20 🔗 niyaje (~niyaje@[redacted]) has joined #internetarchive.bak
09:19 🔗 niyaje has quit (Ping timeout: 600 seconds)
09:20 🔗 niyaje (~niyaje@[redacted]) has joined #internetarchive.bak
09:42 🔗 Start has quit (ircd.shaw.ca irc.shaw.ca)
09:42 🔗 csssuf has quit (ircd.shaw.ca irc.shaw.ca)
09:42 🔗 garyrh has quit (ircd.shaw.ca irc.shaw.ca)
09:42 🔗 pikhq has quit (ircd.shaw.ca irc.shaw.ca)
09:42 🔗 wp494 has quit (ircd.shaw.ca irc.shaw.ca)
10:02 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
10:02 🔗 csssuf (~csssuf@[redacted]) has joined #internetarchive.bak
10:02 🔗 garyrh (garyrh@[redacted]) has joined #internetarchive.bak
10:02 🔗 irc.shaw.ca gives channel operator status to Start garyrh pikhq
10:02 🔗 pikhq (~pikhq@[redacted]) has joined #internetarchive.bak
10:02 🔗 wp494 (~wickedpla@[redacted]) has joined #internetarchive.bak
10:02 🔗 niyaje has quit (Ping timeout: 600 seconds)
10:10 🔗 niyaje (~niyaje@[redacted]) has joined #internetarchive.bak
10:29 🔗 cloudmons has quit (Read error: Connection reset by peer)
10:29 🔗 cloudmons (~quassel@[redacted]) has joined #internetarchive.bak
10:37 🔗 niyaje2 (~niyaje@[redacted]) has joined #internetarchive.bak
10:37 🔗 niyaje has quit (Ping timeout: 600 seconds)
10:41 🔗 niyaje (~niyaje@[redacted]) has joined #internetarchive.bak
10:47 🔗 niyaje3 (~niyaje@[redacted]) has joined #internetarchive.bak
10:47 🔗 niyaje2 has quit (Ping timeout: 600 seconds)
10:49 🔗 niyaje3 has quit (Client Quit)
10:54 🔗 niyaje has quit (Ping timeout: 600 seconds)
11:21 🔗 SketchCo1 (~jscott@[redacted]) has joined #internetarchive.bak
11:21 🔗 Kazzy_ (~Kaz@[redacted]) has joined #internetarchive.bak
11:26 🔗 Kazzy has quit (hub.se efnet.portlane.se)
11:26 🔗 SketchCow has quit (hub.se efnet.portlane.se)
11:26 🔗 underscor has quit (hub.se efnet.portlane.se)
11:42 🔗 underscor (~quassel@[redacted]) has joined #internetarchive.bak
11:42 🔗 Kazzy_ is now known as Kazzy
11:42 🔗 svchfoo2 gives channel operator status to Kazzy
13:46 🔗 Start has quit (Disconnected.)
14:22 🔗 bzc6p (~bzc6p@[redacted]) has joined #internetarchive.bak
14:34 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
14:35 🔗 Start has quit (Read error: Connection reset by peer)
14:35 🔗 Start_ (~Start@[redacted]) has joined #internetarchive.bak
14:46 🔗 Start_ has quit (Read error: Connection reset by peer)
14:46 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
15:18 🔗 Start has quit (Disconnected.)
15:25 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
15:26 🔗 Start has quit (Read error: Connection reset by peer)
15:28 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
15:29 🔗 kofica (~TheBuda@[redacted]) has joined #internetarchive.bak
15:29 🔗 kofica @find jerusalem bible
15:29 🔗 kofica (~TheBuda@[redacted]) has left #internetarchive.bak
15:30 🔗 kofica (~TheBuda@[redacted]) has joined #internetarchive.bak
15:30 🔗 kofica (~TheBuda@[redacted]) has left #internetarchive.bak
15:43 🔗 zottelbey has quit (Remote host closed the connection)
15:51 🔗 garyrh has quit (Remote host closed the connection)
15:51 🔗 Start has quit (Disconnected.)
16:01 🔗 VADemon (~VADemon@[redacted]) has joined #internetarchive.bak
16:22 🔗 zottelbey (~zottelbey@[redacted]) has joined #internetarchive.bak
16:24 🔗 zottelbey has quit (Remote host closed the connection)
16:26 🔗 zottelbey (~zottelbey@[redacted]) has joined #internetarchive.bak
16:28 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
16:29 🔗 Start has quit (Read error: Connection reset by peer)
16:29 🔗 Start_ (~Start@[redacted]) has joined #internetarchive.bak
16:29 🔗 zottelbey has quit (Remote host closed the connection)
16:29 🔗 Start_ is now known as Start
16:33 🔗 zottelbey (~zottelbey@[redacted]) has joined #internetarchive.bak
16:45 🔗 Start has quit (Disconnected.)
16:55 🔗 patricko- is now known as patrickod
17:08 🔗 patrickod is now known as patricko-
17:43 🔗 SketchCo1 is now known as SketchCOw
17:43 🔗 SketchCOw is now known as SketchCow
17:43 🔗 svchfoo2 gives channel operator status to SketchCow
17:50 🔗 SketchCow closure: I'm around
17:54 🔗 closure SketchCow: so, we need a machine or container that can be the git server. Any thoughts what to use?
17:56 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
18:01 🔗 SketchCow How much space does it need?
18:01 🔗 SketchCow I can probably throw you on sisyphus
18:02 🔗 closure I think not a whole lot of space.. let's see
18:03 🔗 SketchCow Also, this MIGHT be a job for Kenshin
18:03 🔗 SketchCow Who wants to help but IA may be slow talking about CDN, while I am much happier to utilize resources for a project like this.
18:04 🔗 yipdw get closure on sisphyus and get the ball rollin'
18:04 🔗 closure the demo shard is 51 mb
18:04 🔗 closure times 1770 shards
18:04 🔗 closure hardly nothing
18:07 🔗 closure we may end up wanting a separate unix account per shard though, or something like that, to limit the ssh keys that can access it
18:09 🔗 SketchCow 90gb, if I see that correctly.
18:09 🔗 SketchCow yipdw: Yes, I agree
18:11 🔗 yipdw oh I was going for the rock puns
18:12 🔗 SketchCow So, game plan.
18:12 🔗 SketchCow 1. Ignore people saying it can't be done, we're fucked, look I have numbers
18:12 🔗 SketchCow 2. Set up a version of this with teamarchive1/sisyphus
18:13 🔗 SketchCow 3. Have a few folks using it who step forward, who are not anywhere on IA infrastructure
18:13 🔗 SketchCow 4. Deal with the 1,409 problems that crop up
18:13 🔗 SketchCow 5. Hack away at a pretty interface once this is working, to show it wokring
18:13 🔗 SketchCow 6. Increase backup size when we think it works and more people step forward
18:13 🔗 SketchCow 7. Repeat 6
18:14 🔗 SketchCow 8. Success/Failure/Wonder/Sadness
18:14 🔗 yipdw where does "try a restore" fit in?
18:14 🔗 yipdw I mostly just want to see that work; I know it's going to be largely provided by git-annex
18:16 🔗 closure that's basically the same list I have at http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation
18:21 🔗 closure btw the dshr article is interesting, entangled storage is a neat thing (that I'd be happy to add a git-annex backend for if someone points me at the tools), but I think he has some holes in his analysis
18:21 🔗 SketchCow closure: teamarchive1.fnf.archive.org
18:26 🔗 Kenshin closure: let me know how much resource you need in terms of ram/hdd storage (please don't tell me the more the merrier), how many VMs you need, plan to need, etc
18:27 🔗 Kenshin i just got back so i'm going to clear up some pending stuff, but i'll jump in shortly after
18:28 🔗 SketchCow closure: Let me know if you hit snags with logging in
18:28 🔗 closure Kenshin: this is the controller, not the storage, it doesn't need much
18:28 🔗 closure SketchCow: logged in. now what
18:28 🔗 SketchCow Kenshin: I suspect this will.. yeah, what he said
18:28 🔗 SketchCow closure: hahahah
18:28 🔗 SketchCow THE QUESTION
18:28 🔗 SketchCow That question every nerd asks after a successful slack or OS install
18:28 🔗 closure my thought would be: add a SHARD1 user account
18:28 🔗 Kenshin closure: so i assume ssd storage would be better :)
18:28 🔗 Kenshin closure: high RAM requirement?
18:28 🔗 closure Kenshin: absolutely
18:29 🔗 SketchCow closure: Try doing a sudo
18:29 🔗 closure ram should be low, it's just running some git stuff
18:29 🔗 Kenshin where do you prefer it? lax, nyc, uk
18:31 🔗 closure Kenshin: near IA seems to make sense
18:32 🔗 closure closure is not in the sudoers file. This incident will be reported.
18:32 🔗 Kenshin LAX then. what's your preferred distribution?
18:32 🔗 closure debian
18:33 🔗 closure christ SketchCow this has a lot of cpus
18:34 🔗 closure oh, it's qemu
18:35 🔗 SketchCow OH SHIT REPORTED INCIDENT
18:35 🔗 SketchCow and closure run through the streets, dogs barking, lights shining on them
18:35 🔗 SketchCow We had a good run, man
18:35 🔗 SketchCow holds the bullet wound but it's too late
18:36 🔗 SketchCow tell..
18:36 🔗 SketchCow telll underscor his ass was mighty fine
18:36 🔗 closure the box.. it's full of cpus
18:36 🔗 SketchCow closure: Fixed
18:36 🔗 SketchCow Have at, try not destroy the box, I do like it
18:37 🔗 Start has quit (Disconnected.)
18:38 🔗 SketchCow So, just one example where I disagree with Rosenthal
18:38 🔗 SketchCow He talks about the recovery situation
18:38 🔗 SketchCow And he mentions the crater scenario (no IA) and the slightly gon scenario (a drive pair blew up)
18:39 🔗 SketchCow And he then applies current bandwidth situation as his calculation for how long recovery takes.
18:39 🔗 SketchCow Except
18:39 🔗 SketchCow A cratered, returned IA will be hosted somewhere else, and I guarantee the organization would be paying for RIDICULOUS amounts of upstream
18:39 🔗 SketchCow RIDICULOUS
18:40 🔗 SketchCow And we'd have huge amounts of changes for allowing the maximum amount of data to flow in.
18:41 🔗 SketchCow So that's one problem right there. An academic forgets how much money can be thrown at a problem and how much people will do if given enough money for a task.
18:41 🔗 SketchCow But I'm more concerned about lost items. specific ones
18:41 🔗 SketchCow Either by mistake, hard drive failure, etc.
18:59 🔗 patricko- is now known as patrickod
19:03 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
19:19 🔗 closure git clone SHARD1@124.6.40.227:shard1
19:19 🔗 closure I'm using Kenshin's VM for now, I like ssd
19:20 🔗 closure give me a ssh public key and I'll give you access to this repo, to tes it
19:25 🔗 Start has quit (Disconnected.)
19:29 🔗 patrickod is now known as patricko-
19:32 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
19:33 🔗 Start has quit (Read error: Connection reset by peer)
19:33 🔗 Start_ (~Start@[redacted]) has joined #internetarchive.bak
19:36 🔗 Start_ is now known as Start
19:38 🔗 closure anyone? cat .ssh/id_rsa.pub to me
19:43 🔗 Sanqui how big is it?
19:45 🔗 Start has quit (Read error: Connection reset by peer)
19:45 🔗 Start_ (~Start@[redacted]) has joined #internetarchive.bak
19:45 🔗 Start_ is now known as Start
19:46 🔗 closure 50 mb
19:46 🔗 patricko- is now known as patrickod
19:48 🔗 closure sep332: you should be able to clone it now, and see http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation#demo_shard for other commands
19:49 🔗 closure I have also done some initial tweaks to make the repo mostly read-only
19:53 🔗 patrickod is now known as patricko-
20:00 🔗 sep332 so my username is SHARD1 ?
20:01 🔗 closure that's the git repo's username, yes
20:07 🔗 sep332 got it. what's a good way to test?
20:08 🔗 sep332 after a git clone
20:08 🔗 closure pick some of the items, and git annex get them, to start
20:08 🔗 closure then git annex sync , and it should tell the server what you have
20:08 🔗 sep332 oh you updated the wiki already :p
20:08 🔗 sep332 thanks
20:09 🔗 closure you could just "git annex get ." to start downloading from the top
20:09 🔗 closure obvs we'll need better ways to allocate items to clients later
20:10 🔗 closure it's around 2 tb if you download everything ;)
20:10 🔗 sep332 ok, i've got ~800GB from the other test shard so far
20:10 🔗 closure which other one? the old http url?
20:11 🔗 sep332 on another box though so I can't just cp them... i should just fix that box i guess
20:11 🔗 closure if you already downloaded that much, you should convert it to use the new repo as origin. It's the same git repo, same daya
20:11 🔗 closure data
20:11 🔗 sep332 from testrepo1
20:11 🔗 closure yep, shard1 *is* testrepo1
20:12 🔗 closure good lord 800 gb?
20:12 🔗 sep332 gotcha. my git-fu is nonexistant :)
20:12 🔗 closure oh right, this is ArchiveTeam, give them a list of urls, and wham
20:12 🔗 sep332 well i had a 4TB HD sitting here all lonely
20:13 🔗 closure so, you'll need to give me the ssh public key for the account on that other box, and then edit .git/config, swap out the http url with SHARD1@124.6.40.227:shard1
20:13 🔗 closure and then it will just switch over
20:14 🔗 closure nice to know you could pull 800 gb this way from IA over the past 2 weeks I was away
20:14 🔗 sep332 i can use the same key.
20:14 🔗 closure sure
20:17 🔗 closure so here's something you could do.. once you set up that other box, run git annex sync in both repos. Then, in the new repo: git annex get --not --copies=2
20:17 🔗 closure that will only download files that don't have 2 known copies, and the IA counts as 1 copy, so it will get files you have not already gotten in the old repo
20:17 🔗 sep332 ok cool
20:18 🔗 sep332 i'm getting an error on git annex sync though
20:18 🔗 sep332 testshard: error while loading shared libraries: testshard: cannot open shared object file: No such file or directory
20:18 🔗 closure hmm
20:18 🔗 sep332 Please make sure you have the correct access rights and the repository exists.
20:19 🔗 closure I think that's a bug with your installation of git-annex
20:19 🔗 sep332 it says that twice and then
20:19 🔗 sep332 (non-fast-forward problems can be solved by setting receive.denyNonFastforwards to false in the remote's git config)
20:19 🔗 closure I guess you're using the standalone tarball of git-annex and maybe not in the right way
20:19 🔗 sep332 that's... spookily accurate
20:19 🔗 closure I actually tried these shards with the git-annex in debian stable, and it works ok
20:20 🔗 closure it just can't git annex fsck, otherwise things work
20:20 🔗 sep332 oh! ok then. i'll do that.
20:20 🔗 closure or you might try git-annex.linux/runshell , which should give you a shell environment using the right libs
20:22 🔗 Start has quit (Disconnected.)
20:23 🔗 sep332 runshell doesn't seem to help. i'm going to try whatever ubuntu has for git-annex
20:23 🔗 closure hmm, I'm not so sure it's the local install
20:24 🔗 closure paste git-annex sync --debug
20:29 🔗 sep332 http://pastebin.com/raw.php?i=FdbmsBQn
20:32 🔗 closure what does this say? git config remote.origin.url
20:33 🔗 sep332 testshard:shard1
20:33 🔗 sep332 testshard is a line in my .ssh/config
20:33 🔗 sep332 124.6.40.227, user SHARD1
20:33 🔗 closure yeah, I think you got that wrong somehow :)
20:34 🔗 closure maybe run: git config remote.origin.url SHARD1@124.6.40.227:shard1
20:36 🔗 sep332 i get the same error on git-annex sync though
20:37 🔗 closure git fetch origin is what's failing there
20:38 🔗 sep332 ok now how do i tell git to use my key without putting it in .ssh/config ?
20:39 🔗 closure oh, I see your problem there.. yeah, that's a pain to set up. Why not just give me a second ssh key?
20:40 🔗 sep332 i reverted to testshard:shard1 and "git fetch origin" doesn't fail
20:40 🔗 sep332 doesn't seem to do anything
20:42 🔗 sep332 ok i got SHARD1@124.6.40.227:shard1 to work using ssh-agent
20:42 🔗 sep332 but same old error
20:51 🔗 closure dude, I dunno. you're way down some nonstandard, bad-idea config rabbithole
20:51 🔗 sep332 alright, don't sweat it then
20:51 🔗 closure when git starts trying to load your ssh/.config dummy hostname as a shared library, you've probably done something wrong
20:52 🔗 sep332 i'll sleep on it and poke it with a stick tomorrow :)
21:45 🔗 zottelbey has quit (Remote host closed the connection)
22:00 🔗 patricko- is now known as patrickod
22:02 🔗 patrickod is now known as patricko-
22:27 🔗 kaizoku (~kaizoku@[redacted]) has left #internetarchive.bak
22:39 🔗 Start (~Start@[redacted]) has joined #internetarchive.bak
22:39 🔗 svchfoo1 gives channel operator status to Start
23:00 🔗 db48x closure: this isn't really your department, but when I try to install git annex from source it fails to install one of the dependencies (Test.Tasty.QuickCheck). have you noticed this?

irclogger-viewer