[00:27] *** patricko- is now known as patrickod
[00:30] *** patrickod is now known as patricko-
[01:03] *** patricko- is now known as patrickod
[01:16] *** patrickod is now known as patricko-
[02:26] <closure> db48x: maybe
[02:28] <db48x> thanks :)
[02:28] <db48x> https://pastebin.mozilla.org/8827827
[02:29] <db48x> this is when I do a cabal build
[02:32] <closure> hmm, that error message doesn't make a great deal of sense. It's complaining about database code generated by a macro
[02:33] <closure> what version does this show? ghc-pkg list persistent
[02:45] <tpw_rules> i'm still not getting very good speed
[02:45] <closure> tpw_rules: one thing you could do is run multiple git-annex get in parallel
[02:45] <closure> it'll parallellize fine
[02:46] <tpw_rules> will they not munch over each other?
[02:46] <closure> nope
[02:46] <closure> be a bit faster if you run them on different directories, but even on the same one they'll cooperate
[02:51] <tpw_rules> coolio. any other shards out yet?
[02:53] <tpw_rules> 8 in parallel!
[02:53] <tpw_rules> http://cl.ly/image/0F0G1T2F0c42/Screen%20Shot%202015-03-31%20at%209.52.34%20PM.png
[03:00] <closure> -------------------------
[03:00] <closure> ----------------------
[03:00] <closure> remember to 'git annex sync' every now and then so we know what you have
[03:00] <closure> also, I really want to put together some runner scripts in a repo, to handle this stuff
[03:01] <tpw_rules> can it sync while it downloads?
[03:01] <closure> yes, it can
[03:01] <closure> every single git-annex command parallelizes with every other command, except for git annex assistant (and things will detect it andd refuse to run)
[03:01] <tpw_rules> wow, that's really neat
[03:02] <closure> it'd be neater if I had -jN built in, but I don't. On the plus side, I have an open ticket about it from $EMPLOYER :)
[03:02] <tpw_rules> did you make it?
[03:02] <closure> yes
[03:04] <tpw_rules> how often is now and then? once a day?
[03:05] <closure> sounds ok.. I was thinking if there's a script it could do it once per hour
[03:05] <tpw_rules> i was gonna set it on a cronjob
[03:09] <tpw_rules> is there a way to rate-limit all annexes running?
[03:10] <tpw_rules> i'm running them all in one tmux session if that's a linux-specific thing
[03:10] <db48x> closure:
[03:10] <db48x>    persistent-1.3.1.1
[03:10] <db48x>    persistent-2.0.8
[03:13] <closure> hmm, my guess is the 2.0.8 one, I've never seen that one
[03:14] <closure> try: cabal configure --constraint "persistent == 1.3.1.1"
[03:28] <db48x> same error
[03:29] <SketchCow> closure: Agreed on runner scripts.
[03:29] <db48x> it's still loading 2.0.8
[03:39] <closure> db48x: maybe try cabal update; cabal install persistent-2.1.2
[03:44] <db48x> heh, same error and it's still loading 2.0.8
[03:45] <closure> cabal configure -v2 or -v3 will tell you wby, but this gets hairy
[03:45] <closure> probably need to update some other library too
[03:46] <db48x> Dependency persistent ==2.0.8: using persistent-2.0.8
[03:47] <db48x> doesn't say what it's a dependency of though...
[03:48] <db48x> ah
[03:49] <db48x> https://pastebin.mozilla.org/8827836
[03:55] <closure> db48x: at this point, I'd just blow away ~/.cabal and ~/.ghc and start over ..
[03:56] <closure> so, I have a runner script. Handles installing git-annex, setting up a dedicated ssh key, cloning repo, running downloads
[03:59] <closure> can someone get me set up on the archiveteam github?
[04:09] <db48x> closure: sure, what's your username on github?
[04:10] <db48x> probably joeyh, unless that's an impersonator...
[04:12] <closure> yeah, that'll do
[04:12] <closure> can you do me a favor and make the repo too?
[04:15] <db48x> what would you like to call it?
[04:15] <closure> that's why I asked you to make it :)
[04:15] <db48x> ah
[04:16] <db48x> github suggests petulant-happiness
[04:18] <db48x> closure: do you have any particular license in mind?
[04:19] <closure> nah, whatever's typical for archiveteam scripts
[04:26] <db48x> https://github.com/ArchiveTeam/IA.BAK
[04:33] <closure> ERROR: Permission to ArchiveTeam/IA.BAK.git denied to joeyh.
[04:33] <closure> hrm
[04:33] <chfoo> i think a team needs to be created for it
[04:34] <db48x> did you accept the invitation?
[04:36] *** db48x sighs
[04:36] <db48x> dist/build/git-annex/git-annex-tmp/Utility/Format.o: In function `sT2A_info':
[04:36] <db48x> (.text+0x57e0): undefined reference to `shakespearezm2zi0zi0zi3_TextziHamlet_zdslookup1_info'
[04:37] <closure> you probably need to cabal clean and restart build
[04:38] <closure> ah here's the invite now
[04:40] <chfoo> oh, never mind about the team. since it was an invite to be an admin, the team would be redundant
[04:42] <SketchCow> Sweet
[04:51] <closure> wow, it seems to work
[04:58] <closure> ok, this is ready for someone else to try
[04:58] <closure> https://github.com/ArchiveTeam/IA.BAK and have at it!
[05:03] <db48x> finally:
[05:03] <db48x> git-annex version: 5.20150331-gf842888
[05:03] <db48x>   anglachel  db48x  ~  src  git-annex  git annex version
[05:06] <closure> db48x: about that persistent-2.0.8 version, was that shipped in some distribution or something?
[05:06] <db48x> no clue
[05:07] <db48x> hrm
[05:07] <db48x> with celebdil and fanuidhol offline, I don't have much disk space...
[05:13] <closure> ------------------------------------------------------------
[05:14] <closure> If you're running git-annex manually, https://github.com/ArchiveTeam/IA.BAK has instructions to switch to using it
[05:14] <closure> ------------------------------------------------------------
[05:17] *** cloudyys (~cloudcake@[redacted]) has joined #internetarchive.bak
[05:31] <db48x> closure: ------------------>8----------------------
[05:31] <db48x> ------------------>8----------------------
[05:31] <db48x> ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLutoB3Ru0W6MHBYBNVGzJZj8anW9eFfXjAkm1pk0+OLWexjuIbLyHZzXhW3umgXvTMazk7RDmAe79t18aEzWDb8/Irm8QiXxhaitp/OAM1yneQG414pCWIKmZLV+cvLGLEd+IUGsaowlvugNdUc7A6VlDZzm9+gpb9tojgFMDuKxrfuMd/HvS+AwGVAXhfuu3BWtD/hN1lQTR/6MtWM6DcyQkkg6ry3OE7KtOfqTcqvno3aXQnLKG9VNmWS0Si/1b80Wse/n2qGGsZGNp0UtBqRV+tFm/OzbziTELYUNfFyiwkLHiJTDsRZVINKGjQ3HI8rzhQjHbPU2YbYCquJRt db48x@anglachel
[05:33] <closure> db48x: set up
[05:34] <db48x> closure: thank you, kind sir!
[05:36] <csssuf> that's a clever scissor emoticon
[05:36] <DFJustin> ✁
[05:37] <SketchCow> closure - Need a webpage soft link on your teamarchive1 account so it can be browsed for updates?
[05:37] <SketchCow> Like some script that runs twice a day to see the updating of how many copies are out there?
[05:38] <SketchCow> https://developers.google.com/chart/ has pretty, pretty charts
[05:42] <closure> SketchCow: yeah, make that link and I'll put something there tomorrow
[05:42] <closure> it'll probably be a json dump but ..
[05:43] <SketchCow> https://google-developers.appspot.com/chart/interactive/docs/gallery/gauge I love this one and I have no idea where it'd be used.
[05:46] <SketchCow> OK, html in your directory on teamarchive1
[05:46] <SketchCow> Or do I have you elsewhere
[05:46] <closure> teamarchive1 is fine
[05:50] <SketchCow> closure - So it's possible to set up an area map we could use.
[05:50] <SketchCow> Do you have the stats handy?
[05:50] <SketchCow> Newest? i'll even take older stats.
[05:51] <db48x> closure: https://github.com/ArchiveTeam/IA.BAK/pull/1
[05:53] <closure> db48x: yeah, I just always installed git-annex rather than check version number
[05:54] <closure> the script uses one feature that needs a rather new git-annex (from january or so)
[05:59] <SketchCow> http://teamarchive1.fnf.archive.org/ia.bak/graph.html
[05:59] <SketchCow> :)
[06:00] <db48x> heh
[06:01] <closure> 	numcopies +0: 68417
[06:01] <closure> 	numcopies +1: 32991
[06:01] <closure> 	numcopies +2: 1932
[06:01] <closure> 	numcopies +3: 3
[06:01] <closure> that's current stats
[06:03] <SketchCow> Do a shift-reload on http://teamarchive1.fnf.archive.org/ia.bak/graph.html and boom
[06:03] <SketchCow> How long do the stats take to generate?
[06:03] <closure> a few minutes
[06:05] <closure> with the new script, clients will be reporting every hour when they're downloading
[06:05] <closure> so I'll just run it every half hour or so
[06:06] <SketchCow> Is it somewhere on teamarchive1 that I can pull from it?
[06:06] <SketchCow> Graph now has a little explanatory top.
[06:10] <closure> ~closure/iabak-stats
[06:10] <closure> pretend that's updating, but it won't until tomorrow..
[06:18] *** bzc6p_ (~bzc6p@[redacted]) has joined #internetarchive.bak
[06:20] *** bzc6p has quit (Read error: Operation timed out)
[06:23] <SketchCow> OK, I wrote the script.
[06:23] <SketchCow> it's sitting in there.
[06:23] <SketchCow> graph-gen.sh
[06:23] <SketchCow> I'm going to make it automatically run every 30 minutes.
[06:24] *** GitHub17 (~GitHub17@[redacted]) has joined #internetarchive.bak
[06:24] *** GitHub17 (~GitHub17@[redacted]) has left #internetarchive.bak
[06:24] <GitHub17> IA.BAK/public-pubic-typo 0e2c810 David Yip: Fix typo.
[06:24] <GitHub17> [IA.BAK] yipdw created public-pubic-typo (+1 new commit): http://git.io/jdLT
[06:25] *** GitHub183 (~GitHub183@[redacted]) has joined #internetarchive.bak
[06:25] *** GitHub183 (~GitHub183@[redacted]) has left #internetarchive.bak
[06:25] <GitHub183> [IA.BAK] yipdw opened pull request #2: Fix typo in iabak public key display (master...public-pubic-typo) http://git.io/jdLZ
[06:25] <closure> hah, pubic keys
[06:26] <yipdw> pubic keys protect private secrets
[06:28] *** GitHub5 (~GitHub5@[redacted]) has joined #internetarchive.bak
[06:28] *** GitHub5 (~GitHub5@[redacted]) has left #internetarchive.bak
[06:28] *** GitHub78 (~GitHub78@[redacted]) has joined #internetarchive.bak
[06:28] *** GitHub78 (~GitHub78@[redacted]) has left #internetarchive.bak
[06:28] <GitHub5> IA.BAK/master 82edd72 Joey Hess: Merge pull request #2 from ArchiveTeam/public-pubic-typo...
[06:28] <GitHub78> [IA.BAK] joeyh closed pull request #2: Fix typo in iabak public key display (master...public-pubic-typo) http://git.io/jdLZ
[06:28] <GitHub5> [IA.BAK] joeyh pushed 1 new commit to master: http://git.io/jdLd
[06:29] *** yipdw sets mode -n #internetarchive.bak
[06:32] <SketchCow> OK, crontab is in for graph.
[06:34] <yipdw> agh, somehow I forgot that my storage pool runs on Solaris
[06:34] <yipdw> this got a bit harder for me, will have to find some way
[06:37] <yipdw> huh, I guess this works over CIFS
[06:44] <closure> probably. However, disregard anything I said above about it supporting parallel operation when on a network filesystem..
[06:48] <yipdw> yeah, that's fine
[07:05] <SketchCow> (Idly checked... script that makes the graph is now properly running and updating.)
[07:47] *** cloudyys has quit (Read error: Connection reset by peer)
[10:00] <hater> anyone here who has write-access to https://github.com/ArchiveTeam/IA.BAK ?
[10:01] <hater> i wrote a simple patch to check if all used tools (in the iabak-helper script) are available
[10:01] <hater> http://pastebin.com/raw.php?i=EdNLPS05
[10:12] <db48x> hater: :D
[10:13] <db48x> hater: closure is probably asleep, but if you make a pull request he'll see it in the morning
[10:13] <hater> oh, i forget the timeshift between countries every time..
[10:16] <db48x> no worries
[10:17] <hater> db48x: i knew i know your nick - i commented your pull request :D
[10:17] <db48x> aha :)
[10:31] <GitHub59/#internetarchive.bak> [IA.BAK] cancerAlot opened pull request #3: checkDeps should check if every tool is available (master...master) http://git.io/jF3E
[10:31] <hater> nice bot
[11:32] *** zottelbey (~zottelbey@[redacted]) has joined #internetarchive.bak
[13:51] <hater> db48x: git-annex version: 5.20150327-g19a1a35   - is the g19a1a35 important?
[14:15] <sep332> hater: anything 5.20150205 or later should work
[14:18] <hater> sep332: i'm writing a version-checker for the iabak-helper
[14:19] <hater> if the g19a1a35-part is not important, the comparison will be much easier
[14:19] <closure> that is the git rev, it's not important
[14:20] <hater> ok, thx
[15:05] <GitHub97/#internetarchive.bak> IA.BAK/server 0d67fac Joey Hess: add my server-side scripts
[15:05] <GitHub97/#internetarchive.bak> IA.BAK/server 3cbe617 Joey Hess: ssh pub keys
[15:05] <GitHub97/#internetarchive.bak> IA.BAK/server 8811b55 Joey Hess: move to server branch
[15:05] <GitHub97/#internetarchive.bak> [IA.BAK] joeyh created server (+3 new commits): http://git.io/jNJb
[15:05] <GitHub142/#internetarchive.bak> IA.BAK/master 0bf2f69 Joey Hess: moved to server branch
[15:05] <GitHub142/#internetarchive.bak> IA.BAK/master 930a844 Joey Hess: Merge remote-tracking branch 'origin/master'
[15:05] <GitHub142/#internetarchive.bak> [IA.BAK] joeyh pushed 2 new commits to master: http://git.io/jNUk
[15:09] <GitHub118/#internetarchive.bak> [IA.BAK] joeyh created pubkey (+1 new commit): http://git.io/jNkT
[15:09] <GitHub118/#internetarchive.bak> IA.BAK/pubkey ec50140 Joey Hess: move pubkey to its own branch...
[15:09] <GitHub149/#internetarchive.bak> IA.BAK/server 7b41b4a Joey Hess: moved pubkeys to a separate branch
[15:09] <GitHub149/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNkk
[15:24] <GitHub137/#internetarchive.bak> IA.BAK/server 37e9b37 Joey Hess: adding web hook for github repo
[15:24] <GitHub137/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNYw
[15:26] <GitHub3/#internetarchive.bak> IA.BAK/server f248947 Joey Hess: cgi gunk
[15:26] <GitHub3/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNOd
[15:29] *** svchfoo2 has quit (Read error: Connection reset by peer)
[15:33] *** svchfoo2 (~chfoo2@[redacted]) has joined #internetarchive.bak
[15:39] <GitHub138/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNCK
[15:39] <GitHub138/#internetarchive.bak> IA.BAK/server 4d86ca5 Joey Hess: finished web hook
[15:40] <GitHub94/#internetarchive.bak> IA.BAK/server 031215b Joey Hess: allow seeing output
[15:40] <GitHub94/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNWU
[15:42] <GitHub99/#internetarchive.bak> IA.BAK/server 8e2252a Joey Hess: auto-update SHARD1 authorized_keys
[15:42] <GitHub99/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNW5
[15:43] <GitHub26/#internetarchive.bak> IA.BAK/server 60dd3f5 Joey Hess: typo
[15:43] <GitHub26/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNlY
[15:44] <GitHub105/#internetarchive.bak> IA.BAK/server 6898ed3 Joey Hess: typo
[15:44] <GitHub105/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jN80
[15:47] <GitHub43/#internetarchive.bak> IA.BAK/server e24c02e Joey Hess: fixes
[15:47] <GitHub43/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNBL
[15:48] <GitHub95/#internetarchive.bak> IA.BAK/server f82b548 Joey Hess: bugfix
[15:48] <GitHub95/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNRv
[15:50] <closure> ok, there's a pubkey branch in IA.BAK git repo now and anyone who can commit to it can add ssh public keys, which will auto-deploy to the server
[15:51] <closure> also, the entire contents of the server is backed up on a couple git repos.. IA.BAK for the scripts and the clones of the shards themselves :)
[15:51] <closure> self-backing up project ;)
[15:52] <SketchCow> Are the stats generating yet? :)
[16:01] <closure> almost there
[16:03] <closure> I need a way to rsync a directory from Kenshin's server to teamarchive1
[16:03] *** svchfoo1 has quit (Read error: Connection reset by peer)
[16:04] <closure> or I could put them up on the web, but for the geo data, the IP addresses would need to be procesed first
[16:05] *** svchfoo1 (~chfoo1@[redacted]) has joined #internetarchive.bak
[16:06] <GitHub34/#internetarchive.bak> IA.BAK/pubkey 16146e9 Joey Hess: add Kazzy
[16:06] <GitHub34/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to pubkey: http://git.io/jNVr
[16:06] <Kazzy> cheers
[16:06] <closure> Kazzy: and nice point the people can pull-request to sign up! Although it might be better for someone to make a web site for registration, that just commits the pubkey
[16:07] <SketchCow> We'll figure out geo in a bit, but the stats generating frequently is a (relatively easy) win.
[16:07] <SketchCow> I figured out in theory how to use the IP list to make a really nice graph
[16:08] <SketchCow> So that's there.
[16:16] *** svchfoo1 has quit (Quit: Closing)
[16:19] *** svchfoo2 has quit (Quit: Closing)
[16:22] <GitHub81/#internetarchive.bak> IA.BAK/server 94399dc Joey Hess: stats gatherer
[16:22] <GitHub81/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNXM
[16:24] <GitHub11/#internetarchive.bak> IA.BAK/server 3c7214f Joey Hess: fix dir name
[16:24] <GitHub11/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNMJ
[16:25] <GitHub126/#internetarchive.bak> IA.BAK/server 79ca1db Joey Hess: typo
[16:25] <GitHub126/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jNMR
[16:25] *** patricko- is now known as patrickod
[16:28] *** svchfoo2 (~chfoo2@[redacted]) has joined #internetarchive.bak
[16:30] *** patrickod is now known as patricko-
[16:35] *** You've invited svchfoo3 to #internetarchive.bak (irc.teksavvy.ca)
[16:35] *** svchfoo3 (~chfoo3@[redacted]) has joined #internetarchive.bak
[16:35] *** chfoo gives channel operator status to svchfoo2 svchfoo3
[16:41] <GitHub94/#internetarchive.bak> IA.BAK/server f481ac9 Joey Hess: set git commiter info (silly git..):
[16:41] <GitHub94/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jN7L
[16:42] <GitHub33/#internetarchive.bak> IA.BAK/server c7f425b Joey Hess: update stats file atomically via tmp file
[16:42] <GitHub33/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jN7r
[16:42] <closure> SketchCow: http://ia-bak.joeyh.name/stats/
[16:43] <GitHub27/#internetarchive.bak> IA.BAK/server c20a5c8 Joey Hess: adjust to use newer apache html dir...
[16:43] <GitHub27/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to server: http://git.io/jN5U
[16:43] <closure> still going up I see!
[16:50] *** patricko- is now known as patrickod
[16:52] <GitHub75/#internetarchive.bak> IA.BAK/server a4a8aab Joey Hess: add scripts I used to make SHARD1
[16:52] <GitHub75/#internetarchive.bak> IA.BAK/server d3c6fa0 Joey Hess: script to setup a shard's git repo hooks
[16:52] <GitHub75/#internetarchive.bak> [IA.BAK] joeyh pushed 2 new commits to server: http://git.io/jNNE
[16:55] <SketchCow> closure: You want me to grab from there instead of the local file?
[16:56] <closure> SketchCow: yes please
[17:00] <SketchCow> OK, it's going. Thank you!
[17:00] <GitHub81/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to master: http://git.io/jNjd
[17:00] <GitHub81/#internetarchive.bak> IA.BAK/master b01ee4b Joey Hess: touch NOSHUF to disable the shuffling, if it's too expensive
[17:01] <SketchCow> http://teamarchive1.fnf.archive.org/ia.bak/ will now update every 30 minutes from your script
[17:01] <closure> btw, if we could get a better dns for the iabak server, something under archiveteam.org..
[17:02] <SketchCow> What's the ip
[17:03] *** svchfoo1 (~chfoo1@[redacted]) has joined #internetarchive.bak
[17:03] *** svchfoo2 gives channel operator status to svchfoo1
[17:04] <closure> 124.6.40.227
[17:04] <GitHub145/#internetarchive.bak> IA.BAK/master a40453a Joey Hess: make sure we have the right git-annex installed even if shard repo already exists
[17:04] <GitHub145/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to master: http://git.io/jAvF
[17:05] <SketchCow> iabak.archiveteam.org now points there.
[17:05] <GitHub194/#internetarchive.bak> IA.BAK/master 278f4e0 Joey Hess: fix path to NOSHUF
[17:05] <GitHub194/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to master: http://git.io/jAfs
[17:06] <closure> SketchCow: so, update your cron job to use the url http://iabak.archiveteam.org/stats/
[17:07] <GitHub32/#internetarchive.bak> IA.BAK/master 2d3ff23 Joey Hess: use shiny new domain name
[17:07] <GitHub32/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to master: http://git.io/jAJT
[17:08] <SketchCow> Done
[17:09] <SketchCow> So, just to put this in here, and I don't want to address it now
[17:09] <SketchCow> IA co-worker asks what the potential is for a contributing user to also be a seed for the torrents for the items.
[17:09] <SketchCow> I think it adds a lot of burden, I'd rather we stay on the current incremental path
[17:10] <closure> well, they have the torrent files, and they have the contents of the torrent.. so they can run a torrent client
[17:10] <closure> but it doesn't help the other ia.bak clients get it any quicker
[17:10] <SketchCow> Sort of.
[17:10] <SketchCow> they have SOME of them
[17:11] <closure> yeah, they can git annex find some that they have
[17:11] <SketchCow> No, no. This is literally to allow someone to download a hot item quicker
[17:11] <closure> there are probably more torrents in a shard than can be managed by most torrent clients
[17:13] <SketchCow> Agreed
[17:13] *** closure goes away to implement distributed global fsck in git-annex
[17:14] *** patrickod is now known as patricko-
[17:39] *** patricko- is now known as patrickod
[17:46] *** patrickod is now known as patricko-
[18:33] <midas> closure: if the download speed is higher from torrents we could use it tho
[18:34] <sep332> maybe just for large files?
[18:35] *** patricko- is now known as patrickod
[18:35] *** patrickod is now known as patricko-
[18:53] <GitHub166/#internetarchive.bak> [IA.BAK] db48x opened pull request #4: prefer downloading all files in an item together (master...items-together) http://git.io/jAQd
[18:59] *** patricko- is now known as patrickod
[19:02] <GitHub193/#internetarchive.bak> [IA.BAK] cancerAlot closed pull request #3: checkDeps should check if every tool is available (master...master) http://git.io/jF3E
[19:02] *** bzc6p__ (~bzc6p@[redacted]) has joined #internetarchive.bak
[19:05] <SketchCow> I haven't responded to DHSR's post on how IA.BAK is pretty impossible because too busy doing IA.BAK
[19:05] <SketchCow> It's a tricky layer, just like encrypted is a tricky layer.
[19:05] *** SketchCow drop mic
[19:05] *** bzc6p_ has quit (Read error: Operation timed out)
[19:16] *** wtron (4df54bfa@[redacted]) has joined #internetarchive.bak
[19:16] *** wtron has quit (Connection closed)
[19:17] <yipdw> buy my new book, Throwin' Shade With git-annex
[19:27] *** patrickod is now known as patricko-
[19:31] <GitHub186/#internetarchive.bak> IA.BAK/master c5306f5 Joey Hess: Merge pull request #4 from db48x/items-together...
[19:31] <GitHub186/#internetarchive.bak> IA.BAK/master d3539eb Daniel Brooks: prefer downloading all files in an item together...
[19:31] <GitHub69/#internetarchive.bak> [IA.BAK] joeyh closed pull request #4: prefer downloading all files in an item together (master...items-together) http://git.io/jAQd
[19:31] <GitHub186/#internetarchive.bak> [IA.BAK] joeyh pushed 2 new commits to master: http://git.io/jxIL
[19:35] <GitHub198/#internetarchive.bak> IA.BAK/pubkey 96bfe8f Joey Hess: zottelbey's key
[19:35] <GitHub198/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to pubkey: http://git.io/jxLU
[19:50] *** patricko- is now known as patrickod
[19:51] <db48x> oh yea, a 2GB blob of Ladies' Home Journal
[19:53] *** patrickod is now known as patricko-
[20:03] <GitHub8/#internetarchive.bak> [IA.BAK] cancerAlot opened pull request #5: don't reinstall git-annex everytime (master...updateLogic) http://git.io/jxWC
[20:05] <GitHub103/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to master: http://git.io/jxll
[20:05] <GitHub103/#internetarchive.bak> IA.BAK/master aa29021 Joey Hess: last pull request introduced a bug
[20:08] <db48x> closure: what bug does that fix?
[20:09] <closure> missing - in there
[20:09] <closure> hater: I'm using tabs for indentation in the script, could you update your PR?
[20:10] <db48x> yes, I see the change, but what is that going to do?
[20:10] <hater> closure: damn, i configured vi to replace tabs with whitespaces
[20:10] *** acridAxid has quit (Read error: Operation timed out)
[20:11] <db48x> dirname - foo/bar gives two answers; "." and "foo"
[20:11] <db48x> did you mean dirname --?
[20:12] *** acridAxid (~acridAxid@[redacted]) has joined #internetarchive.bak
[20:12] *** svchfoo2 gives channel operator status to acridAxid
[20:13] <closure> well zottelbey said xargs needed a - to read stdin. Mine doesn't, but I guess his does
[20:13] <db48x> dirname isn't reading stdin here
[20:13] <db48x> xargs is reading stdin and feeding it to dirname on the command line
[20:13] <db48x> echo foo/bar bar/baz | xargs dirname
[20:14] <zottelbey> i got an error dirname: missing operand
[20:14] <closure> oic, you mean the option is misplaced
[20:14] <db48x> zottelbey: what does echo foo/bar bar/baz | xargs -t dirname print out?
[20:15] <zottelbey> dirname foo/bar bar/baz
[20:15] <zottelbey> foo
[20:15] <zottelbey> bar
[20:15] <db48x> that's what I expect...
[20:15] <zottelbey> hm.
[20:15] *** underscor has quit (Ping timeout: 370 seconds)
[20:16] <db48x> does your dirname have any options besides --zero, --help and --version?
[20:16] <zottelbey> nope.
[20:16] <db48x> weird
[20:16] *** underscor (~quassel@[redacted]) has joined #internetarchive.bak
[20:16] <db48x> what version of dirname do you have?
[20:17] <zottelbey> 8.23
[20:17] <zottelbey> this is the whole printout for iabak. doesnt seem to be working for me at all btw: http://pastebin.com/eV3TMien
[20:18] <GitHub21/#internetarchive.bak> IA.BAK/master 4812624 Joey Hess: ok, so that fix was wrong
[20:18] <GitHub21/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to master: http://git.io/jxEl
[20:18] <db48x> hmm, ok
[20:18] <hater> closure: https://github.com/cancerAlot/IA.BAK/commit/bcfdefc3b8bee2d8a4d74899281e430c96aaf521
[20:18] <db48x> zottelbey: can you modify your iabak-helper script so that the first line ends in -x?
[20:18] <db48x> then when you rerun it it'll print out everything it does
[20:19] <closure> so, I get "missing operand" when dirname is passed no parameter at all
[20:19] <zottelbey> sure.
[20:19] <db48x> aha: http://lingrok.org/xref/coreutils/src/dirname.c#116
[20:19] <closure> so, xargs is running it on empty input
[20:19] *** SketchCow has quit (Ping timeout: 255 seconds)
[20:19] <closure> that can be fixed. But.. why is the input empty
[20:19] <zottelbey> db48x, http://pastebin.com/jVe8dhML
[20:20] <db48x> xargs -r, yea
[20:21] <closure> zottelbey: does this find any files? git annex find --not --copies 4
[20:21] <GitHub43/#internetarchive.bak> IA.BAK/master 3ad019c Joey Hess: pass --no-run-if-empty to xargs
[20:21] <GitHub43/#internetarchive.bak> [IA.BAK] joeyh pushed 1 new commit to master: http://git.io/jxu2
[20:21] <zottelbey> nope. no output at all
[20:22] <closure> well, it finds stuff in my shard, so we're not actually done with the backup :)
[20:22] <zottelbey> i would have doubted that in the first place
[20:22] <closure> do you have a git-annex branch in the shard1 repo?
[20:23] <closure> and you ran that in shard1, right?
[20:23] <zottelbey> yes i did and yes, the folder shard1/.git/annex exists.
[20:23] <zottelbey> if that is what you are asking
[20:23] <closure> what does this say: git annex whereis internetarchivebooks/animalparasiteshe2chan/animalparasiteshe2chan_dc.xml
[20:24] <closure> no, I mean does git branch command in shard1 show a git-annex branch
[20:24] <zottelbey> yes it does
[20:25] <zottelbey> "git-annex: internetarchivebooks/animalparasiteshe2chan/animalparasiteshe2chan_dc.xml not found"
[20:25] <GitHub198/#internetarchive.bak> [IA.BAK] db48x opened pull request #6: don't let filenames get interpreted as command-line args (master...protect-against-filenames) http://git.io/jxzM
[20:25] <closure> ok, that seems to be the problem..
[20:25] <closure> what filesystem and OS is this repo on?
[20:25] <zottelbey> ext4 on kubuntu
[20:25] <closure> db48x: fwiw, git-annex does guard all pathnames to avoid args. I use the ./$file method
[20:26] <closure> which works passing files to any program at all
[20:26] <closure> zottelbey: does the shard1 repository have any branch checked out?
[20:27] *** SketchCow (~jscott@[redacted]) has joined #internetarchive.bak
[20:27] *** svchfoo1 gives channel operator status to SketchCow
[20:28] <db48x> closure: hrm:
[20:28] <db48x> internetarchivebooks/1910catholicencyclop07herb/1910catholicencyclop07herb_archive.torrent
[20:28] <db48x>   anglachel  db48x  ~  projects  IA.BAK  shard1  git annex find --not --copies 4 | head
[20:28] <db48x> internetarchivebooks/1910catholicencyclop07herb/1910catholicencyclop07herb_dc.xml
[20:28] <closure> db48x: your pull request should also pass -- to git-annex get
[20:28] <closure> that's what I see too with the command
[20:29] <db48x> pushed
[20:35] <closure> db48x: can you merge stuff on github? I'm out of time and BW
[20:35] <db48x> sure
[20:35] <closure> I think both these pull reqs are good
[20:36] <GitHub197/#internetarchive.bak> IA.BAK/master 2467688 Daniel Brooks: don't let filenames get interpreted as command-line args...
[20:36] <GitHub197/#internetarchive.bak> IA.BAK/master 4efee24 Daniel Brooks: Merge pull request #6 from db48x/protect-against-filenames...
[20:36] <GitHub197/#internetarchive.bak> IA.BAK/master 576344b Daniel Brooks: also protect git annex get against filenames that look like args
[20:36] <GitHub118/#internetarchive.bak> [IA.BAK] db48x closed pull request #6: don't let filenames get interpreted as command-line args (master...protect-against-filenames) http://git.io/jxzM
[20:36] <GitHub197/#internetarchive.bak> [IA.BAK] db48x pushed 3 new commits to master: http://git.io/jxwS
[20:37] <zottelbey> ill get back to this tomorrow. have to leave now.
[20:38] <GitHub183/#internetarchive.bak> [IA.BAK] db48x closed pull request #5: don't reinstall git-annex everytime (master...updateLogic) http://git.io/jxWC
[20:38] <GitHub104/#internetarchive.bak> IA.BAK/master 56ef85f Daniel Brooks: Merge pull request #5 from cancerAlot/updateLogic...
[20:38] <GitHub104/#internetarchive.bak> IA.BAK/master 8fc19b2 cancerAlot: don't reinstall git-annex everytime
[20:38] <GitHub104/#internetarchive.bak> IA.BAK/master bcfdefc cancerAlot: now with tabs
[20:38] <GitHub104/#internetarchive.bak> [IA.BAK] db48x pushed 3 new commits to master: http://git.io/jxrz
[20:38] <db48x> zottelbey: thanks!
[20:42] *** zottelbey has quit (Remote host closed the connection)
[20:50] *** bzc6p__ is now known as bzc6p
[21:29] *** londoncal (~londoncal@[redacted]) has joined #internetarchive.bak
[21:55] <closure> http://git-annex.branchable.com/devblog/day_270__distributed_fsck/
[22:11] *** hatsefla1 has quit (Read error: Operation timed out)
[22:52] <GitHub122/#internetarchive.bak> [IA.BAK] cancerAlot opened pull request #8: install the right arch of git-annex (master...master) http://git.io/jpBr
[23:26] *** Kenshin has quit (Ping timeout: 246 seconds)
[23:34] *** Kenshin (~rurouni@[redacted]) has joined #internetarchive.bak
[23:34] *** svchfoo3 gives channel operator status to Kenshin
[23:48] *** patricko- is now known as patrickod
[23:49] <tpw_rules> closure: is there a way for git-annex to show the total size of everything annexed
[23:49] <tpw_rules> anyway, i've downloaded 347 gigs so far
[23:51] *** patrickod is now known as patricko-
[23:53] <SketchCow> closure: There's been no change in http://iabak.archiveteam.org/stats/SHARD1
[23:53] <SketchCow> in something like 9 hours.
[23:57] <tpw_rules> closure: is there a way to rate-limit all the running git-annexes?