[01:14] 03registrar 05master f613933 06other 10SHARD23/pubkeys registration of hcross on SHARD23 [03:25] 03registrar 05master a812df2 06other 10SHARD16/pubkeys registration of hcross on SHARD16 [03:34] *** kyan has quit IRC (Quit: Leaving) [03:34] *** kyan has joined #internetarchive.bak [04:51] 03registrar 05master 1388fb8 06other 10SHARD21/pubkeys registration of hcross on SHARD21 [04:58] 03registrar 05master e82c691 06other 10SHARD18/pubkeys registration of hcross on SHARD18 [07:21] *** svchfoo3 has quit IRC (Remote host closed the connection) [07:23] *** svchfoo3 has joined #internetarchive.bak [07:24] *** svchfoo1 sets mode: +o svchfoo3 [08:04] 03registrar 05master 7d03fe6 06other 10SHARD14/pubkeys registration of hcross on SHARD14 [08:22] 03registrar 05master 98dde50 06other 10SHARD20/pubkeys registration of hcross on SHARD20 [08:29] *** atomotic has joined #internetarchive.bak [09:32] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [09:39] 03registrar 05master ce230bf 06other 10SHARD15/pubkeys registration of hcross on SHARD15 [10:17] *** atomotic has joined #internetarchive.bak [11:28] 03registrar 05master bcce85f 06other 10SHARD10/pubkeys registration of hcross on SHARD10 [11:40] 03registrar 05master d4646f5 06other 10SHARD6/pubkeys registration of hcross on SHARD6 [12:59] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [14:37] *** atomotic has joined #internetarchive.bak [15:09] *** jmeyer has joined #internetarchive.bak [15:26] hey, all. I've been running the client for a couple months now, and have now received my second "expiry warning" email, despite client running continuously. Any thoughts? Does the client only "check in" at startup? [15:27] When I force-stop the client and restart it, the webpage status updates within a day, but I get another warning, weeks later. [15:28] *** db48x has quit IRC (Read error: Operation timed out) [15:29] *** db48x has joined #internetarchive.bak [15:30] don't know the answer jmeyer but I basically have the same problem(s) [15:31] I have parts of two shards but I think somehow they're registered to different UUIDs [15:31] and running ./iabak only ever updates one of them [15:32] Do you have the cronjob/systemd-thing installed? (It should have installed automatically, but...) [15:32] I think I've heard of the cronjob not running if the downloader is going too [15:38] Jon: I also have parts of two shards (don't know if relevant), but if running ./iabak only ever updates one, shouldn't that mean the other has reached 1+3 copies? [15:41] Senji: I believe so, at least when I restart it tells me "You've already got the iabak-cronjob installed. I didn't have to do anything." or some such, so maybe the latter case you mentioned is happening. [15:42] jmeyer: "checking in" with the server is just a matter of running "git annex sync" every now and then. both the client and the cronjob do this, or you can do it manually [15:42] jmeyer: check your logs to see if the cron job (or systemd service, as appropriate for your system) has actually been running [15:43] Jon: run "git annex info" in a shard to see what UUID it is [15:43] db48x: the client will only sync on the shard it's working on though, won't it? [15:43] Senji: yes [15:43] well [15:43] when it starts up it syncs every shard it can find, then as it works on a shard it periodically syncs it [15:47] Senji: in the short term, the email should have identified a shard repository by UUID. you can just run "git annex sync" in that repository to update it while you figure out what the problem is [15:50] db48x: will do. thanks [15:51] you're welcome [15:55] db48x: thanks for the hint. that failed wth (No space left on device) [15:55] at least for one of my shards. it's working for another. I seem to have 3 now. [15:56] my layout is to have separate filesystems for ~iabak (within which the git clone, local git-annex copy etc.) -- not full; ~iabak/shard3 (full); ~iabak/shard4 (not full) [15:56] this is forced upon me somewhat by my local HDD arrangement, I'm dedicating 1T from one drive and 1T from another [15:57] wow git annex info takes a while to run. still waiting for 'local annex keys:' [15:57] If the disk is full then sync won't work; it needs about 0.5-2GB to run depending on the shard [15:58] why has it filled the disk up then? shouldn't it limit itself to available space? [15:58] or alternatively, can I get it to use another location for its working directory? [15:59] 'git annex status' in my third shard exits with no output and a success status code (filesystem is not full) [15:59] There is a configurable for the amount of disk space to leave empty at the end. I thought it was set to a sensible value by default but I don't know. [15:59] status tells you whether you have any changes to commit [15:59] Senji: that's only used by iabak, not by git-annex directly [16:00] or rather, it's only used by git-annex when downloading files (such as when iabak is running) [16:00] if you don't leave room for git metadata, it can grow through that extra space [16:00] db48x: yes, but git-annex directly won't fill up the disk except for neeing a bit of space to make sync work [16:01] I imagine if I zapped all this I could start over and get something non-broken, but I'd rather not throw away the 1.1T I've already got (looking , seems I have 1T of shard3 = full filesystem, and only 200G of shard4 which is seems to never try to fetch more for) [16:02] Jon: to free up some space, just use 'git annex drop' on a few things [16:02] OK, thanks [16:04] Jon: can the client even automatically use/ manage multiple hardrives/ separate filesystems? [16:04] jmeyer: honestly I don't know. I figured if it was fetching multiple shards, it would "just work" if the shard subdirs were on separate filesystems, but idk [16:04] I guess I could run completely separate instances on each filesystem. not sure I could share a UUID in that situation [16:04] Jon: or do you mean you have 2*1TB married JBOD? [16:05] Jon: you could also run 'git annex find --copies=5' to find files that we have a lot of copies of, and drop those [16:05] I have apparently 32 different shards; mostly one per fileystem across three computers. [16:07] hmm git-annex drop won't function either. but movign/symlinking .git/annex/misctmp to ~ (with space) seems to help [16:07] jmeyer: no they are distinct. I'd rather avoid JBODding them since that increases likelyhood of failure [16:07] db48x: that's a cool idae [16:08] .git/annex/misctmp is usually empty [16:08] "Warning: This repository is currently marked as dead." aha that's useful to know [16:08] I think that's where it stores incomplete downloads [16:08] db48x: it seems to be where it puts working files to run 'git annex drop' , 'git annex info' etc. -- both failed until I symlinked it to a FS with space [16:08] ah, I see [16:09] OK so I know now that my shard3 copy (1T) is marked dead. [16:09] Jon: run 'git annex semitrust ' to mark it as non-dead [16:09] then fsk it [16:09] fsck [16:09] OK [16:10] and then sync it, of course [16:10] ah slight problem. my UUID is not output in the initial part of the 'git annex info' output, unless it's the UUID with no descriptive suffix (for my shard4, it was clearly labelled iabak@phobos:~/IA.BAK/shard4 [here]) [16:12] oh git config -l shows it [16:13] you beat me to it :) [16:14] Jon: btw, is this you? http://iabak.archiveteam.org/client/eb34830fd1da871513693e2e6c9ca4a09e69babc.html [16:16] db48x: no, that's me [16:17] ah :) [16:17] You know something when you recognise your own UUID... :) [16:17] heh [16:18] Senji: a lot of yours need syncing as well [16:19] Yeah, the hardware hooking the drives up to oklina is a bit ropey and occasionally needs poking. [16:20] as long as we're all here, I'd like to sync some of our repositories directly, so that we can catch up on some files that have gone dark [16:20] Jon, Senji, jmeyer: can we exchange ssh access? [16:23] my ssh key is ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJQkqIgZ7D8WHW5Y3o+fpZC/4xtv/3IQrORJrTPCt7KY db48x@erebor [16:24] Hmm, most of mine are behind NAT (and my bandwidth is metered except for a 4 hour window overnight, which might present a problem) :) [16:24] hmm [16:24] Senji: well, I could give you access to my machine, and you could push files to it [16:25] Oh, and cleopatra's sshd doesn't support ed25519 :-D [16:25] I could do that [16:26] bah, one sec [16:27] ssh-rsa [16:27] AAAAB3NzaC1yc2EAAAADAQABAAAEAQC7/PISfISdeRw/cVWgszHu9Cy/Pz+T8r6zQYqZaDqkRTp4qLOHXbcwYVgNEXGQSq7f97qNQC2gv1Mae0JfArocmIrFcN5vlLoKovaemvBjk86tvDKwlcI5ZO3neWWBKN9dwdju1KvsAO/mwOa0QxnZjKz47CCOVKls1SnjdydEMmFLTMoArb8Qn/eAxv3wCHGFNpqYU4ErZftVfE8jzIb2YzX67zWfymEI0niuBNgvJ/lylCq0e8/Vdf5w2fHLxmBSwc637FdZZxE4pqEgnA2Qny73IBL3+U4ugETsdONYtsE/QiQotMcRosLHHU863kYCXTezVd10nhzUx3xrPyXV1Rir9jzqOkn10sAHMT0/0e/x3uDBrgLa/tI3eUgPerSVvnnWy4v [16:27] db48x@erebor [16:30] That key should now have access to jdamery-iabak@cleopatra.thy.me.uk (which machine has passphraseless login to the other two) [16:31] send me your ssh key as well [16:32] I've got to run, but I'll take a look later [16:33] Jon: ok :) [16:33] ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAAut2lpYEg71g+dZaONUJxNFJ5mEaAx33aq5u9TiCCBPuorZo9L/Qiwe187txKExolkviVCJipTADNIPTBK05akRQAqTqqG80WodO7Ul5lSsrz0C4Acvn56jMRkij0a34b2BELhjFyKYKSiHg9n9vwkOJ/yRaLWTNUJuej9lxM2UFvgnQ== jdamery-iabak@cleopatra [16:33] if you're happy with dodgy government elpitic curves :) [16:34] for this purpose I think it will be fine [16:36] Senji: ok, you should be able to log in as senji@erebor.db48x.net [16:37] It's not accepting that key [16:39] what's your ip address? [16:40] cleopatra.thy.me.uk has address 81.187.132.36 [16:40] cleopatra.thy.me.uk has IPv6 address 2001:8b0:3b9:1:20a:cdff:fe22:8a5d [16:41] curious... [16:42] oh, right! port 23 :) [16:43] "no hostkey alg" [16:43] I've never seen that error before [16:44] it means you have a super-old ssh, and I've turned off the super-old key types [16:45] Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u2 [16:45] I knew I had an old ssh though, see the comments about ed25519 :) [16:46] 6.0, sheesh [16:47] db48x: what would git-annex key exchange help with? [16:48] db48x: yeah, upgrades are on my todo list :( [16:48] I hate todo lists [16:48] Senji: ok, I told it to advertise an ecdsa key, which your ssh client should accept (in addition to an ed25519 key) [16:49] jmeyer: files on IA occasionally go dark, and are no longer available [16:49] jmeyer: which means that git-annex cannot download them [16:49] jmeyer: if we allow each other access to our machines via ssh, we can tell git annex to download files directly from each other [16:50] jmeyer: we would use this to make additional copies of files not currently available from the web, as needed [16:50] Now it's managing host key acceptance; but your server is still not accepting my ecdsa key [16:50] hmm [16:52] db48x: ah ,gotcha. while this would be cool, i'm still a newb to SSH to be fully confident in my security skills :-/ if I can spend some time in the next week or two square some things away on this machine, I'll msg you back....but no promises with 2 kids. [16:52] jmeyer: fair enough :) [16:54] Senji: try again? [16:55] No luck [16:55] Sorry if I'm a bit slow; I'm talking to tech support about a work problem at the same time :) [16:55] (work's systems don't even support ECDSA, so they're even older...) [16:56] db48x: also, thank you for the log pointer, it would appear when I moved my iabak storage pool, I borked the systemd timer. I though I had made the move cleanly. [16:56] jmeyer: you're welcome :) [16:57] been a gnu/linux user for 15 years...still a newbie in so many ways [16:57] Senji: ok, try it again. it doesn't really say why it's rejecting it, but I can confirm that you're presenting the key that I added to your authorized_keys file [16:59] I'm getting Connection refused now [16:59] Debugging these kinds of ssh authentication issues is a real pain; I've been on the other end of that :( [17:00] odd [17:00] daemon is running... [17:02] firewall? [17:03] the NAT is NATting [17:12] oh, right [17:13] https://hastebin.com/yowunupuna :) [17:15] ok, I unbanned you [17:15] aww [17:15] Still doesn't work [17:15] ! [17:15] Does get to presenting the key though [17:16] ok, I turned up the log level. could you try it again? [17:17] done [17:17] well: "Mar 09 09:17:01 erebor sshd[622]: debug1: Could not open authorized keys '/home/senji/.ssh/authorized_keys': Permission denied" [17:17] yea, selinux :) [17:17] ok, 7th time's the charm [17:18] no :( [17:20] heh, minor details: sudo chown senji.senji ~senji/.ssh/authorized_keys [17:20] not sure if that would break it or not [17:20] :-D [17:20] I've been locked out by systemd again [17:20] you are unbanned again [17:21] for some reason, fail2ban doesn't actually ignore your ip address [17:21] [senji@erebor ~]$ [17:21] success [17:21] Which is good because I have to go make dinner soon [17:21] aha! [17:21] cool. my shards are in ~db48x/archives/IA.BAK [17:22] and I'm starting to think about breakfast [17:23] you can use "git remote add db48x ..." in a shard, then "git annex copy --to=db48x --not --copies=4" or similar to copy files to mine [17:23] And you have lots of free disk space [17:24] some, yes [17:24] Saying that just fried my brain; because in the other conversation I'm talking about petabytes of data... [17:24] lol [17:24] I wish I had a petabyte [17:24] Professionally a terabyte is a small amount of data :) [17:24] We have about 4 petabytes on tapes [17:25] Sadly the costs of such storage is well beyond the IA [17:25] that's a lot of tapes [17:27] I'd better see if SWMBO is ready to do dinner yet. I'll look at pushing some data to you overnight my time (about 7 hours from now or so) [17:28] sounds good to me [17:49] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [21:58] *** jmeyer has quit IRC (Quit: http://chat.efnet.org (Session timeout)) [22:36] db48x: while syncing: [22:36] remote: error: insufficient permission for adding an object to repository database ./objects [22:36] remote: fatal: failed to write object [22:36] error: unpack failed: unpack-objects abnormal exit [22:37] Did you get the same "explanation"? [22:37] Bah, wrong channel. Sorry [22:37] ignore the previous, but not the previous previous :) [22:55] Senji: which shard? [22:57] To erebor.db48x.net:~db48x/archives/IA.BAK/shard6 [22:57] which just happened to be the first one I looked at [22:59] got it [22:59] some of the files/directories had the wrong group [22:59] fixed that, and made all the directories setgid again [23:01] I'll try sync again. It's rather slow :) [23:02] Hmm, same error again [23:03] http://pastebin.com/gPwgAMuz [23:07] oh, huh [23:07] it didn't even have a .git/annex/objects [23:13] must have been one that got created during testing, but where I never actually downloaded anything [23:13] Ahh [23:13] Yeah, you're not on the top-25 list [23:14] we should report that as a bug [23:15] closure: can't copy into a repository unless it's already downloaded something itself [23:16] ok, I downloaded one item in every shard that didn't have an objects directory :) [23:16] Hmm [23:17] sync worked now but [23:17] jdamery-iabak@cleopatra:/mnt/nasi/ia/IA.BAK/shard6$ PATH=../git-annex.linux/:$PATH git annex copy --to=db48x --not --copies=4 [23:17] git-annex: cannot determine uuid for db48x (perhaps you need to run "git annex sync"?) (remote.db48x.annex-ignore is set) [23:17] hmm [23:17] that's a new one [23:18] I can see your uuid in 'git annex info' [23:19] try setting remote.origin.annex-uuid to that value? [23:19] Oh, but your line in 'git annex info' doesn't have the remote name listed [23:20] hmm. it may be that git-annex-shell isn't in the path, if I'm reading this correctly [23:20] Right, I 'git annex remove' and then 'git annex add' [23:20] and it worked [23:21] so presumably that was fixed when you fixed one of the intermediate errors but had left legacy stuff [23:21] hrm [23:21] I'm now pushing a number of small files to you [23:23] I don't know if this is the best shard to be looking at if you don't have any files in it yet though [23:25] it's got a pretty good set of files that are still IA-only, so that's a good choice [23:25] If they're IA-only this isn't going to help :) [23:25] It'll only help for files that are 1-copy [23:26] true :) [23:27] there's a decent chunk of those as well [23:28] you might try -J 5 or something [23:28] yeah. [23:30] Hmm, I think I've managed to get myself firewalled out again with that. [23:32] I think I want --fast; assuming you're not downloading on that shard at the time [23:34] yep [23:34] it should only ban you if you fail to log in several times in a row [23:37] Yes, it just suddenly said that it couldn't access [23:39] ok, I set it to ignore your ip [23:47] I think this might be an application for ControlMaster [23:47] but I'm not going to try setting that up right now [23:47] git annex sets that up for you [23:48] Hmm, then why is it apparently making so many connections. [23:48] At the moment I'm waiting on sync again [23:49] might have been faster just to not worry about re-transfering the files I'd already transfered [23:56] OK, that looks to be copying at a reasonable rate; and I'm off to bed