#internetarchive.bak 2015-05-17,Sun

↑back Search

Time Nickname Message
00:00 πŸ”— db48x when you check files X, Y and Z into a git repository, git creates a tree object which lists the files, their modes, and the hash of their content
00:00 πŸ”— db48x however, this is recursive, so files in a directory get their own tree objects
00:01 πŸ”— tpw_rules i think it's more the logistics of a git repo with 100000 files
00:01 πŸ”— tpw_rules and using that as the primary data structure
00:01 πŸ”— db48x yes
00:01 πŸ”— Senji It's not as if git-annex is slow
00:02 πŸ”— db48x larger git repositories are generally slower
00:02 πŸ”— tpw_rules the other problem is i'm not exactly using fask disks
00:03 πŸ”— db48x shards only have approximately 100k files because putting all 271 million files in a single git repository would be insane indeed
00:04 πŸ”— Senji Now that would be slow
00:04 πŸ”— tpw_rules i'm attaching all the storage i have lying around and fiddling around with union filesystems
00:04 πŸ”— tpw_rules aufs seems to be doing okayish
00:05 πŸ”— Senji I have a number of little shardlets on disks of their own
00:05 πŸ”— db48x it's probably pretty rare to have that many free inodes sitting around as well
00:05 πŸ”— Senji And an IDE dock
00:06 πŸ”— tpw_rules Senji: how do you split a shard across two disks so you don't double-download anything?
00:06 πŸ”— tpw_rules the real question is that i have no clue what comes after /dev/sdz
00:06 πŸ”— Senji tpw_rules: downloaded it first and split it later
00:07 πŸ”— Senji db48x: jdamery-iabak@cleopatra:/mnt/daled/ia/IA.BAK/shard2 has 385M free inodes
00:07 πŸ”— pikhq tpw_rules: Looks like what comes after /dev/sdz is /dev/sdaa.
00:07 πŸ”— Senji (but only 639G free space)
00:08 πŸ”— pikhq So, it's basically base-26 numbered. Simple enough.
00:08 πŸ”— tpw_rules aww i didn't want you to spoil it
00:08 πŸ”— pikhq Sorry.
00:08 πŸ”— tpw_rules Senji how did you split it?
00:09 πŸ”— db48x Senji: fair enough; I only have 60M
00:09 πŸ”— db48x plus, you need two per file once the repository fills up :)
00:09 πŸ”— Senji db48x: scary thing; my personal backup disk is using 170M inides
00:10 πŸ”— Senji tpw_rules: guddle around in .git/annex/objects :)
00:10 πŸ”— tpw_rules ah okay
00:10 πŸ”— tpw_rules so there's no like git annex cut yet
00:10 πŸ”— db48x tpw_rules: http://git-annex.branchable.com/design/balanced_preferred_content/
00:10 πŸ”— db48x that'd let you cut a repository in have
00:11 πŸ”— db48x add both repositories to a group, and make it want one copy out of all the repositories in that grounp
00:11 πŸ”— db48x group
00:14 πŸ”— tpw_rules hmm. that looks a little complex
00:29 πŸ”— db48x no more so than the rest of git annex :)
00:33 πŸ”— tpw_rules well you put in a nice script for me :P
00:33 πŸ”— tpw_rules does fscking only start when all the shards are downloaded?
00:37 πŸ”— primus has quit IRC (Read error: Operation timed out)
00:37 πŸ”— primus104 has quit IRC (Read error: Operation timed out)
00:38 πŸ”— primus104 has joined #internetarchive.bak
00:39 πŸ”— primus has joined #internetarchive.bak
00:49 πŸ”— VADemon has quit IRC (Read error: Connection reset by peer)
00:51 πŸ”— primus105 has joined #internetarchive.bak
00:57 πŸ”— primus104 has quit IRC (Read error: Operation timed out)
01:00 πŸ”— primus105 has quit IRC (Read error: Operation timed out)
01:01 πŸ”— primus104 has joined #internetarchive.bak
01:15 πŸ”— garyrh has joined #internetarchive.bak
01:59 πŸ”— primus104 has quit IRC (Leaving.)
02:29 πŸ”— db48x tpw_rules: it depends on what mode the shard is in
02:58 πŸ”— tpw_rules it seems to be skipping the fsck of shard 1
03:36 πŸ”— tpw_rules i'd like to force that to run to make sure it's still good
03:38 πŸ”— tpw_rules i think somehow the shards got out of order so it's working on 5 first
05:26 πŸ”— db48x well, all it does is grab a list of them, then run handleshard in each one, sequentially
05:26 πŸ”— db48x it doesn't make any effort to do them in any particular order, it just does them in whatever order your filesystem coughs them up
05:27 πŸ”— db48x usually that's the order of creation, but I don't think filesystems make any gaurantees about the order in which they return directory entries
06:46 πŸ”— chazchaz has quit IRC (Ping timeout: 369 seconds)
06:50 πŸ”— chazchaz has joined #internetarchive.bak
07:57 πŸ”— primus104 has joined #internetarchive.bak
08:29 πŸ”— primus104 has quit IRC (Leaving.)
09:14 πŸ”— iabak-reg 03registrar 05master 48ecb76 06other 10SHARD4/pubkeys registration of db48x iabak on SHARD4
10:04 πŸ”— db48x hmm
10:04 πŸ”— db48x Timeout, server iabak.archiveteam.org not responding.236.00 KiB/s
10:04 πŸ”— db48x fatal: The remote end hung up unexpectedly
10:04 πŸ”— db48x seconds after registering; weird coincidence
10:05 πŸ”— db48x well, dunno about seconds
10:05 πŸ”— db48x at some point before it finished the initial clone
10:05 πŸ”— db48x anyway, time to sleep
11:18 πŸ”— primus104 has joined #internetarchive.bak
11:40 πŸ”— Senji db48x: install-fsck-service will try and install a systemd service even if there's a crontab line
12:25 πŸ”— ohhdemgir has quit IRC (Ping timeout: 512 seconds)
12:55 πŸ”— primus104 has quit IRC (Leaving.)
13:00 πŸ”— ohhdemgir has joined #internetarchive.bak
14:04 πŸ”— protodev has quit IRC (Ping timeout: 606 seconds)
14:08 πŸ”— protodev has joined #internetarchive.bak
14:08 πŸ”— GLaDOS has quit IRC (Excess Flood)
14:09 πŸ”— GLaDOS has joined #internetarchive.bak
14:09 πŸ”— svchfoo3 sets mode: +o GLaDOS
16:32 πŸ”— iabak-reg 03registrar 05master 8782199 06other 10SHARD1/pubkeys registration of twatson52 on SHARD1
16:32 πŸ”— iabak-reg 03registrar 05master c16516a 06other 10SHARD5/pubkeys registration of twatson52 on SHARD5
16:32 πŸ”— tpw_rules sorry, breaking everything
16:32 πŸ”— iabak-reg 03registrar 05master 9eb16e5 06other 10SHARD6/pubkeys registration of twatson52 on SHARD6
16:32 πŸ”— primus104 has joined #internetarchive.bak
16:39 πŸ”— tpw_rules i ruined everything again
16:42 πŸ”— iabak-reg 03registrar 05master ee3df6b 06other 10SHARD1/pubkeys registration of twatson52 on SHARD1
16:43 πŸ”— tpw_rules Control socket connect(.git/annex/ssh/SHARD1@iabak.archiveteam.org): Connection refused i keep getting
16:47 πŸ”— iabak-reg 03registrar 05master c6f6fc5 06other 10SHARD4/pubkeys registration of twatson52 on SHARD4
16:48 πŸ”— tpw_rules sorry
16:50 πŸ”— iabak-reg 03registrar 05master b779ea5 06other 10SHARD5/pubkeys registration of twatson52 on SHARD5
16:51 πŸ”— iabak-reg 03registrar 05master 4881f9e 06other 10SHARD6/pubkeys registration of twatson52 on SHARD6
16:51 πŸ”— tpw_rules that should be the last one until i catch everything on fire again
16:57 πŸ”— tpw_rules syncing is just not happening. Control socket connect(.git/annex/ssh/SHARD1@iabak.archiveteam.org): Connection refused
16:57 πŸ”— tpw_rules i deleted everything, copied the objects folder from the old one, and i still can't sync
16:59 πŸ”— tpw_rules does it create any kind of sockets on the filesystem or something?
17:15 πŸ”— tpw_rules oh ok, i did a "git config annex.sshcaching false" and that fixed it because i'm on a weird filesystem
19:18 πŸ”— primus104 has quit IRC (Leaving.)
19:51 πŸ”— primus104 has joined #internetarchive.bak
20:07 πŸ”— tpw_rules oops i already broke a hard drive
20:12 πŸ”— tpw_rules i somehow got a 'data phase error'
20:12 πŸ”— tpw_rules not a clue what that means
20:32 πŸ”— db48x fun :)
20:32 πŸ”— db48x the net says http://www.spinics.net/lists/usb/msg01782.html
20:33 πŸ”— db48x Senji: hmm, I suppose that's true. why would you want to use a crontab line when you have systemd though?
20:35 πŸ”— tpw_rules well it's a sata adapter so *shrug*
20:35 πŸ”— tpw_rules i think i overheated it
20:35 πŸ”— tpw_rules also fsck resume and parallelization doesn't seem to be working right
20:57 πŸ”— db48x how so?
20:59 πŸ”— db48x oops
21:01 πŸ”— db48x Β ξ‚’Β anglachelΒ ξ‚°Β db48xΒ ξ‚°Β ~Β ξ‚±Β projectsΒ ξ‚±Β IA.BAKΒ ξ‚±Β shard4Β ξ‚°Β du -chsL 2>&1 | sort -h
21:01 πŸ”— db48x sort: write failed: /tmp/sortfSTPMa: No space left on device
21:03 πŸ”— db48x it completely ignored my diskreserve
21:11 πŸ”— tpw_rules db48x: two fscks checksum the same files
21:11 πŸ”— tpw_rules i think
21:14 πŸ”— tpw_rules i'll let it complete a full one and see what happens
21:15 πŸ”— tpw_rules ugh no this drive is broken. next time i can dick around with it is wednesday
21:16 πŸ”— tpw_rules or it at least need to be zeroed
21:19 πŸ”— tpw_rules but that might mean i'll have to reregister all the shards because i r dum
21:39 πŸ”— db48x reregistering is fine :)
21:41 πŸ”— Senji db48x: I don't have systemd; I have enough bits of systemd to make other things work, which apparently includes the executable that you're checking for :)
21:42 πŸ”— Senji I have no idea if a systemd user service will work or not
21:44 πŸ”— Senji db48x: there's at least one bug with diskreserve that closure has fixed in his development version
21:44 πŸ”— Senji (of git annex)
22:05 πŸ”— db48x Senji: hmm. what's your PID 1?
22:24 πŸ”— Senji sysvinit
22:25 πŸ”— balrog Senji: not a symlink to systemd?
22:25 πŸ”— balrog ;)
22:30 πŸ”— Senji no
22:39 πŸ”— db48x huh
22:40 πŸ”— db48x how does having systemctl make other things work?
22:40 πŸ”— db48x and what does it actually do?
22:40 πŸ”— db48x do you have /bin/systemd as well?
22:42 πŸ”— Senji db48x: debian packages systemd in fairly large lumps I think.
22:42 πŸ”— Senji I do have /bin/systemd
22:48 πŸ”— trs80 Senji: what does "dpkg -l sysvinit systemd-sysv" say?
22:52 πŸ”— Senji un systemd-sysv <none> <none> (no description available)
22:52 πŸ”— Senji ii sysvinit 2.88dsf-59 amd64 System-V-like init utilities - tr
23:11 πŸ”— db48x so you have both installed, but you're not using systemd at all
23:11 πŸ”— db48x except to confuse programs that try to do the right thing in both init systems
23:15 πŸ”— Senji Yes, because things depend on it
23:19 πŸ”— ohhdemgir has quit IRC (Quit: Leaving)
23:19 πŸ”— Senji Looks as if the fundamental thing pulling systemd in is xfce4's session manager
23:24 πŸ”— ryang has quit IRC (Connection closed)
23:25 πŸ”— db48x does xcfe4's session manager work if you have systemd but aren't actually using it?
23:26 πŸ”— Senji I have no idea, I don't use xfce myself; but I've not had any commplaints :)
23:26 πŸ”— Senji Various bits of gnome would pull it in too
23:26 πŸ”— db48x heh
23:26 πŸ”— Senji But I don't do gnome
23:27 πŸ”— ryang has joined #internetarchive.bak
23:41 πŸ”— Senji Apparently the official way to check for running systemd is the existence of /run/systemd/system
23:41 πŸ”— Senji I don't ahve a systemd system to try that on :)
23:47 πŸ”— db48x that's easy enough to check for
23:47 πŸ”— db48x do you have documentation though?
23:50 πŸ”— Senji Only stackexchange and reddit answers
irclogger-viewer