Time |
Nickname |
Message |
00:00
π
|
db48x |
when you check files X, Y and Z into a git repository, git creates a tree object which lists the files, their modes, and the hash of their content |
00:00
π
|
db48x |
however, this is recursive, so files in a directory get their own tree objects |
00:01
π
|
tpw_rules |
i think it's more the logistics of a git repo with 100000 files |
00:01
π
|
tpw_rules |
and using that as the primary data structure |
00:01
π
|
db48x |
yes |
00:01
π
|
Senji |
It's not as if git-annex is slow |
00:02
π
|
db48x |
larger git repositories are generally slower |
00:02
π
|
tpw_rules |
the other problem is i'm not exactly using fask disks |
00:03
π
|
db48x |
shards only have approximately 100k files because putting all 271 million files in a single git repository would be insane indeed |
00:04
π
|
Senji |
Now that would be slow |
00:04
π
|
tpw_rules |
i'm attaching all the storage i have lying around and fiddling around with union filesystems |
00:04
π
|
tpw_rules |
aufs seems to be doing okayish |
00:05
π
|
Senji |
I have a number of little shardlets on disks of their own |
00:05
π
|
db48x |
it's probably pretty rare to have that many free inodes sitting around as well |
00:05
π
|
Senji |
And an IDE dock |
00:06
π
|
tpw_rules |
Senji: how do you split a shard across two disks so you don't double-download anything? |
00:06
π
|
tpw_rules |
the real question is that i have no clue what comes after /dev/sdz |
00:06
π
|
Senji |
tpw_rules: downloaded it first and split it later |
00:07
π
|
Senji |
db48x: jdamery-iabak@cleopatra:/mnt/daled/ia/IA.BAK/shard2 has 385M free inodes |
00:07
π
|
pikhq |
tpw_rules: Looks like what comes after /dev/sdz is /dev/sdaa. |
00:07
π
|
Senji |
(but only 639G free space) |
00:08
π
|
pikhq |
So, it's basically base-26 numbered. Simple enough. |
00:08
π
|
tpw_rules |
aww i didn't want you to spoil it |
00:08
π
|
pikhq |
Sorry. |
00:08
π
|
tpw_rules |
Senji how did you split it? |
00:09
π
|
db48x |
Senji: fair enough; I only have 60M |
00:09
π
|
db48x |
plus, you need two per file once the repository fills up :) |
00:09
π
|
Senji |
db48x: scary thing; my personal backup disk is using 170M inides |
00:10
π
|
Senji |
tpw_rules: guddle around in .git/annex/objects :) |
00:10
π
|
tpw_rules |
ah okay |
00:10
π
|
tpw_rules |
so there's no like git annex cut yet |
00:10
π
|
db48x |
tpw_rules: http://git-annex.branchable.com/design/balanced_preferred_content/ |
00:10
π
|
db48x |
that'd let you cut a repository in have |
00:11
π
|
db48x |
add both repositories to a group, and make it want one copy out of all the repositories in that grounp |
00:11
π
|
db48x |
group |
00:14
π
|
tpw_rules |
hmm. that looks a little complex |
00:29
π
|
db48x |
no more so than the rest of git annex :) |
00:33
π
|
tpw_rules |
well you put in a nice script for me :P |
00:33
π
|
tpw_rules |
does fscking only start when all the shards are downloaded? |
00:37
π
|
|
primus has quit IRC (Read error: Operation timed out) |
00:37
π
|
|
primus104 has quit IRC (Read error: Operation timed out) |
00:38
π
|
|
primus104 has joined #internetarchive.bak |
00:39
π
|
|
primus has joined #internetarchive.bak |
00:49
π
|
|
VADemon has quit IRC (Read error: Connection reset by peer) |
00:51
π
|
|
primus105 has joined #internetarchive.bak |
00:57
π
|
|
primus104 has quit IRC (Read error: Operation timed out) |
01:00
π
|
|
primus105 has quit IRC (Read error: Operation timed out) |
01:01
π
|
|
primus104 has joined #internetarchive.bak |
01:15
π
|
|
garyrh has joined #internetarchive.bak |
01:59
π
|
|
primus104 has quit IRC (Leaving.) |
02:29
π
|
db48x |
tpw_rules: it depends on what mode the shard is in |
02:58
π
|
tpw_rules |
it seems to be skipping the fsck of shard 1 |
03:36
π
|
tpw_rules |
i'd like to force that to run to make sure it's still good |
03:38
π
|
tpw_rules |
i think somehow the shards got out of order so it's working on 5 first |
05:26
π
|
db48x |
well, all it does is grab a list of them, then run handleshard in each one, sequentially |
05:26
π
|
db48x |
it doesn't make any effort to do them in any particular order, it just does them in whatever order your filesystem coughs them up |
05:27
π
|
db48x |
usually that's the order of creation, but I don't think filesystems make any gaurantees about the order in which they return directory entries |
06:46
π
|
|
chazchaz has quit IRC (Ping timeout: 369 seconds) |
06:50
π
|
|
chazchaz has joined #internetarchive.bak |
07:57
π
|
|
primus104 has joined #internetarchive.bak |
08:29
π
|
|
primus104 has quit IRC (Leaving.) |
09:14
π
|
iabak-reg |
03registrar 05master 48ecb76 06other 10SHARD4/pubkeys registration of db48x iabak on SHARD4 |
10:04
π
|
db48x |
hmm |
10:04
π
|
db48x |
Timeout, server iabak.archiveteam.org not responding.236.00 KiB/s |
10:04
π
|
db48x |
fatal: The remote end hung up unexpectedly |
10:04
π
|
db48x |
seconds after registering; weird coincidence |
10:05
π
|
db48x |
well, dunno about seconds |
10:05
π
|
db48x |
at some point before it finished the initial clone |
10:05
π
|
db48x |
anyway, time to sleep |
11:18
π
|
|
primus104 has joined #internetarchive.bak |
11:40
π
|
Senji |
db48x: install-fsck-service will try and install a systemd service even if there's a crontab line |
12:25
π
|
|
ohhdemgir has quit IRC (Ping timeout: 512 seconds) |
12:55
π
|
|
primus104 has quit IRC (Leaving.) |
13:00
π
|
|
ohhdemgir has joined #internetarchive.bak |
14:04
π
|
|
protodev has quit IRC (Ping timeout: 606 seconds) |
14:08
π
|
|
protodev has joined #internetarchive.bak |
14:08
π
|
|
GLaDOS has quit IRC (Excess Flood) |
14:09
π
|
|
GLaDOS has joined #internetarchive.bak |
14:09
π
|
|
svchfoo3 sets mode: +o GLaDOS |
16:32
π
|
iabak-reg |
03registrar 05master 8782199 06other 10SHARD1/pubkeys registration of twatson52 on SHARD1 |
16:32
π
|
iabak-reg |
03registrar 05master c16516a 06other 10SHARD5/pubkeys registration of twatson52 on SHARD5 |
16:32
π
|
tpw_rules |
sorry, breaking everything |
16:32
π
|
iabak-reg |
03registrar 05master 9eb16e5 06other 10SHARD6/pubkeys registration of twatson52 on SHARD6 |
16:32
π
|
|
primus104 has joined #internetarchive.bak |
16:39
π
|
tpw_rules |
i ruined everything again |
16:42
π
|
iabak-reg |
03registrar 05master ee3df6b 06other 10SHARD1/pubkeys registration of twatson52 on SHARD1 |
16:43
π
|
tpw_rules |
Control socket connect(.git/annex/ssh/SHARD1@iabak.archiveteam.org): Connection refused i keep getting |
16:47
π
|
iabak-reg |
03registrar 05master c6f6fc5 06other 10SHARD4/pubkeys registration of twatson52 on SHARD4 |
16:48
π
|
tpw_rules |
sorry |
16:50
π
|
iabak-reg |
03registrar 05master b779ea5 06other 10SHARD5/pubkeys registration of twatson52 on SHARD5 |
16:51
π
|
iabak-reg |
03registrar 05master 4881f9e 06other 10SHARD6/pubkeys registration of twatson52 on SHARD6 |
16:51
π
|
tpw_rules |
that should be the last one until i catch everything on fire again |
16:57
π
|
tpw_rules |
syncing is just not happening. Control socket connect(.git/annex/ssh/SHARD1@iabak.archiveteam.org): Connection refused |
16:57
π
|
tpw_rules |
i deleted everything, copied the objects folder from the old one, and i still can't sync |
16:59
π
|
tpw_rules |
does it create any kind of sockets on the filesystem or something? |
17:15
π
|
tpw_rules |
oh ok, i did a "git config annex.sshcaching false" and that fixed it because i'm on a weird filesystem |
19:18
π
|
|
primus104 has quit IRC (Leaving.) |
19:51
π
|
|
primus104 has joined #internetarchive.bak |
20:07
π
|
tpw_rules |
oops i already broke a hard drive |
20:12
π
|
tpw_rules |
i somehow got a 'data phase error' |
20:12
π
|
tpw_rules |
not a clue what that means |
20:32
π
|
db48x |
fun :) |
20:32
π
|
db48x |
the net says http://www.spinics.net/lists/usb/msg01782.html |
20:33
π
|
db48x |
Senji: hmm, I suppose that's true. why would you want to use a crontab line when you have systemd though? |
20:35
π
|
tpw_rules |
well it's a sata adapter so *shrug* |
20:35
π
|
tpw_rules |
i think i overheated it |
20:35
π
|
tpw_rules |
also fsck resume and parallelization doesn't seem to be working right |
20:57
π
|
db48x |
how so? |
20:59
π
|
db48x |
oops |
21:01
π
|
db48x |
Β ξ’Β anglachelΒ ξ°Β db48xΒ ξ°Β ~Β ξ±Β projectsΒ ξ±Β IA.BAKΒ ξ±Β shard4Β ξ°Β du -chsL 2>&1 | sort -h |
21:01
π
|
db48x |
sort: write failed: /tmp/sortfSTPMa: No space left on device |
21:03
π
|
db48x |
it completely ignored my diskreserve |
21:11
π
|
tpw_rules |
db48x: two fscks checksum the same files |
21:11
π
|
tpw_rules |
i think |
21:14
π
|
tpw_rules |
i'll let it complete a full one and see what happens |
21:15
π
|
tpw_rules |
ugh no this drive is broken. next time i can dick around with it is wednesday |
21:16
π
|
tpw_rules |
or it at least need to be zeroed |
21:19
π
|
tpw_rules |
but that might mean i'll have to reregister all the shards because i r dum |
21:39
π
|
db48x |
reregistering is fine :) |
21:41
π
|
Senji |
db48x: I don't have systemd; I have enough bits of systemd to make other things work, which apparently includes the executable that you're checking for :) |
21:42
π
|
Senji |
I have no idea if a systemd user service will work or not |
21:44
π
|
Senji |
db48x: there's at least one bug with diskreserve that closure has fixed in his development version |
21:44
π
|
Senji |
(of git annex) |
22:05
π
|
db48x |
Senji: hmm. what's your PID 1? |
22:24
π
|
Senji |
sysvinit |
22:25
π
|
balrog |
Senji: not a symlink to systemd? |
22:25
π
|
balrog |
;) |
22:30
π
|
Senji |
no |
22:39
π
|
db48x |
huh |
22:40
π
|
db48x |
how does having systemctl make other things work? |
22:40
π
|
db48x |
and what does it actually do? |
22:40
π
|
db48x |
do you have /bin/systemd as well? |
22:42
π
|
Senji |
db48x: debian packages systemd in fairly large lumps I think. |
22:42
π
|
Senji |
I do have /bin/systemd |
22:48
π
|
trs80 |
Senji: what does "dpkg -l sysvinit systemd-sysv" say? |
22:52
π
|
Senji |
un systemd-sysv <none> <none> (no description available) |
22:52
π
|
Senji |
ii sysvinit 2.88dsf-59 amd64 System-V-like init utilities - tr |
23:11
π
|
db48x |
so you have both installed, but you're not using systemd at all |
23:11
π
|
db48x |
except to confuse programs that try to do the right thing in both init systems |
23:15
π
|
Senji |
Yes, because things depend on it |
23:19
π
|
|
ohhdemgir has quit IRC (Quit: Leaving) |
23:19
π
|
Senji |
Looks as if the fundamental thing pulling systemd in is xfce4's session manager |
23:24
π
|
|
ryang has quit IRC (Connection closed) |
23:25
π
|
db48x |
does xcfe4's session manager work if you have systemd but aren't actually using it? |
23:26
π
|
Senji |
I have no idea, I don't use xfce myself; but I've not had any commplaints :) |
23:26
π
|
Senji |
Various bits of gnome would pull it in too |
23:26
π
|
db48x |
heh |
23:26
π
|
Senji |
But I don't do gnome |
23:27
π
|
|
ryang has joined #internetarchive.bak |
23:41
π
|
Senji |
Apparently the official way to check for running systemd is the existence of /run/systemd/system |
23:41
π
|
Senji |
I don't ahve a systemd system to try that on :) |
23:47
π
|
db48x |
that's easy enough to check for |
23:47
π
|
db48x |
do you have documentation though? |
23:50
π
|
Senji |
Only stackexchange and reddit answers |