| Time |
Nickname |
Message |
|
03:14
🔗
|
|
tpw_rules has quit IRC (Ping timeout: 600 seconds) |
|
03:14
🔗
|
|
tpw_rules has joined #internetarchive.bak |
|
03:43
🔗
|
|
dnova_ has quit IRC (Read error: Connection reset by peer) |
|
03:48
🔗
|
|
hatsefla1 has quit IRC (Ping timeout: 600 seconds) |
|
03:53
🔗
|
|
dnova_ has joined #internetarchive.bak |
|
04:20
🔗
|
|
hatseflat has joined #internetarchive.bak |
|
06:42
🔗
|
|
yipdw has quit IRC (Read error: Operation timed out) |
|
06:43
🔗
|
|
yipdw has joined #internetarchive.bak |
|
06:43
🔗
|
|
svchfoo1 sets mode: +o yipdw |
|
07:14
🔗
|
|
ryang has quit IRC (Read error: Operation timed out) |
|
07:21
🔗
|
|
ryang has joined #internetarchive.bak |
|
07:53
🔗
|
iabak-reg |
03registrar 05master 9c0bb5f 06other 10SHARD7/pubkeys registration of dcm on SHARD7 |
|
08:10
🔗
|
iabak-reg |
03registrar 05master 9321cc9 06other 10SHARD3/pubkeys registration of dcm on SHARD3 |
|
09:18
🔗
|
dnova_ |
If diskreserve is set to less than the sum of the reserved blocks for the filesystem plus the amount of data that needs to be downloaded to run a check, iabak and iabak-cronjob are unable to perform checks due lack of free space. Default reserved blocks seems to be 5% so diskreserve should probably be set to ~7-8% of total capacity unless reserved blocks percentage is reduced. |
|
09:35
🔗
|
db48x |
hmm |
|
09:50
🔗
|
db48x |
that seems reasonable |
|
09:52
🔗
|
dnova_ |
It's a bit unexpected from a user perspective. The script asks for how much free space to leave but that's not really what it's doing. Might be better to have it work off of the reported free space, rather than subtracting diskreserve from the reported total capacity, which is not actually the usable capacity. |
|
09:54
🔗
|
|
primus104 has joined #internetarchive.bak |
|
09:57
🔗
|
dnova_ |
e.g. if you set it to reserve 1GB of free space on a 1TB volume, the script wants to keep writing 99GB past the point that the volume is full |
|
09:57
🔗
|
dnova_ |
49GB rather |
|
09:57
🔗
|
dnova_ |
so you have 0 bytes free and the scripts will fail |
|
09:57
🔗
|
db48x |
yea |
|
09:57
🔗
|
db48x |
that's down to git annex, really |
|
10:21
🔗
|
db48x |
ah: https://github.com/joeyh/git-annex/blob/master/Annex/Content.hs#L286 |
|
10:22
🔗
|
db48x |
free <- liftIO . getDiskFree =<< dir |
|
10:23
🔗
|
db48x |
let delta = need + reserve - have - alreadythere + inprogress |
|
10:23
🔗
|
db48x |
have is the amount of space available |
|
10:25
🔗
|
db48x |
need is the size that must be downloaded |
|
10:25
🔗
|
db48x |
reserve is the diskreserve setting |
|
10:26
🔗
|
dnova_ |
if I understand that correctly, then it should work as expected |
|
10:26
🔗
|
dnova_ |
since it is checking free space rather than total capacity for the math |
|
10:29
🔗
|
db48x |
https://github.com/joeyh/git-annex/blob/master/Utility/libdiskfree.c#L62 |
|
10:30
🔗
|
db48x |
man page says: fsblkcnt_t f_bavail; /* # free blocks for unprivileged users */ |
|
10:31
🔗
|
dnova_ |
then it should work as expected.... so wtf |
|
10:34
🔗
|
dnova_ |
http://pastebin.com/NZ6teBQe this is the behavior after it filled the disk. It was set to 2GB reserve. All I had to do was reduce the reserved blocks and increase the reserve setting and everything is ok, but from what you just showed this shouldn't have happened. |
|
10:36
🔗
|
db48x |
maybe some other program as writing to the disk at the same time? |
|
10:37
🔗
|
dnova_ |
no there is nothing else on that disk |
|
10:38
🔗
|
db48x |
maybe your statvfs lies? |
|
10:38
🔗
|
dnova_ |
in any case I'm glad it is supposed to work the way I thought it did originally. Whatever went wrong now I'm not sure how to explain. |
|
10:38
🔗
|
db48x |
yea, but I wish I could explain it |
|
10:41
🔗
|
dnova_ |
is diskreserve powers of 1000 or 1024? |
|
10:43
🔗
|
dnova_ |
I now have 7.5GiB free and the script is refusing to download more with a setting of 8G for diskreserve |
|
10:44
🔗
|
dnova_ |
and with 460MiB roughly in reserved blocks |
|
10:46
🔗
|
dnova_ |
that's 8.0GB free |
|
10:46
🔗
|
dnova_ |
hm. |
|
10:47
🔗
|
dnova_ |
er, sorry, 4.6GiB reserved not 460MiB |
|
10:47
🔗
|
dnova_ |
so now it's fine/as expected I think. |
|
11:39
🔗
|
dnova_ |
ok I think the problem is that when if finishes looking for small files to download when the diskreserve limit reached, it then says it looks like I have more free space and downloads the next shard data, then it scans through all those files without downloading anything, and then downloads the next shard data |
|
11:41
🔗
|
dnova_ |
if I delete the new shardN directories, free space goes back to the diskreserve setting |
|
11:56
🔗
|
db48x |
ah, interesting |
|
12:01
🔗
|
db48x |
hrm, I can't see how that would happen |
|
12:02
🔗
|
db48x |
the script checks to see that the free space is larger than the reserve before it decides to go on to another shard |
|
12:13
🔗
|
db48x |
can you edit iabak-helper slightly, adding a "set -x" at the top, then rerun it while capturing the output? |
|
12:14
🔗
|
db48x |
(so ./iabak 2>&1 >log) |
|
12:15
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
12:21
🔗
|
dnova_ |
do you want me to delete NOMORE before doing that? |
|
12:28
🔗
|
dnova_ |
deleted NOMORE and running it |
|
12:31
🔗
|
db48x |
yes, thanks |
|
12:58
🔗
|
dnova_ |
http://pastebin.com/YrszUfZb |
|
13:00
🔗
|
dnova_ |
it's going extremely slowly through shard3 files, and downloading the shard3 used an additional 1.2GB of the 4GB reserve, putting me down to 2.8GB free |
|
13:00
🔗
|
dnova_ |
the shard3 data |
|
13:01
🔗
|
dnova_ |
http://pastebin.com/MTMFLaXY you might have been looking for this |
|
13:03
🔗
|
dnova_ |
when I pressed ctrl+c it gave up on 3 and started 7, then I pressed ctrl+c again and it tried 9... |
|
13:06
🔗
|
db48x |
hrm |
|
13:06
🔗
|
db48x |
outofspace returned true... |
|
13:13
🔗
|
db48x |
oooh |
|
13:13
🔗
|
db48x |
I bet you don't have numfmt |
|
13:14
🔗
|
db48x |
if you don't have numfmt, outofspace will always return false, which means stillhavespace will always return true |
|
13:16
🔗
|
dnova_ |
I do not have numfmt |
|
13:38
🔗
|
dnova_ |
thanks for helping ferret that out |
|
14:14
🔗
|
db48x |
dnova_: you're welcome |
|
14:14
🔗
|
db48x |
dnova_: would you be willing to test a fix? |
|
14:15
🔗
|
db48x |
if so, add https://github.com/db48x/IA.BAK.git as a remote, and pull the replace-numfmt branch |
|
14:16
🔗
|
dnova_ |
well, shit, I can't anymore because I am running dist-upgrade and I will have numfmt after it finishes :/ |
|
14:16
🔗
|
db48x |
heh |
|
14:16
🔗
|
dnova_ |
sorry :( |
|
14:16
🔗
|
db48x |
no worries |
|
14:17
🔗
|
db48x |
actually, I just thought of a bug; I don't handle numbers with decimal points in them... |
|
14:18
🔗
|
|
primus104 has joined #internetarchive.bak |
|
14:20
🔗
|
db48x |
I guess I can just strip the fractional parts off |
|
14:20
🔗
|
db48x |
it's either that or write my own arithmetic functions |
|
14:20
🔗
|
db48x |
(I already wrote pow though...) |
|
14:21
🔗
|
db48x |
pushed |
|
14:22
🔗
|
db48x |
dnova_: it might be too much of a bother, but once your dist-upgrade finishes you could rename the numfmt binary to something else... |
|
14:23
🔗
|
dnova_ |
sure, gonna be a little while though |
|
14:24
🔗
|
db48x |
no problem |
|
14:31
🔗
|
db48x |
on the gripping hand, I should probably just be using dc or something |
|
14:31
🔗
|
db48x |
it's probably more readily available than numfmt |
|
16:28
🔗
|
|
Kenshin has quit IRC (Ping timeout: 246 seconds) |
|
16:32
🔗
|
|
Kenshin has joined #internetarchive.bak |
|
16:32
🔗
|
|
svchfoo1 sets mode: +o Kenshin |
|
17:07
🔗
|
|
Cyrus has quit IRC (Quit: Leaving) |
|
17:08
🔗
|
|
ohhdemgir has joined #internetarchive.bak |
|
17:21
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
17:48
🔗
|
|
zhongfu has quit IRC (Ping timeout: 240 seconds) |
|
17:50
🔗
|
|
zhongfu has joined #internetarchive.bak |
|
18:09
🔗
|
|
SketchCow has quit IRC (Ping timeout: 240 seconds) |
|
18:09
🔗
|
|
SketchCow has joined #internetarchive.bak |
|
18:10
🔗
|
|
svchfoo1 sets mode: +o SketchCow |
|
18:45
🔗
|
|
primus104 has joined #internetarchive.bak |
|
18:51
🔗
|
|
Sanqui|go is now known as Sanqui |