Time |
Nickname |
Message |
02:25
🔗
|
|
wp494 has quit IRC (Read error: Connection reset by peer) |
06:17
🔗
|
|
wp494 has joined #internetarchive.bak |
09:53
🔗
|
|
atomotic has joined #internetarchive.bak |
11:56
🔗
|
|
db48x has quit IRC (Ping timeout: 258 seconds) |
12:01
🔗
|
|
tpw_rules has quit IRC (Ping timeout: 258 seconds) |
12:01
🔗
|
|
tpw-rules has joined #internetarchive.bak |
12:01
🔗
|
|
tpw-rules is now known as tpw_rules |
12:06
🔗
|
|
atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) |
12:07
🔗
|
|
beardicus has quit IRC (Read error: Operation timed out) |
12:09
🔗
|
|
closure has quit IRC (Read error: Operation timed out) |
12:10
🔗
|
|
atomotic has joined #internetarchive.bak |
12:27
🔗
|
Senji |
I've got a file in shard3 that's crashing git annex fsck |
12:27
🔗
|
Senji |
fsck ephemera/DedicationTheRelifSocietyMonumentToWomen/IA000012_DedicationThe RelifSocietyMonumentToWomen.mov git-annex: out of memory (requested 38025560064 bytes) |
12:50
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
12:58
🔗
|
|
toad2 has joined #internetarchive.bak |
12:59
🔗
|
|
toad1 has quit IRC (Read error: Operation timed out) |
13:06
🔗
|
|
beardicus has joined #internetarchive.bak |
13:08
🔗
|
|
closure has joined #internetarchive.bak |
14:01
🔗
|
Senji |
closure: I've got a file in shard3 that's crashing git annex fsck |
14:01
🔗
|
Senji |
fsck ephemera/DedicationTheRelifSocietyMonumentToWomen/IA000012_DedicationThe RelifSocietyMonumentToWomen.mov git-annex: out of memory (requested 38025560064 bytes) |
14:09
🔗
|
|
db48x` has joined #internetarchive.bak |
14:10
🔗
|
|
atomotic has joined #internetarchive.bak |
15:40
🔗
|
|
VADemon has joined #internetarchive.bak |
16:26
🔗
|
|
xhdr has quit IRC (Excess Flood) |
16:26
🔗
|
|
xhdr has joined #internetarchive.bak |
16:28
🔗
|
closure |
hmm, interesting. How large is that file? |
16:29
🔗
|
Senji |
-r--r--r-- 1 jdamery-iabak jdamery-iabak 38025046878 Aug 14 2014 .git/annex/objects/1v/XW/MD5-s38025046878--60a8a62c9a99abdde6ea99f62cd3cc1b/MD5-s38025046878--60a8a62c9a99abdde6ea99f62cd3cc1b |
16:29
🔗
|
closure |
well, that looks strangely similar, doesn't it |
16:29
🔗
|
Senji |
So I guess it's trying to malloc the filesize plus a bit ? :) |
16:29
🔗
|
closure |
hmm, it should not be reading the whole file into memory, afaik.. but seems it is in this case |
16:30
🔗
|
* |
closure bets it's specific to md5 hashing |
16:37
🔗
|
closure |
hmm, I don't seem to reproduce it here. What OS , git-annex version, and how was it installed? |
16:39
🔗
|
Senji |
'git-annex version: 6.20160217-g95bbdb8' |
16:39
🔗
|
Senji |
As installed by iabak |
16:39
🔗
|
Senji |
Debian jessie on amd64 |
16:45
🔗
|
closure |
hmm, using the same build here, it fscks a 200 mb file using 6 mb ram. |
16:46
🔗
|
closure |
unless there's something about much larger files that makes it want to read in the whoe file.. |
16:47
🔗
|
* |
closure goes looking for a box with 38 gb download quota .. |
16:47
🔗
|
Senji |
1960 jdamery+ 20 0 35.785g 5.506g 29728 D 6.3 35.2 0:03.29 git-annex |
16:48
🔗
|
Senji |
Although the mmap succeeded this time |
16:48
🔗
|
Senji |
(just trying to fsck that one file rather than run the whole fsck again) |
16:51
🔗
|
closure |
aha! I reproduced it with a 40 gb file |
16:51
🔗
|
closure |
weird that small files down't have the same problem |
16:52
🔗
|
closure |
1441109 joey 20 0 39.373g 0.038t 30432 R 104.0 31.0 0:28.25 git-annex |
16:54
🔗
|
closure |
well, that's 3 gb at least, certianly seems too much |
16:54
🔗
|
Senji |
This fsck is currently using up 14 of the 16gb of memory in this computer |
16:54
🔗
|
closure |
any maybe a 39 gb allocation, not sure what's going on there between VIRT and RES |
17:01
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
17:07
🔗
|
Senji |
They have ammonites too |
17:08
🔗
|
Senji |
Bah, mischat :) |
17:26
🔗
|
closure |
Senji: well, I've managed to root cause the bug, but I don't entirely know how to fix it just yet. |
17:26
🔗
|
closure |
as a workaround, and git-annex version from last year does not have the bug |
17:26
🔗
|
closure |
2210444 joey 20 0 383384 34636 29872 R 97.1 0.0 0:47.19 git-annex |
17:43
🔗
|
|
CyberJaco is now known as zz_CyberJ |
17:54
🔗
|
closure |
Senji: here's a fixed version : https://downloads.kitenet.net/git-annex/autobuild/amd64/git-annex-standalone-amd64.tar.gz |
20:26
🔗
|
Senji |
closure: thanks; I'll grab that and give it a go |
20:50
🔗
|
closure |
66.82user 11.39system 1:18.33elapsed 99%CPU (0avgtext+0avgdata 40476maxresident) |
20:50
🔗
|
closure |
muchos betters |
20:51
🔗
|
Senji |
My fsck hasn't got to the problem file yet; but it's looking a lot better on memory footprint |
20:56
🔗
|
closure |
underlying problem in all its evil glory https://github.com/haskell-crypto/cryptonite/issues/70 |
20:57
🔗
|
Senji |
Eww |
20:58
🔗
|
* |
closure leaves to tear down and upgrade the house's battery bank |
20:59
🔗
|
closure |
my battery greased hands will feel so much cleaner than that laziness mess |
21:00
🔗
|
Senji |
Heh. Good luck :) |
21:34
🔗
|
SketchCow |
closure: Let me know when you're up for us going next level all up in here |
22:35
🔗
|
|
xhdr has quit IRC (hub.se irc.du.se) |
22:50
🔗
|
|
xhdr- has joined #internetarchive.bak |
22:50
🔗
|
|
xhdr- has quit IRC (Excess Flood) |
22:55
🔗
|
|
xhdr has joined #internetarchive.bak |
22:55
🔗
|
|
xhdr has quit IRC (Excess Flood) |
22:57
🔗
|
|
xhdr has joined #internetarchive.bak |
23:18
🔗
|
|
xhdr- has joined #internetarchive.bak |
23:19
🔗
|
|
xhdr has quit IRC (hub.se irc.du.se) |