[02:25] *** wp494 has quit IRC (Read error: Connection reset by peer) [06:17] *** wp494 has joined #internetarchive.bak [09:53] *** atomotic has joined #internetarchive.bak [11:56] *** db48x has quit IRC (Ping timeout: 258 seconds) [12:01] *** tpw_rules has quit IRC (Ping timeout: 258 seconds) [12:01] *** tpw-rules has joined #internetarchive.bak [12:01] *** tpw-rules is now known as tpw_rules [12:06] *** atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) [12:07] *** beardicus has quit IRC (Read error: Operation timed out) [12:09] *** closure has quit IRC (Read error: Operation timed out) [12:10] *** atomotic has joined #internetarchive.bak [12:27] I've got a file in shard3 that's crashing git annex fsck [12:27] fsck ephemera/DedicationTheRelifSocietyMonumentToWomen/IA000012_DedicationThe RelifSocietyMonumentToWomen.mov git-annex: out of memory (requested 38025560064 bytes) [12:50] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [12:58] *** toad2 has joined #internetarchive.bak [12:59] *** toad1 has quit IRC (Read error: Operation timed out) [13:06] *** beardicus has joined #internetarchive.bak [13:08] *** closure has joined #internetarchive.bak [14:01] closure: I've got a file in shard3 that's crashing git annex fsck [14:01] fsck ephemera/DedicationTheRelifSocietyMonumentToWomen/IA000012_DedicationThe RelifSocietyMonumentToWomen.mov git-annex: out of memory (requested 38025560064 bytes) [14:09] *** db48x` has joined #internetarchive.bak [14:10] *** atomotic has joined #internetarchive.bak [15:40] *** VADemon has joined #internetarchive.bak [16:26] *** xhdr has quit IRC (Excess Flood) [16:26] *** xhdr has joined #internetarchive.bak [16:28] hmm, interesting. How large is that file? [16:29] -r--r--r-- 1 jdamery-iabak jdamery-iabak 38025046878 Aug 14 2014 .git/annex/objects/1v/XW/MD5-s38025046878--60a8a62c9a99abdde6ea99f62cd3cc1b/MD5-s38025046878--60a8a62c9a99abdde6ea99f62cd3cc1b [16:29] well, that looks strangely similar, doesn't it [16:29] So I guess it's trying to malloc the filesize plus a bit ? :) [16:29] hmm, it should not be reading the whole file into memory, afaik.. but seems it is in this case [16:30] * closure bets it's specific to md5 hashing [16:37] hmm, I don't seem to reproduce it here. What OS , git-annex version, and how was it installed? [16:39] 'git-annex version: 6.20160217-g95bbdb8' [16:39] As installed by iabak [16:39] Debian jessie on amd64 [16:45] hmm, using the same build here, it fscks a 200 mb file using 6 mb ram. [16:46] unless there's something about much larger files that makes it want to read in the whoe file.. [16:47] * closure goes looking for a box with 38 gb download quota .. [16:47] 1960 jdamery+ 20 0 35.785g 5.506g 29728 D 6.3 35.2 0:03.29 git-annex [16:48] Although the mmap succeeded this time [16:48] (just trying to fsck that one file rather than run the whole fsck again) [16:51] aha! I reproduced it with a 40 gb file [16:51] weird that small files down't have the same problem [16:52] 1441109 joey 20 0 39.373g 0.038t 30432 R 104.0 31.0 0:28.25 git-annex [16:54] well, that's 3 gb at least, certianly seems too much [16:54] This fsck is currently using up 14 of the 16gb of memory in this computer [16:54] any maybe a 39 gb allocation, not sure what's going on there between VIRT and RES [17:01] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [17:07] They have ammonites too [17:08] Bah, mischat :) [17:26] Senji: well, I've managed to root cause the bug, but I don't entirely know how to fix it just yet. [17:26] as a workaround, and git-annex version from last year does not have the bug [17:26] 2210444 joey 20 0 383384 34636 29872 R 97.1 0.0 0:47.19 git-annex [17:43] *** CyberJaco is now known as zz_CyberJ [17:54] Senji: here's a fixed version : https://downloads.kitenet.net/git-annex/autobuild/amd64/git-annex-standalone-amd64.tar.gz [20:26] closure: thanks; I'll grab that and give it a go [20:50] 66.82user 11.39system 1:18.33elapsed 99%CPU (0avgtext+0avgdata 40476maxresident) [20:50] muchos betters [20:51] My fsck hasn't got to the problem file yet; but it's looking a lot better on memory footprint [20:56] underlying problem in all its evil glory https://github.com/haskell-crypto/cryptonite/issues/70 [20:57] Eww [20:58] * closure leaves to tear down and upgrade the house's battery bank [20:59] my battery greased hands will feel so much cleaner than that laziness mess [21:00] Heh. Good luck :) [21:34] closure: Let me know when you're up for us going next level all up in here [22:35] *** xhdr has quit IRC (hub.se irc.du.se) [22:50] *** xhdr- has joined #internetarchive.bak [22:50] *** xhdr- has quit IRC (Excess Flood) [22:55] *** xhdr has joined #internetarchive.bak [22:55] *** xhdr has quit IRC (Excess Flood) [22:57] *** xhdr has joined #internetarchive.bak [23:18] *** xhdr- has joined #internetarchive.bak [23:19] *** xhdr has quit IRC (hub.se irc.du.se)