#internetarchive.bak 2016-02-26,Fri

↑back Search

Time Nickname Message
02:25 🔗 wp494 has quit IRC (Read error: Connection reset by peer)
06:17 🔗 wp494 has joined #internetarchive.bak
09:53 🔗 atomotic has joined #internetarchive.bak
11:56 🔗 db48x has quit IRC (Ping timeout: 258 seconds)
12:01 🔗 tpw_rules has quit IRC (Ping timeout: 258 seconds)
12:01 🔗 tpw-rules has joined #internetarchive.bak
12:01 🔗 tpw-rules is now known as tpw_rules
12:06 🔗 atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)
12:07 🔗 beardicus has quit IRC (Read error: Operation timed out)
12:09 🔗 closure has quit IRC (Read error: Operation timed out)
12:10 🔗 atomotic has joined #internetarchive.bak
12:27 🔗 Senji I've got a file in shard3 that's crashing git annex fsck
12:27 🔗 Senji fsck ephemera/DedicationTheRelifSocietyMonumentToWomen/IA000012_DedicationThe RelifSocietyMonumentToWomen.mov git-annex: out of memory (requested 38025560064 bytes)
12:50 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
12:58 🔗 toad2 has joined #internetarchive.bak
12:59 🔗 toad1 has quit IRC (Read error: Operation timed out)
13:06 🔗 beardicus has joined #internetarchive.bak
13:08 🔗 closure has joined #internetarchive.bak
14:01 🔗 Senji closure: I've got a file in shard3 that's crashing git annex fsck
14:01 🔗 Senji fsck ephemera/DedicationTheRelifSocietyMonumentToWomen/IA000012_DedicationThe RelifSocietyMonumentToWomen.mov git-annex: out of memory (requested 38025560064 bytes)
14:09 🔗 db48x` has joined #internetarchive.bak
14:10 🔗 atomotic has joined #internetarchive.bak
15:40 🔗 VADemon has joined #internetarchive.bak
16:26 🔗 xhdr has quit IRC (Excess Flood)
16:26 🔗 xhdr has joined #internetarchive.bak
16:28 🔗 closure hmm, interesting. How large is that file?
16:29 🔗 Senji -r--r--r-- 1 jdamery-iabak jdamery-iabak 38025046878 Aug 14 2014 .git/annex/objects/1v/XW/MD5-s38025046878--60a8a62c9a99abdde6ea99f62cd3cc1b/MD5-s38025046878--60a8a62c9a99abdde6ea99f62cd3cc1b
16:29 🔗 closure well, that looks strangely similar, doesn't it
16:29 🔗 Senji So I guess it's trying to malloc the filesize plus a bit ? :)
16:29 🔗 closure hmm, it should not be reading the whole file into memory, afaik.. but seems it is in this case
16:30 🔗 * closure bets it's specific to md5 hashing
16:37 🔗 closure hmm, I don't seem to reproduce it here. What OS , git-annex version, and how was it installed?
16:39 🔗 Senji 'git-annex version: 6.20160217-g95bbdb8'
16:39 🔗 Senji As installed by iabak
16:39 🔗 Senji Debian jessie on amd64
16:45 🔗 closure hmm, using the same build here, it fscks a 200 mb file using 6 mb ram.
16:46 🔗 closure unless there's something about much larger files that makes it want to read in the whoe file..
16:47 🔗 * closure goes looking for a box with 38 gb download quota ..
16:47 🔗 Senji 1960 jdamery+ 20 0 35.785g 5.506g 29728 D 6.3 35.2 0:03.29 git-annex
16:48 🔗 Senji Although the mmap succeeded this time
16:48 🔗 Senji (just trying to fsck that one file rather than run the whole fsck again)
16:51 🔗 closure aha! I reproduced it with a 40 gb file
16:51 🔗 closure weird that small files down't have the same problem
16:52 🔗 closure 1441109 joey 20 0 39.373g 0.038t 30432 R 104.0 31.0 0:28.25 git-annex
16:54 🔗 closure well, that's 3 gb at least, certianly seems too much
16:54 🔗 Senji This fsck is currently using up 14 of the 16gb of memory in this computer
16:54 🔗 closure any maybe a 39 gb allocation, not sure what's going on there between VIRT and RES
17:01 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
17:07 🔗 Senji They have ammonites too
17:08 🔗 Senji Bah, mischat :)
17:26 🔗 closure Senji: well, I've managed to root cause the bug, but I don't entirely know how to fix it just yet.
17:26 🔗 closure as a workaround, and git-annex version from last year does not have the bug
17:26 🔗 closure 2210444 joey 20 0 383384 34636 29872 R 97.1 0.0 0:47.19 git-annex
17:43 🔗 CyberJaco is now known as zz_CyberJ
17:54 🔗 closure Senji: here's a fixed version : https://downloads.kitenet.net/git-annex/autobuild/amd64/git-annex-standalone-amd64.tar.gz
20:26 🔗 Senji closure: thanks; I'll grab that and give it a go
20:50 🔗 closure 66.82user 11.39system 1:18.33elapsed 99%CPU (0avgtext+0avgdata 40476maxresident)
20:50 🔗 closure muchos betters
20:51 🔗 Senji My fsck hasn't got to the problem file yet; but it's looking a lot better on memory footprint
20:56 🔗 closure underlying problem in all its evil glory https://github.com/haskell-crypto/cryptonite/issues/70
20:57 🔗 Senji Eww
20:58 🔗 * closure leaves to tear down and upgrade the house's battery bank
20:59 🔗 closure my battery greased hands will feel so much cleaner than that laziness mess
21:00 🔗 Senji Heh. Good luck :)
21:34 🔗 SketchCow closure: Let me know when you're up for us going next level all up in here
22:35 🔗 xhdr has quit IRC (hub.se irc.du.se)
22:50 🔗 xhdr- has joined #internetarchive.bak
22:50 🔗 xhdr- has quit IRC (Excess Flood)
22:55 🔗 xhdr has joined #internetarchive.bak
22:55 🔗 xhdr has quit IRC (Excess Flood)
22:57 🔗 xhdr has joined #internetarchive.bak
23:18 🔗 xhdr- has joined #internetarchive.bak
23:19 🔗 xhdr has quit IRC (hub.se irc.du.se)

irclogger-viewer