[00:46] *** db421 has joined #jsmess [00:57] *** SketchCo2 has joined #jsmess [00:57] *** SketchCo1 has quit IRC (se.hub irc.underworld.no) [00:57] *** dboard420 has quit IRC (se.hub irc.underworld.no) [01:12] *** db421 is now known as dboard420 [05:18] http://fos.textfiles.com/dfjustin/dccons/dc/ [05:18] DFJustin: I tired to slot in dccons but it's just not liking life [06:14] Meanwhile, I'm happy to say, I'm visiting the arcade repair shop [06:14] And I've made 5 come to life [06:15] I'm essentially sitting here with the full ROM and CHD sets, and slotting them in and reloading [06:15] and it's going [06:21] the loader js that I'm using on fos and teamarchive1 needs a tweak in each of the system folders [06:21] to accommodate the new version [06:22] luckily the IA emularity loader didn't, but that's one of the things I was referring to the other day [06:26] seems to be booting (slowly) now [06:28] 20 megs is 20 megs [06:28] So, next question. [06:28] Did we ever figure out CHDs? [06:28] I'm looking at the arcade repair shop and fixing things left and right [06:28] yes ages ago [06:28] Going to try shoving stuff in [06:28] I thought so. [06:28] But we'll see, I'm trying now [06:29] DC is CLEARLY faster. [06:29] Running at 22% [06:30] than what [06:30] I remember it being 15% or 16% before this update [06:31] https://archive.org/details/arcade_tetriskr works! [06:38] Trying one with a CHD [06:38] did we need to do anything special? [06:38] (It's been a while.) [06:42] I think it is clearly saying "not found". [06:42] I tested with https://archive.org/details/arcade_azumanga and now I am testing with https://archive.org/details/arcade_blitz2k [06:42] blitz2k.chd NOT FOUND [06:43] But it said "loading game drive" and definitely loaded the chd in [06:44] Yeah, I don't think we fixed this. It definitely loads the chd and then can't find it. [06:45] there's code in loader.js to handle it so it must have worked at some point [06:45] I don't have time to look at it right now [06:45] I'll check my logs [06:45] it'll be emularity and not mame though [06:50] 12:38 < SketchCow> Hey bai [06:50] 12:53 < bai> morning SketchCow [06:50] 12:58 < SketchCow> I had a thing [06:50] 12:58 < SketchCow> I assume you're on the third kid by now [06:50] 12:58 < SketchCow> The first one can watch them [06:50] 12:59 < SketchCow> We're trying to add a single line to the loader.js that says (and DFJustin can verify) [06:50] 12:59 < SketchCow> "If there's a chd file in the item, put it in /emulator/emulatorname/ [06:50] 14:23 < SketchCow> I can't get the loader for CHD until bai frees up [06:51] That's it. [06:52] We had a solution, needed it added to loader.js, and then bai had 5,000 kids [07:17] I uh....I haven't really worked much with the chd stuff before, lemme see [07:17] That's the line. If there's CHD in the item, move it in the /emulator to /emulator/emulatorname [07:18] Where emulatorname is the name of the emulator. [07:18] We can be crude and use arcade_XXXX and use XXXX as the arcade name, or another method if you find it. [07:18] In the example we had, we found: [07:19] for arcade_blitz [07:19] it looks like it is in that directory already [07:19] FS.readdir('/emulator/azumanga') [07:19] (3) ['gdl-0018.chd', '.', '..'] [07:19] blitz.chd (all chds) needed to be in a blitz subdirectory [07:20] The heck you say [07:21] < DFJustin> basically if get_mame_files() finds a chd it needs to load it into the browserfs in a subfolder with the same name as the "driver" property in the json [07:21] I wish IA loaded these files faster than 1MB/s for me, I think that's probably why I never get past the initial debugging stages whenever this comes up, my ADHD kicks in and it gets lost 10 tabs deep [07:22] I wish we could find a chd that was smaller. [07:22] Let me check. [07:22] < DFJustin> yes, emulator/blitz/blitz.chd [07:22] loading blitz now to see what its fs looks like [07:24] In the future, here's zoofari [07:24] https://archive.org/details/arcade_zoofari [07:24] 22mb [07:25] https://archive.org/details/arcade_fbaitmc (10mb) [07:25] smallest we'll find. https://archive.org/details/arcade_fbaitmc [07:25] Good test one for you. [07:26] I'm fixing it up to ensure it is JUST the chd problem [07:27] hmm, ok so the file is there but it throws an FS error when you try to run stat on it [07:28] FS.readdir('/emulator/blitz2k') [07:28] (3) ['blitz2k.chd', '.', '..'] [07:28] FS.stat('/emulator/blitz2k/blitz2k.chd') [07:28] mameseattle.js.gz:1 Uncaught [07:28] ErrnoError {node: FSNode, errno: 2, message: 'FS error', setErrno: ƒ} [07:29] Verified for the future, by the way: https://archive.org/details/arcade_fbaitmc is now failing the same as blitz, but is a 10mb load. [07:29] Loads two game files and a game drive. [07:30] But, the next issue, the one you're finding. [07:30] FS.readdir('/emulator/') [07:30] (6) ['fbaitmc', 'sys573.zip', 'fbaitmc.zip', 'fbaitmc.cfg', '.', '..'] [07:30] FS.readdir('/emulator/fbaitmc') [07:30] (3) ['889ea.chd', '.', '..'] [07:33] Do you think this is a JVilk issue? [07:33] A big in the FSloader? [07:37] I'm not sure, looking into what's different about how we're loading those files now [07:37] I sent this all to Jvilk, just in case he goes "OH NO" [07:40] // Grab the BrowserFS Emscripten FS plugin. [07:40] var BFS = new BrowserFS.EmscriptenFS(); [07:40] // Create the folder that we'll turn into a mount point. [07:40] FS.createFolder(FS.root, 'data', true, true); [07:40] // Mount BFS's root folder into the '/data' folder. [07:40] FS.mount(BFS, {root: '/'}, '/data'); [07:43] That's where my mind goes to - that the file in the subdirectory isn't set to readable [07:45] *** Sgeo has quit IRC (Read error: Connection reset by peer) [07:46] I'm having a hard time remembering how this interfaces with the loader in the IA environment, eg, which script says "add this chd to our mountable filesystem" [07:48] Isn't that loader.js [07:49] https://github.com/db48x/emularity/blob/master/loader.js#L403 [07:50] loader.js is the code that accepts all the parameters and does things with them yeah, but somewhere else says "for fbaitmc, grab 889ea.chd and mount it at /emulator/fbaitmc/889ea.chd" [07:51] That link I put you at did it [07:52] yeah, https://archive.org/details/arcade_fbaitmc calls in some script which does it [07:52] or a json file maybe [07:52] The emulator uses: [07:53] https://ia601902.us.archive.org/18/items/emularity_engine_v1/fbaitmc.json [07:53] https://ia601902.us.archive.org/cors_get.php?path=/18/items/emularity_engine_v1/fbaitmc.json doesn't mention the chd files anywhere [07:53] No [07:53] Or the .zip files [07:53] The loader.js does that work. [07:53] Creates /emulator, puts in the required files from the system [07:53] ohh, it pulls in the xml for the item and parses it out of there, doesn't it [07:55] var get_meta_url = function (game_path) { [07:55] var path = game_path.split('/'); [07:55] return "//cors.archive.org/cors/"+ path[0] +"/"+ path[0] +"_meta.xml"; [07:55] }; [07:55] var get_files_url = function (game_path) { [07:55] var path = game_path.split('/'); [07:55] return "//cors.archive.org/cors/"+ path[0] +"/"+ path[0] +"_files.xml"; [07:55] }; [07:55] var get_zip_url = function (game_path, item_path) { [07:55] if (item_path) { [07:55] return "//cors.archive.org/cors/"+ item_path +"/"+ game_path; [07:55] } [07:55] return "//cors.archive.org/cors/"+ game_path; [07:55] }; [07:55] https://github.com/db48x/emularity/blob/master/loader.js#L626 [07:56] yeah, I think it grabs itemname_files.xml and just looks for any files that end with .chd [08:00] yeah it's mounting it the same way as it's jamming the .zip into the filesystem, I was thinking it was a difference of zipfs mounting a zip vs injecting a file directly into the filesystem by path, but we're doing the same thing with the zips [08:01] Is there possibly a permissions issue? [08:01] // Create the folder that we'll turn into a mount point. [08:01] FS.createFolder(FS.root, 'data', true, true); [08:01] // Mount BFS's root folder into the '/data' folder. [08:01] FS.mount(BFS, {root: '/'}, '/data'); [08:01] In that example, it's doing stuff to the data folder [08:09] We call loader.js [08:09] I don't think it's permissions, because a stat of a file which you don't have permissions to read should still return data [08:10] like, that's how you would read the permissions [08:10] We definitely think the file is there [08:10] We can see it [08:10] and then it gives that error accessing it [08:11] Well, we have [08:12] 1. A good test case that's 10mb so you don't get bored [08:12] 2. An idea of the issue (it can't read the .chd for some rwason) [08:12] 3. An awareness it is not MAME, now that you can reproduce it outside [08:12] Chances are, in some way, I feel it's setting up the file in the directory in a way it fails to read [08:13] yeah [08:13] I was hoping there was something obviously different about how we mount those files from how we mount the other files, but it's the same [08:14] for the 500 meg ones I'd maybe throw up my hands and say "well, maybe that's just too big for our bizarro browser virtual filesystem doohickey to handle" [08:14] but 10mb is definitely workable, we know that [08:14] the browser seems to be downloading the chd without issue as well [08:19] I need to lie down. But that seems to be what we're seeking. [08:24] yeah maybe it's the kind of thing where jvilk can look at it for 30 seconds and say "oh, it's not returning valid data because of _____" [08:25] it all SEEMS to be ok on our end [08:26] this is tge last big hurdle. [08:26] 500 games come alive with this [08:27] I cant find this error anywhere on the net [08:32] https://github.com/emscripten-core/emscripten/blob/1ae5dd3431fa0ccbe939c740d256dfb587b05198/src/library_fs.js#L1452 [08:32] if we compile with assertions we might get something slightly more useful [08:36] https://github.com/emscripten-core/emscripten/blob/8e1e305519e1027726a48861a1fec5662f7e18a2/src/library.js#L1570 [08:36] errno 2 is "no such file or directory" but again, we see it there, it just....won't read [08:54] you want me to recompile that emulator with assertions? [17:38] *** Sgeo has joined #jsmess [18:33] (The next morning...) [18:33] I'll see about doing it shortly. [19:19] OK, going to pause the master build (setting it so it can rebuild) and then will do an ASSERTIONS=1 build of fbaitmc [19:28] I seem to be having some trouble getting assertions in there [19:41] Building ksys573 (the fbaitmc support driver) [20:02] 889ea.chd NOT FOUND [20:02] arcade_fbaitmc:1 Uncaught (in promise) 10912480 - Exception catching is disabled, this exception cannot be caught. Compile with -s NO_DISABLE_EXCEPTION_CATCHING or -s EXCEPTION_CATCHING_ALLOWED=[..] to catch. [20:02] Promise.then (async) [20:02] instantiateArrayBuffer @ mameksys573.js.gz:2699 [20:02] instantiateAsync @ mameksys573.js.gz:2731 [20:02] createWasm @ mameksys573.js.gz:2748 [20:02] (anonymous) @ mameksys573.js.gz:15536 [20:02] 073a0f56:0x1510e15 Uncaught RuntimeError: memory access out of bounds at 073a0f56:0x1510e15 at 073a0f56:0x159ce64 at Object.__ZN13sound_manager4muteEbh (mameksys573.js.gz:2585) at MAMERunner.value (loader.js?v=51c75c55:1163) at loader.js?v=51c75c55:1685 [20:02] $func79649 @ 073a0f56:0x1510e15 [20:02] $_ZN13sound_manager4muteEbh @ 073a0f56:0x159ce64 [20:02] (anonymous) @ mameksys573.js.gz:2585 [20:02] value @ loader.js?v=51c75c55:1163 [20:02] (anonymous) @ loader.js?v=51c75c55:1685 [20:02] setTimeout (async) [20:02] (anonymous) @ loader.js?v=51c75c55:1683 [20:02] (anonymous) @ loader.js?v=51c75c55:1069 [20:02] EmscriptenRunner.Module.preRun @ loader.js?v=51c75c55:1067 [20:02] callRuntimeCallbacks @ mameksys573.js.gz:2812 [20:02] preRun @ mameksys573.js.gz:2357 [20:02] run @ mameksys573.js.gz:16238 [20:02] runCaller @ mameksys573.js.gz:16181 [20:02] removeRunDependency @ mameksys573.js.gz:2514 [20:03] receiveInstance @ mameksys573.js.gz:2674 [20:03] receiveInstantiationResult @ mameksys573.js.gz:2691 [20:03] Promise.then (async) [20:03] instantiateArrayBuffer @ mameksys573.js.gz:2699 [20:03] instantiateAsync @ mameksys573.js.gz:2731 [20:03] createWasm @ mameksys573.js.gz:2748 [20:03] (anonymous) @ mameksys573.js.gz:15536 [20:03] Recompiling with -s NO_DISABLE_EXCEPTION_CATCHING [20:09] Then I have to go record some podcasts. [20:23] Recompiled with the new line [20:23] Ok, officially [20:23] -------------------------------------------------------------------------------- [20:23] 889ea.chd NOT FOUND [20:23] arcade_fbaitmc:1 Uncaught (in promise) 10911656 [20:23] Promise.then (async) [20:23] instantiateArrayBuffer @ mameksys573.js.gz:2699 [20:23] instantiateAsync @ mameksys573.js.gz:2731 [20:23] createWasm @ mameksys573.js.gz:2748 [20:23] (anonymous) @ mameksys573.js.gz:15603 [20:23] 073d53ca:0x15113d3 Uncaught RuntimeError: memory access out of bounds at 073d53ca:0x15113d3 at 073d53ca:0x159d422 at Object.__ZN13sound_manager4muteEbh (mameksys573.js.gz:2585) at MAMERunner.value (loader.js?v=51c75c55:1163) at loader.js?v=51c75c55:1685 [20:24] ... I don't think this is mame [20:24] I think this is the loader and how Jvik's browser isn't seeing the file. [20:28] *** jvilk has joined #jsmess [20:28] hey all. so the stat call is failing because the mode on the file is 0x4000, which is... no bits set. not sure why yet. [20:28] this is reflected in the thrown error message; there is a `node` property on the error, and its mode property is 0x4000. [20:29] (for FS.stat('/emulator/fbaitmc/889ea.chd')) [20:30] Hey there, I almost called the entire Internet on this one [20:31] So it's loading it in, then it's setting the mode wrong. [20:31] OK, here's the loader.js code doing this work: [20:31] yeah, or reading the mode from the zip file wrong? [20:31] or the mode in the zip is wrong / et [20:31] Oh no [20:31] It's using the MODE INSIDE THE ZIP? [20:32] Is there a way in the command to override it? [20:32] are we using browserfs to mount the zip file [20:32] I forget [20:32] Yes [20:32] Wait one moment [20:32] https://github.com/db48x/emularity/blob/master/loader.js#L402 [20:33] root@teamarchive2:/1/0/HIPHOP/ASSAULT/MAME 0.238 CHDs (merged)/fbaitmc# ls -l [20:33] total 10336 [20:33] -rw-r--r-- 1 root root 10583635 Dec 31 02:08 889ea.chd [20:33] Are you saying, set that differently? The CHD is not in a zip [20:33] It's in a separate file and subdirectory [20:35] somehow /emulator/fbaitmc/889ea.chd has no permission bits so no one can read/write/execute. looking into how this gets into thre file system now... [20:40] Clearly, this is not MAME. I'm going to go back to compiles. [20:40] hmm, is it because we're using a mountpoint that's inside of a directory that's mounted as a zipfs? [20:40] the directory itself has all permission bits set. what are we storing the CHD in? [20:42] the chd is just being loaded as is, it's not inside of a zip or anything [20:43] I believe it gets written using fs.writeFileSync here https://github.com/db48x/emularity/blob/05390a4c915790825b67431803cada9ef66fa97f/loader.js#L1539-L1554 [20:43] :looking: [20:44] (this is what i was looking for, thx) [20:48] OK, when it gets written there it has the correct mode [20:48] trying to untangle exactly what we're doing as far as mounts, I *think* (but still tracing) what we're doing is, / is a MountableFS with an OverlayFS, we mount the zip at /emulator, and then we write a file to /emulator// [20:48] Technically emulatorname, but yes. [20:49] So /emulator/blitz2k/blitz2k.chd [20:49] But sometimes it'll be, like /emulator/jboogie.zip [20:49] and then /emulator/jboogie/harddrv_jb.chd [20:49] actually I guess /emulator is a BFS, not the ZipFS [20:50] sorry, an EMscriptenFS [20:50] https://github.com/db48x/emularity/blob/05390a4c915790825b67431803cada9ef66fa97f/loader.js#L1061-L1064 [20:53] I think I'm confusing the issue with ZipFS, I don't think we're actually using ZipFS for these items - those are for dosbox drives, not for the arcades. for the arcades we're writing .zip files into the filesystem in the same way as we're writing the CHD (but to /emulator instead of /emulator/) [20:53] and mame is handling reading the .zip [20:54] you had the write writeFileSync callsite. it's writing the file to the filesystem with a readable mode, and i confirmed it's in BFS with a readable mode. now i'm debugging the _emscripten_ file system nodes BFS is creating for bugs [20:54] right* [20:57] as a side note, for one of my other projects I recently built a visual explorer for BrowserFS, it might be worthwhile to us if I split that out into a standalone package and added more debugging tools, I always end up having to drop to console and explore the FS manually for the IA setup [20:57] https://bai.dev/images/articles/janusxr-editor-improvements/vfs-file-browser.png [20:58] i'm out of time -- my son just woke up. [20:58] oof [20:58] Well, thanks for helping us determine the scope of the issue. [20:59] It SOUNDS like you believe it's loader.js setting a permission wrong based on a misunderstanding [20:59] loader.js is setting a readable permission when writing into browserfs, and browserfs reflects that mode... but somehow emscripten is getting a mode of 0. [21:00] dang layers [21:01] OK, good luck all. I probably won't have time to pop back on tonight due to new years (doing some remote stuff with family) [21:01] *** jvilk has quit IRC (Quit: leaving) [21:06] I've resumed the building of the replacement WASMs [21:06] emularity_engine_v1 needs a good cleanout anyway [21:07] 15 machines that were working or needed minor things working got fixed so far, as I went "are you in the repair shop but don't need a CHD"? [21:07] I wonder if bringing on Alon will make a difference for this. [21:51] I'm not sure it's quite THAT low level, I think it's some interaction between MountableFilesystem, IndexedDBFilesystem, OverlayFilesystem, and AsyncMirrorFilesystem [21:52] gonna see if I can tease out what the final FS stack ends up at so we can give that info to jvilk next year [22:00] OverlayFS( { readable: MountableFS, writable: AsyncMirrorFS( { sync: InMemoryFS, async: IndexedDBFS } ) } ) [22:00] there's another InMemoryFS which gets created but I think never used [22:01] but all these files that we load and then write into the FS using fs.writeFileSync() end up being cloned into the IndexedDBFS, ideally that should just be for files that the emulator creates, not for stuff we're writing while getting the emulator stood up [22:06] ok, I think I found something [22:07] if I call game_data.fs.getOverlayedFileSystems().readable.mkdirSync('/foo'): and then run game_data.fs.statSync('/foo'), I get a mode integer value of 16895 [22:08] if I call game_data.fs.getOverlayedFileSystems().writable.mkdirSync('/foo2'): and then run game_data.fs.statSync('/foo2'), I get a mode integer value of 16384 [22:08] 16384 is the bad "no permissions" value, whereas 16895 is what it should be [22:09] I thiiiiiiiiiiink that we can kill two birds with one stone here, fix the "emulator zips and chds shouldn't persist to IndexedDBFS" bug and the "chd file is unreadable" bug at the same time [22:15] sounds like still loader.js, right [22:15] yeah [22:15] I mean [22:16] I think we have uncovered a BrowserFS bug, where creating files on the AsyncOverlayFS isn't assigning right permissions [22:16] but I also think we can work around this relatively easily and get a bonus bugfix out of it [22:17] have at [22:17] (About to record last podcast episode of the month) [22:18] I'm not sure if we're supposed to be creating directories and writing files directly into the MountableFS, I think it just acts like an InMemoryFS at that point which is what we want (and might be why we're creating this InMemoryFS but never actually using it) [22:36] https://github.com/db48x/emularity/pull/89 [22:53] Let's experiment. [22:54] Is there a way to grab the loader.js from that [22:56] Did it [22:56] Now, hopefully this is current to our rest. [22:57] I'm pushing it into the experimenter for external_js [22:57] (Last loaderlab was in February, by the way.) [22:59] https://archive.org/details/arcade_fbaitmc&external_js=1 [22:59] Let's find out together [22:59] [23:00] Well, it fuckin' works, man [23:02] I'm quickly testing a bunch of other things to see if you broke it [23:11] They all seem to be working [23:11] Now begging our staff to put in the fix. [23:14] woohoo [23:15] I'm doing a bunch of external_js=1 tests [23:16] IN THEORY [23:16] I can fix all the playstations now [23:16] So they only load them in once [23:20] By the way, one of these emulations has 10.7gb of game drive [23:20] That should run well [23:25] oof [23:26] I'm having a lot of these blow up. [23:26] Trying to determine why. [23:27] So, obviously, before this point, we didn't have any functioning ones with CHDs [23:27] So they're WAY behind - the new drivers may expect different ROMs, or CHDs, etc. [23:27] Example: Blitz2k blows up with exception thrown: 142058912 [23:32] So I'll just step as I go. [23:46] there's also the whole thing where if your IndexedDB filesystem is full of cruft from before this fix, you may still be getting the broken files [23:50] https://archive.org/details/arcade_vnight&external_js=1 black screen so far but works [23:54] dunno if you read the full notes on that pull request yet, but IN THEORY if jvilk figures out what's causing the bad file modes in the OverlayFS, with a small tweak we could actually cache these 200+ mb files locally and not have to reload them every. single. time.