#jsmess 2016-07-13,Wed

↑back Search

Time Nickname Message
00:12 🔗 SketchCow Yes
04:15 🔗 jvilk has joined #jsmess
04:15 🔗 jvilk SketchCow: you rang? what's up?
04:17 🔗 bai hey jvilk
04:17 🔗 jvilk hey bai
04:17 🔗 bai I think he wants to talk about the feasibility of adding range reads to browserfs, specifically for reading ISO files remotely
04:18 🔗 jvilk ah
04:18 🔗 bai so you could mount an iso image without having to download the whole thing - it would make chunked http requests as byte ranges were accessed
04:18 🔗 jvilk does emscripten pass through ranged reads, though? iirc their FS asks for the entire thing, but maybe that's changed
04:19 🔗 bai hmm....good question. there's files and there's devices - maybe this would be better to expose as a device
04:20 🔗 bai like if we actually exposed a /dev/cdrom or whatever
04:20 🔗 jvilk perhaps.
04:20 🔗 jvilk I haven't looked into how those function.
04:20 🔗 bai I use them a bit for ffmpeg encoding in the browser, I run ffmpeg in a worker and feed frame data to it via a device
04:21 🔗 jvilk oh actually, it does give me an offset and length. https://github.com/jvilk/BrowserFS/blob/master/src/generic/emscripten_fs.ts#L119
04:21 🔗 bai pretty straightforward, I think you just register a device id with a set of functions for read, write, seek, open, and close
04:21 🔗 bai oh cool
04:21 🔗 jvilk so maybe it just passes through the raw read parameters
04:22 🔗 bai even easier
04:23 🔗 jvilk anyway, if that's the case, this should be possible via ranged reads.... but the problem is that those requests would be *synchronous*.
04:23 🔗 jvilk and blocking.
04:23 🔗 jvilk iirc we preload everything into memory before launching the emulators to avoid stalls like that
04:24 🔗 bai hmm, yeah we might need to tweak the list of emterpreted function calls - but from a user experience perspective I think blocking is fine
04:25 🔗 jvilk and if the emulator is doing a lot of small reads in a row, that's a lot of round trips on the network on a single JavaScript event
04:25 🔗 bai since back in the day reading from a cdro at 1x was a blocking operation anyway
04:25 🔗 jvilk no, the browser might kill our JavaScript
04:25 🔗 bai yeah, good point, if it's configured with like, 8k reads
04:25 🔗 jvilk the UI isn't responsive during those synchronous reads
04:26 🔗 jvilk and if it's reading a data structure from the CD-ROM and jumping around based on offsets in the data... could potentially be bad.
04:26 🔗 jvilk with intelligent prefetching, we could reduce the issue to just occasional stalls
04:26 🔗 bai oh, I see what you mean. I believe that with the right emterpreter options we can do them as async though, right?
04:26 🔗 jvilk hmm yeah, maybe? I'm not completely up-to-snuff on Emscripten, but I know you can mark certain functions for the emterpreter
04:27 🔗 bai yeah, it's a lot more of a manual process
04:27 🔗 jvilk if the functions are on the critical path, though, that would be bad news bears for performance.
04:27 🔗 jvilk iirc you have to mark all functions in a call chain to an async function
04:27 🔗 bai yeah. so far it's turned out to not be a problem performance-wise, either for dosbox or mame
04:27 🔗 bai but it depends how deep those calls go
04:28 🔗 jvilk oh? dosbox and mame use the emterpreter?
04:28 🔗 bai for file reads it might be a lot deeper than what we've played with so far
04:28 🔗 bai yup
04:28 🔗 jvilk what for?
04:28 🔗 bai trying to recall
04:28 🔗 bai https://github.com/dreamlayers/em-dosbox/blob/em-dosbox-svn-sdl2/src/emterpretify.txt
04:28 🔗 jvilk I would highly recommend looking into how deep file operations are as a first step
04:29 🔗 bai ah yeah, in dosbox the whole interactive shell thing requires it
04:29 🔗 jvilk ahhh the shell
04:29 🔗 bai without emterpreter you'd get a SimulateInfiniteLoop exception
04:29 🔗 jvilk I forgot about handling that!
04:29 🔗 jvilk yeah, I remember that issue
04:31 🔗 jvilk i'm skimming through mame's code to see where it touches the file system
04:31 🔗 jvilk (or is DOSBOX the primary target?)
04:32 🔗 bai probably everywhere :(
04:32 🔗 bai hmm, good question. I know we have a whole msdos cdrom collection, but I don't know what we have for our other systems. I think most of the non-arcade mame systems like appleIIe and such tend to be floppies and images generally small enough to just preload
04:33 🔗 bai the problem is, even if dosbox/mame were really good about wrapping all file IO in a specific module, emterpreter doesn't care about that, it cares about the stack trace
04:34 🔗 bai so everywhere that leads to a file read needs to be emterpretified (god that's such an awkward word)
04:34 🔗 jvilk I should mention up-front that I probably can't do anything significant until August 15th. Going on vacation for a week and a half on Sunday, and then when I get back I'm trying to make a research paper deadline.
04:35 🔗 jvilk If we an emterpretify the paths to file I/O without much perf loss, then that paves the way for a nice solution in browserfs.
04:35 🔗 jvilk can*
04:36 🔗 jvilk if we cannot do that, we'll have to experiment with synchronous XHR with intelligent prefetch in an attempt to mitigate the badness
04:36 🔗 jvilk synchronous XHR also comes with other baggage: you can't request an array buffer, so I have to request as string *and convert the thing into an arraybuffer*
04:37 🔗 jvilk that's real bad for large reads
04:39 🔗 bai oh really, no arraybuffers in synchronous requests? didn't know that
04:39 🔗 jvilk yeah, they don't let you.
04:39 🔗 jvilk I think IE lets you. But no other browser lets you.
04:39 🔗 bai actually I thought chrome had completely deprecated synchronous ajax requests but maybe that was just for browser extensions
04:39 🔗 jvilk they deprecated them, yes. it prints a warning to the console.
04:39 🔗 jvilk but they are still there.
04:39 🔗 bai ah ok
04:39 🔗 jvilk they may yank them in the future, though -- I'd like to not bank on them
04:40 🔗 bai yeah
04:40 🔗 bai generally not a good thing
04:40 🔗 jvilk unless we move to a webworker! synchronous XHR in a webworker is A-OK
04:40 🔗 bai oh, that was what I was wondering
04:40 🔗 bai but then you're back to async as far as browserfs/emscripten are concerned
04:40 🔗 jvilk how so?
04:40 🔗 jvilk browserfs can live in a webworker
04:41 🔗 bai right, but passing the data from that to the main thread for your emscripten app to use would be asynchronous, right?
04:41 🔗 jvilk oh, I meant the emscripten app would need to be in a webworker with the file system code
04:41 🔗 bai ah I see, the whole thing in a thread. that's a distinct possibility yes
04:42 🔗 jvilk it's the best solution if we really need synchronous blocking reads
04:42 🔗 bai (I need to get out of the habit of saying thread, heh)
04:42 🔗 bai yeah, that's worth considering, I want to do that anyway
04:42 🔗 jvilk I know emscripten once had super experimental webworker support at one point. Dunno if it supports all of the SDL stuff (mouse/keyboard/gamepad)
04:43 🔗 bai yeah at some point a year or so back I had a build which ran in a worker, the most notable thing that didn't work at the time was sound
04:43 🔗 jvilk webworkering would also make crazy things less browser crashy, which is nice
04:43 🔗 bai but both emscripten and browsers have advanced on that front since then
04:43 🔗 jvilk oh crap yeah, sound
04:44 🔗 bai the solution at the time was to just proxy the sound data to the main thread, but now browsers have some support for sound in workers
04:44 🔗 bai I believe it uses the same technique for webgl calls as well
04:45 🔗 jvilk yeah, it has to
04:45 🔗 jvilk mouse/keyboard too, and ditto on gamepad if that's also supported
04:46 🔗 bai I believe that worked but I can't remember if I specifically tested it
04:46 🔗 jvilk who is active these days?
04:46 🔗 bai keyboard and mouse definitely did
04:46 🔗 jvilk (on this project)
04:47 🔗 bai depends what you mean by active...we've mostly been doing content ingestion lately (pretty much all sketchcow) since things have been working smoothly, but DFJustin, db48x, and myself are still poking around at things, and other people still pop in to offer suggestions
04:48 🔗 jvilk cuz I'm totally maxed out for awhile. maybe i'll poke at browserfs stuff as a diversion, but there are non-browserfs things that need to be examined
04:48 🔗 bai dreamlayers is still around and working on em-dosbox but he doesn't come around here unless there's a big problem. we actually had a problem recently with chrome 51 where they broke all the dosbox stuff by messing up some assembly command
04:49 🔗 jvilk yeah, I saw jason's tweet
04:49 🔗 bai yeah, I've had it on my list to make things run in a worker for a while now
04:49 🔗 jvilk the emterpreter hack should be looked into first
04:50 🔗 bai I worry that with file accesses it'll basically end up being the whole thing running through emterpreter. which will probably work, but not as fast as the other stuff which we could be more selective with
04:52 🔗 jvilk if it's in the main DOSBOX interpreter loop, then yeah
04:55 🔗 jvilk I have some hope for DOXBox. It looks like the I/O devices are added to a multiplex list, and then the function that iterates through those to do operations is registered with DOSBox as a "callback"
04:57 🔗 bai hmm, if it's a callback then you're right, that would be nice and clean
04:58 🔗 DFJustin so dosbox would be more attractive in terms of the amount of cd-rom software available but I think dosbox can only read uncompressed ISOs whereas mame does CHD
04:58 🔗 DFJustin which would be a lot better for network transmission
04:58 🔗 jvilk hard to tell statically though. they write a pointer to the callback at a particular memory position?
04:59 🔗 DFJustin since chd uses 7z/flac/etc
04:59 🔗 bai well, web browsers can also do gzip compression the fly
04:59 🔗 jvilk (`src/cpu/callback.cpp`)
04:59 🔗 bai I mean, web servers
04:59 🔗 DFJustin does that work with range requests
04:59 🔗 bai I believe so yes
05:00 🔗 DFJustin also mame is NOT currently emterpreted
05:00 🔗 jvilk I see no reason why not. the server interprets the range request and prepares a response.
05:00 🔗 bai the offsets are on the original file and as far as the browser is concerned the gzip is transparent
05:00 🔗 DFJustin last time I tried it it blew up the compiler
05:00 🔗 bai DFJustin: ah, it's not? I was using emterpreter for some of the save/restore stuff but yeah I never actually sent a pull request for that stuff
05:01 🔗 bai there was a little work needed to figure out the proper list of functions to whitelist yeah
05:01 🔗 jvilk so, webworker + synchronous XHR would be the best long term solution. but it makes deployment trickier / requires changes in emularity / etc
05:01 🔗 DFJustin it's probably worth fighting through to get that going though for other reasons
05:01 🔗 jvilk the emterpreter option is definitely the way to go if the function list isn't too crazy.
05:02 🔗 bai yeah there were several functions which subtly don't work in mame with the way it's set up now, which running it through the emterpreter allowed to work
05:02 🔗 bai like save states, frameskipping
05:02 🔗 bai couple other things which I forget now, it's been a while
05:03 🔗 jvilk anyway, it's late; i should sleep
05:03 🔗 bai ok. we'll poke at that stuff - thanks for brainstorming on this
05:03 🔗 jvilk if either of you feel like looking into it, it would be nice to know how deeply file open/read/etc are called
05:03 🔗 bai yeah
05:03 🔗 jvilk yeah, no problem. it's fun stuff.
05:04 🔗 jvilk I imagine DOSBox would be our first target? for win95 stuff?
05:04 🔗 jvilk anyway, sleep time
05:04 🔗 SketchCow Is there a way for us to test this?
05:04 🔗 jvilk oh hey
05:04 🔗 bai that's a question for sketchcow, I'm pretty sure dosbox is the main target yeah
05:05 🔗 bai oh hi :D
05:05 🔗 SketchCow dosbox, yes.
05:05 🔗 jvilk testing which?
05:05 🔗 SketchCow MAME much, much, much less so
05:05 🔗 SketchCow We can do dosbox-cd and have it work differently
05:06 🔗 jvilk SketchCow: what did you want tested?
05:07 🔗 SketchCow Sorry, sometimes I don't notice the -more- in my IRC client.
05:07 🔗 jvilk (btw this is where DOSBox adds a callback for CD I/O: https://github.com/dreamlayers/em-dosbox/blob/e879975b248564a0be3d90c44196b2f550e6521d/src/dos/dos_misc.cpp#L34 and this is where this module registers a callback to main DOSBox to multiplex over all I/O devices: https://github.com/dreamlayers/em-dosbox/blob/e879975b248564a0be3d90c44196b2f550e6521d/src/dos/dos_misc.cpp#L207 )
05:08 🔗 bai ah nice
05:08 🔗 jvilk and this is where I go oh god it's writing a pointer to a memory address fuck https://github.com/dreamlayers/em-dosbox/blob/e879975b248564a0be3d90c44196b2f550e6521d/src/cpu/callback.cpp#L144
05:11 🔗 bai I'll have to set up a new dev environment, I haven't build em-dosbox in a while
05:12 🔗 jvilk so i'm not entirely sure where the callback is actually triggered from within an application
05:12 🔗 bai yeah, I'll have to trace that
05:12 🔗 jvilk maybe callback is even a misnomer
05:12 🔗 bai I got good at tracing that stuff a while back but have since forgot
05:12 🔗 jvilk yeah, i'm not completely following the logic
05:13 🔗 bai easiest to just run it and set a breakpoint and then just copy that whole stack trace into the emterpretify whitelist
05:13 🔗 bai relink, try again, repeat until it runs
05:14 🔗 jvilk it looks like this maybe does the thing? https://github.com/dreamlayers/em-dosbox/blob/e879975b248564a0be3d90c44196b2f550e6521d/src/cpu/callback.cpp#L60 i'm guessing all of the i/o is interrupt driven? i'm less experience at this level of abstraction
05:14 🔗 SketchCow Fundamentally, the THEORY is using these range grabs means that we'd be able to have much faster grabbing of a CD-ROM and functioning it. CD-ROMs could be "mounted".
05:14 🔗 jvilk yeah, probably a good idea, bai
05:14 🔗 SketchCow So anything we're asking you is just kind of from that.
05:15 🔗 SketchCow That's where this is coming from.
05:15 🔗 bai SketchCow: yeah, jvilk brings up a goodpoint about efficiency though - if an app is built to do, eg, 8k filesystem reads, that'll end up being an xhr request per 8k chunk
05:16 🔗 bai but that's something to deal with once we prove it's even possible to do what we think, heh
05:16 🔗 bai if it works but is sort of slow it'll probably still be faster than preloading a 600mb io
05:16 🔗 bai iso
05:17 🔗 jvilk SketchCow: yeah, I understand. the issue is how these things are doing I/O in the web environment -- they call a JavaScript equivalent of read() and expect the data as a return value. If read() needs to trigger a network request that must finish before read() returns, it locks up the UI while it does that
05:17 🔗 jvilk if you emterpreterify the function calling read(), then you can make it asynchronous and not lock up the UI
05:18 🔗 jvilk similarly, if you webworker the whole darned thing, then the problem goes away -- no emterpreter needed, since worker stuff doesn't block the UI
05:18 🔗 jvilk that's the current issue, in a nutshell
05:20 🔗 jvilk bai: I would totally do larger-block-prefetching & maintain a map of what part of the file I've read
05:20 🔗 jvilk anyway, need sleep.
05:20 🔗 bai yeah, that would be good
05:21 🔗 bai cool man - talk to you later
05:21 🔗 jvilk yup! I'll try to poke my head in tomorrow
05:21 🔗 jvilk *goes afk*
05:27 🔗 SketchCow Anything stopping amiga??
05:27 🔗 SketchCow Or is it just a case of "what the fuck does all this do"
05:28 🔗 bai it's like taking two giant multi-tentacled demons and making them dance hand in hand in hand in hand, you have to match each tentacle up with the right pair
05:28 🔗 bai we have our file preloader stuff, so it needs to cram its data into the amiga config structure, then call AMIGA.start()
05:29 🔗 bai and probably do other stuff to wire up sound, keyboard, etc
05:32 🔗 SketchCow Maybe a shim/powerpack for now in the .js.gz
05:32 🔗 SketchCow fixridiculousshit.js
05:33 🔗 bai yeah, if we can just call a specific function when the files have funished preloading then we can use that to trigger whatever crazy shit we have to do
05:33 🔗 bai just a matter of tracing through it piece by piece, probably take a few sessions to dig through it
06:12 🔗 SketchCow I forgot to mention why CD-ROMs interest me so
06:13 🔗 SketchCow So, I'd been told there was "a collection" out there at the warehouse.
06:13 🔗 SketchCow Some stuff which sounded a tad off but interesting
06:13 🔗 SketchCow Finally got to it, they found it and are repackaging it so it's in better shape.
06:13 🔗 SketchCow 50,000 CD-ROMs
06:13 🔗 SketchCow Many of them utterly unique, professionally printed.
06:13 🔗 SketchCow Tiny runs
06:15 🔗 bai nice
06:17 🔗 SketchCow So basically.... an incentive
14:13 🔗 DFJustin I /thought/ I remembered hearing about a collection like that at the archive
15:26 🔗 SketchCow Yes
15:26 🔗 SketchCow It was not indicated to me how unique and interesting it is.
15:26 🔗 SketchCow Macromedia licensing required sending a copy of the final product to them
15:26 🔗 SketchCow This is that collection, from Macromedia
15:34 🔗 DFJustin ohhhh
15:53 🔗 SketchCow Yes, it is a very big deal.
16:35 🔗 db48x bai: how can I help with the amiga?
16:40 🔗 SketchCow Part of it is that it'sa tangly mess of hoobiehoo
16:40 🔗 SketchCow We're trying two things, I think. Make it work at ALL within Emularity by any means necessary, and then do a round of cleanup so it's not a war crime.
16:57 🔗 db48x it's pretty unlikely to be a war crime even the first time around
17:00 🔗 SketchCow I think it depends how much we just bite the bullet and choose a specific machine type for our prototype
17:00 🔗 SketchCow Also, the creator seemed pretty intent on having 14+ different .js files that are loaded in order,for example.
17:00 🔗 db48x that's easy to fix :)
17:09 🔗 SketchCow Yeah, just needs attention.
17:13 🔗 db48x we'll run it through the google closure compiler and it'll all be sorted
17:13 🔗 db48x 5 minutes work, tops
17:24 🔗 SketchCow https://archive.org/download/emularity_engine_v1/scriptedamigaemulator.js.gz
18:04 🔗 SketchCow No idea why the thing isn't blowing to console or anything
18:04 🔗 SketchCow It just kind of "ends"
18:18 🔗 bai yeah, it's all loaded but is waiting some functions to be called
18:19 🔗 bai what we have to do from here is:
18:19 🔗 bai 1) call SAE({cmd: 'init'})
18:20 🔗 bai 2) set up a bunch of variables in AMIGA.config incliding AMIGA.config.rom.data and AMIGA.floppy.*
18:20 🔗 bai 3) call AMIGA.start()
18:23 🔗 SketchCow Is that something easily done?
18:23 🔗 SketchCow Add a little .js?
18:24 🔗 bai yeah that's what you can help with db48x - we basically need to be able to specify a custom js startup function
18:25 🔗 bai a way to call some function after everything has finished loading
18:47 🔗 db48x yea, it calls callbacks.before_run
18:48 🔗 db48x we'll write an AMIGALoader which sets it
18:48 🔗 bai oh yeah if we make an AMIGALoader then this is all a lot easier
18:49 🔗 db48x :)
18:50 🔗 bai there's a lot of convenience functions that the SAE guy just crams into index.js which we could put into there instead
18:50 🔗 bai seems to mostly be stuff for setting up the rom and floppy data, but there's other subsystems in there too

irclogger-viewer