| Time |
Nickname |
Message |
|
00:12
🔗
|
SketchCow |
Yes |
|
04:15
🔗
|
|
jvilk has joined #jsmess |
|
04:15
🔗
|
jvilk |
SketchCow: you rang? what's up? |
|
04:17
🔗
|
bai |
hey jvilk |
|
04:17
🔗
|
jvilk |
hey bai |
|
04:17
🔗
|
bai |
I think he wants to talk about the feasibility of adding range reads to browserfs, specifically for reading ISO files remotely |
|
04:18
🔗
|
jvilk |
ah |
|
04:18
🔗
|
bai |
so you could mount an iso image without having to download the whole thing - it would make chunked http requests as byte ranges were accessed |
|
04:18
🔗
|
jvilk |
does emscripten pass through ranged reads, though? iirc their FS asks for the entire thing, but maybe that's changed |
|
04:19
🔗
|
bai |
hmm....good question. there's files and there's devices - maybe this would be better to expose as a device |
|
04:20
🔗
|
bai |
like if we actually exposed a /dev/cdrom or whatever |
|
04:20
🔗
|
jvilk |
perhaps. |
|
04:20
🔗
|
jvilk |
I haven't looked into how those function. |
|
04:20
🔗
|
bai |
I use them a bit for ffmpeg encoding in the browser, I run ffmpeg in a worker and feed frame data to it via a device |
|
04:21
🔗
|
jvilk |
oh actually, it does give me an offset and length. https://github.com/jvilk/BrowserFS/blob/master/src/generic/emscripten_fs.ts#L119 |
|
04:21
🔗
|
bai |
pretty straightforward, I think you just register a device id with a set of functions for read, write, seek, open, and close |
|
04:21
🔗
|
bai |
oh cool |
|
04:21
🔗
|
jvilk |
so maybe it just passes through the raw read parameters |
|
04:22
🔗
|
bai |
even easier |
|
04:23
🔗
|
jvilk |
anyway, if that's the case, this should be possible via ranged reads.... but the problem is that those requests would be *synchronous*. |
|
04:23
🔗
|
jvilk |
and blocking. |
|
04:23
🔗
|
jvilk |
iirc we preload everything into memory before launching the emulators to avoid stalls like that |
|
04:24
🔗
|
bai |
hmm, yeah we might need to tweak the list of emterpreted function calls - but from a user experience perspective I think blocking is fine |
|
04:25
🔗
|
jvilk |
and if the emulator is doing a lot of small reads in a row, that's a lot of round trips on the network on a single JavaScript event |
|
04:25
🔗
|
bai |
since back in the day reading from a cdro at 1x was a blocking operation anyway |
|
04:25
🔗
|
jvilk |
no, the browser might kill our JavaScript |
|
04:25
🔗
|
bai |
yeah, good point, if it's configured with like, 8k reads |
|
04:25
🔗
|
jvilk |
the UI isn't responsive during those synchronous reads |
|
04:26
🔗
|
jvilk |
and if it's reading a data structure from the CD-ROM and jumping around based on offsets in the data... could potentially be bad. |
|
04:26
🔗
|
jvilk |
with intelligent prefetching, we could reduce the issue to just occasional stalls |
|
04:26
🔗
|
bai |
oh, I see what you mean. I believe that with the right emterpreter options we can do them as async though, right? |
|
04:26
🔗
|
jvilk |
hmm yeah, maybe? I'm not completely up-to-snuff on Emscripten, but I know you can mark certain functions for the emterpreter |
|
04:27
🔗
|
bai |
yeah, it's a lot more of a manual process |
|
04:27
🔗
|
jvilk |
if the functions are on the critical path, though, that would be bad news bears for performance. |
|
04:27
🔗
|
jvilk |
iirc you have to mark all functions in a call chain to an async function |
|
04:27
🔗
|
bai |
yeah. so far it's turned out to not be a problem performance-wise, either for dosbox or mame |
|
04:27
🔗
|
bai |
but it depends how deep those calls go |
|
04:28
🔗
|
jvilk |
oh? dosbox and mame use the emterpreter? |
|
04:28
🔗
|
bai |
for file reads it might be a lot deeper than what we've played with so far |
|
04:28
🔗
|
bai |
yup |
|
04:28
🔗
|
jvilk |
what for? |
|
04:28
🔗
|
bai |
trying to recall |
|
04:28
🔗
|
bai |
https://github.com/dreamlayers/em-dosbox/blob/em-dosbox-svn-sdl2/src/emterpretify.txt |
|
04:28
🔗
|
jvilk |
I would highly recommend looking into how deep file operations are as a first step |
|
04:29
🔗
|
bai |
ah yeah, in dosbox the whole interactive shell thing requires it |
|
04:29
🔗
|
jvilk |
ahhh the shell |
|
04:29
🔗
|
bai |
without emterpreter you'd get a SimulateInfiniteLoop exception |
|
04:29
🔗
|
jvilk |
I forgot about handling that! |
|
04:29
🔗
|
jvilk |
yeah, I remember that issue |
|
04:31
🔗
|
jvilk |
i'm skimming through mame's code to see where it touches the file system |
|
04:31
🔗
|
jvilk |
(or is DOSBOX the primary target?) |
|
04:32
🔗
|
bai |
probably everywhere :( |
|
04:32
🔗
|
bai |
hmm, good question. I know we have a whole msdos cdrom collection, but I don't know what we have for our other systems. I think most of the non-arcade mame systems like appleIIe and such tend to be floppies and images generally small enough to just preload |
|
04:33
🔗
|
bai |
the problem is, even if dosbox/mame were really good about wrapping all file IO in a specific module, emterpreter doesn't care about that, it cares about the stack trace |
|
04:34
🔗
|
bai |
so everywhere that leads to a file read needs to be emterpretified (god that's such an awkward word) |
|
04:34
🔗
|
jvilk |
I should mention up-front that I probably can't do anything significant until August 15th. Going on vacation for a week and a half on Sunday, and then when I get back I'm trying to make a research paper deadline. |
|
04:35
🔗
|
jvilk |
If we an emterpretify the paths to file I/O without much perf loss, then that paves the way for a nice solution in browserfs. |
|
04:35
🔗
|
jvilk |
can* |
|
04:36
🔗
|
jvilk |
if we cannot do that, we'll have to experiment with synchronous XHR with intelligent prefetch in an attempt to mitigate the badness |
|
04:36
🔗
|
jvilk |
synchronous XHR also comes with other baggage: you can't request an array buffer, so I have to request as string *and convert the thing into an arraybuffer* |
|
04:37
🔗
|
jvilk |
that's real bad for large reads |
|
04:39
🔗
|
bai |
oh really, no arraybuffers in synchronous requests? didn't know that |
|
04:39
🔗
|
jvilk |
yeah, they don't let you. |
|
04:39
🔗
|
jvilk |
I think IE lets you. But no other browser lets you. |
|
04:39
🔗
|
bai |
actually I thought chrome had completely deprecated synchronous ajax requests but maybe that was just for browser extensions |
|
04:39
🔗
|
jvilk |
they deprecated them, yes. it prints a warning to the console. |
|
04:39
🔗
|
jvilk |
but they are still there. |
|
04:39
🔗
|
bai |
ah ok |
|
04:39
🔗
|
jvilk |
they may yank them in the future, though -- I'd like to not bank on them |
|
04:40
🔗
|
bai |
yeah |
|
04:40
🔗
|
bai |
generally not a good thing |
|
04:40
🔗
|
jvilk |
unless we move to a webworker! synchronous XHR in a webworker is A-OK |
|
04:40
🔗
|
bai |
oh, that was what I was wondering |
|
04:40
🔗
|
bai |
but then you're back to async as far as browserfs/emscripten are concerned |
|
04:40
🔗
|
jvilk |
how so? |
|
04:40
🔗
|
jvilk |
browserfs can live in a webworker |
|
04:41
🔗
|
bai |
right, but passing the data from that to the main thread for your emscripten app to use would be asynchronous, right? |
|
04:41
🔗
|
jvilk |
oh, I meant the emscripten app would need to be in a webworker with the file system code |
|
04:41
🔗
|
bai |
ah I see, the whole thing in a thread. that's a distinct possibility yes |
|
04:42
🔗
|
jvilk |
it's the best solution if we really need synchronous blocking reads |
|
04:42
🔗
|
bai |
(I need to get out of the habit of saying thread, heh) |
|
04:42
🔗
|
bai |
yeah, that's worth considering, I want to do that anyway |
|
04:42
🔗
|
jvilk |
I know emscripten once had super experimental webworker support at one point. Dunno if it supports all of the SDL stuff (mouse/keyboard/gamepad) |
|
04:43
🔗
|
bai |
yeah at some point a year or so back I had a build which ran in a worker, the most notable thing that didn't work at the time was sound |
|
04:43
🔗
|
jvilk |
webworkering would also make crazy things less browser crashy, which is nice |
|
04:43
🔗
|
bai |
but both emscripten and browsers have advanced on that front since then |
|
04:43
🔗
|
jvilk |
oh crap yeah, sound |
|
04:44
🔗
|
bai |
the solution at the time was to just proxy the sound data to the main thread, but now browsers have some support for sound in workers |
|
04:44
🔗
|
bai |
I believe it uses the same technique for webgl calls as well |
|
04:45
🔗
|
jvilk |
yeah, it has to |
|
04:45
🔗
|
jvilk |
mouse/keyboard too, and ditto on gamepad if that's also supported |
|
04:46
🔗
|
bai |
I believe that worked but I can't remember if I specifically tested it |
|
04:46
🔗
|
jvilk |
who is active these days? |
|
04:46
🔗
|
bai |
keyboard and mouse definitely did |
|
04:46
🔗
|
jvilk |
(on this project) |
|
04:47
🔗
|
bai |
depends what you mean by active...we've mostly been doing content ingestion lately (pretty much all sketchcow) since things have been working smoothly, but DFJustin, db48x, and myself are still poking around at things, and other people still pop in to offer suggestions |
|
04:48
🔗
|
jvilk |
cuz I'm totally maxed out for awhile. maybe i'll poke at browserfs stuff as a diversion, but there are non-browserfs things that need to be examined |
|
04:48
🔗
|
bai |
dreamlayers is still around and working on em-dosbox but he doesn't come around here unless there's a big problem. we actually had a problem recently with chrome 51 where they broke all the dosbox stuff by messing up some assembly command |
|
04:49
🔗
|
jvilk |
yeah, I saw jason's tweet |
|
04:49
🔗
|
bai |
yeah, I've had it on my list to make things run in a worker for a while now |
|
04:49
🔗
|
jvilk |
the emterpreter hack should be looked into first |
|
04:50
🔗
|
bai |
I worry that with file accesses it'll basically end up being the whole thing running through emterpreter. which will probably work, but not as fast as the other stuff which we could be more selective with |
|
04:52
🔗
|
jvilk |
if it's in the main DOSBOX interpreter loop, then yeah |
|
04:55
🔗
|
jvilk |
I have some hope for DOXBox. It looks like the I/O devices are added to a multiplex list, and then the function that iterates through those to do operations is registered with DOSBox as a "callback" |
|
04:57
🔗
|
bai |
hmm, if it's a callback then you're right, that would be nice and clean |
|
04:58
🔗
|
DFJustin |
so dosbox would be more attractive in terms of the amount of cd-rom software available but I think dosbox can only read uncompressed ISOs whereas mame does CHD |
|
04:58
🔗
|
DFJustin |
which would be a lot better for network transmission |
|
04:58
🔗
|
jvilk |
hard to tell statically though. they write a pointer to the callback at a particular memory position? |
|
04:59
🔗
|
DFJustin |
since chd uses 7z/flac/etc |
|
04:59
🔗
|
bai |
well, web browsers can also do gzip compression the fly |
|
04:59
🔗
|
jvilk |
(`src/cpu/callback.cpp`) |
|
04:59
🔗
|
bai |
I mean, web servers |
|
04:59
🔗
|
DFJustin |
does that work with range requests |
|
04:59
🔗
|
bai |
I believe so yes |
|
05:00
🔗
|
DFJustin |
also mame is NOT currently emterpreted |
|
05:00
🔗
|
jvilk |
I see no reason why not. the server interprets the range request and prepares a response. |
|
05:00
🔗
|
bai |
the offsets are on the original file and as far as the browser is concerned the gzip is transparent |
|
05:00
🔗
|
DFJustin |
last time I tried it it blew up the compiler |
|
05:00
🔗
|
bai |
DFJustin: ah, it's not? I was using emterpreter for some of the save/restore stuff but yeah I never actually sent a pull request for that stuff |
|
05:01
🔗
|
bai |
there was a little work needed to figure out the proper list of functions to whitelist yeah |
|
05:01
🔗
|
jvilk |
so, webworker + synchronous XHR would be the best long term solution. but it makes deployment trickier / requires changes in emularity / etc |
|
05:01
🔗
|
DFJustin |
it's probably worth fighting through to get that going though for other reasons |
|
05:01
🔗
|
jvilk |
the emterpreter option is definitely the way to go if the function list isn't too crazy. |
|
05:02
🔗
|
bai |
yeah there were several functions which subtly don't work in mame with the way it's set up now, which running it through the emterpreter allowed to work |
|
05:02
🔗
|
bai |
like save states, frameskipping |
|
05:02
🔗
|
bai |
couple other things which I forget now, it's been a while |
|
05:03
🔗
|
jvilk |
anyway, it's late; i should sleep |
|
05:03
🔗
|
bai |
ok. we'll poke at that stuff - thanks for brainstorming on this |
|
05:03
🔗
|
jvilk |
if either of you feel like looking into it, it would be nice to know how deeply file open/read/etc are called |
|
05:03
🔗
|
bai |
yeah |
|
05:03
🔗
|
jvilk |
yeah, no problem. it's fun stuff. |
|
05:04
🔗
|
jvilk |
I imagine DOSBox would be our first target? for win95 stuff? |
|
05:04
🔗
|
jvilk |
anyway, sleep time |
|
05:04
🔗
|
SketchCow |
Is there a way for us to test this? |
|
05:04
🔗
|
jvilk |
oh hey |
|
05:04
🔗
|
bai |
that's a question for sketchcow, I'm pretty sure dosbox is the main target yeah |
|
05:05
🔗
|
bai |
oh hi :D |
|
05:05
🔗
|
SketchCow |
dosbox, yes. |
|
05:05
🔗
|
jvilk |
testing which? |
|
05:05
🔗
|
SketchCow |
MAME much, much, much less so |
|
05:05
🔗
|
SketchCow |
We can do dosbox-cd and have it work differently |
|
05:06
🔗
|
jvilk |
SketchCow: what did you want tested? |
|
05:07
🔗
|
SketchCow |
Sorry, sometimes I don't notice the -more- in my IRC client. |
|
05:07
🔗
|
jvilk |
(btw this is where DOSBox adds a callback for CD I/O: https://github.com/dreamlayers/em-dosbox/blob/e879975b248564a0be3d90c44196b2f550e6521d/src/dos/dos_misc.cpp#L34 and this is where this module registers a callback to main DOSBox to multiplex over all I/O devices: https://github.com/dreamlayers/em-dosbox/blob/e879975b248564a0be3d90c44196b2f550e6521d/src/dos/dos_misc.cpp#L207 ) |
|
05:08
🔗
|
bai |
ah nice |
|
05:08
🔗
|
jvilk |
and this is where I go oh god it's writing a pointer to a memory address fuck https://github.com/dreamlayers/em-dosbox/blob/e879975b248564a0be3d90c44196b2f550e6521d/src/cpu/callback.cpp#L144 |
|
05:11
🔗
|
bai |
I'll have to set up a new dev environment, I haven't build em-dosbox in a while |
|
05:12
🔗
|
jvilk |
so i'm not entirely sure where the callback is actually triggered from within an application |
|
05:12
🔗
|
bai |
yeah, I'll have to trace that |
|
05:12
🔗
|
jvilk |
maybe callback is even a misnomer |
|
05:12
🔗
|
bai |
I got good at tracing that stuff a while back but have since forgot |
|
05:12
🔗
|
jvilk |
yeah, i'm not completely following the logic |
|
05:13
🔗
|
bai |
easiest to just run it and set a breakpoint and then just copy that whole stack trace into the emterpretify whitelist |
|
05:13
🔗
|
bai |
relink, try again, repeat until it runs |
|
05:14
🔗
|
jvilk |
it looks like this maybe does the thing? https://github.com/dreamlayers/em-dosbox/blob/e879975b248564a0be3d90c44196b2f550e6521d/src/cpu/callback.cpp#L60 i'm guessing all of the i/o is interrupt driven? i'm less experience at this level of abstraction |
|
05:14
🔗
|
SketchCow |
Fundamentally, the THEORY is using these range grabs means that we'd be able to have much faster grabbing of a CD-ROM and functioning it. CD-ROMs could be "mounted". |
|
05:14
🔗
|
jvilk |
yeah, probably a good idea, bai |
|
05:14
🔗
|
SketchCow |
So anything we're asking you is just kind of from that. |
|
05:15
🔗
|
SketchCow |
That's where this is coming from. |
|
05:15
🔗
|
bai |
SketchCow: yeah, jvilk brings up a goodpoint about efficiency though - if an app is built to do, eg, 8k filesystem reads, that'll end up being an xhr request per 8k chunk |
|
05:16
🔗
|
bai |
but that's something to deal with once we prove it's even possible to do what we think, heh |
|
05:16
🔗
|
bai |
if it works but is sort of slow it'll probably still be faster than preloading a 600mb io |
|
05:16
🔗
|
bai |
iso |
|
05:17
🔗
|
jvilk |
SketchCow: yeah, I understand. the issue is how these things are doing I/O in the web environment -- they call a JavaScript equivalent of read() and expect the data as a return value. If read() needs to trigger a network request that must finish before read() returns, it locks up the UI while it does that |
|
05:17
🔗
|
jvilk |
if you emterpreterify the function calling read(), then you can make it asynchronous and not lock up the UI |
|
05:18
🔗
|
jvilk |
similarly, if you webworker the whole darned thing, then the problem goes away -- no emterpreter needed, since worker stuff doesn't block the UI |
|
05:18
🔗
|
jvilk |
that's the current issue, in a nutshell |
|
05:20
🔗
|
jvilk |
bai: I would totally do larger-block-prefetching & maintain a map of what part of the file I've read |
|
05:20
🔗
|
jvilk |
anyway, need sleep. |
|
05:20
🔗
|
bai |
yeah, that would be good |
|
05:21
🔗
|
bai |
cool man - talk to you later |
|
05:21
🔗
|
jvilk |
yup! I'll try to poke my head in tomorrow |
|
05:21
🔗
|
jvilk |
*goes afk* |
|
05:27
🔗
|
SketchCow |
Anything stopping amiga?? |
|
05:27
🔗
|
SketchCow |
Or is it just a case of "what the fuck does all this do" |
|
05:28
🔗
|
bai |
it's like taking two giant multi-tentacled demons and making them dance hand in hand in hand in hand, you have to match each tentacle up with the right pair |
|
05:28
🔗
|
bai |
we have our file preloader stuff, so it needs to cram its data into the amiga config structure, then call AMIGA.start() |
|
05:29
🔗
|
bai |
and probably do other stuff to wire up sound, keyboard, etc |
|
05:32
🔗
|
SketchCow |
Maybe a shim/powerpack for now in the .js.gz |
|
05:32
🔗
|
SketchCow |
fixridiculousshit.js |
|
05:33
🔗
|
bai |
yeah, if we can just call a specific function when the files have funished preloading then we can use that to trigger whatever crazy shit we have to do |
|
05:33
🔗
|
bai |
just a matter of tracing through it piece by piece, probably take a few sessions to dig through it |
|
06:12
🔗
|
SketchCow |
I forgot to mention why CD-ROMs interest me so |
|
06:13
🔗
|
SketchCow |
So, I'd been told there was "a collection" out there at the warehouse. |
|
06:13
🔗
|
SketchCow |
Some stuff which sounded a tad off but interesting |
|
06:13
🔗
|
SketchCow |
Finally got to it, they found it and are repackaging it so it's in better shape. |
|
06:13
🔗
|
SketchCow |
50,000 CD-ROMs |
|
06:13
🔗
|
SketchCow |
Many of them utterly unique, professionally printed. |
|
06:13
🔗
|
SketchCow |
Tiny runs |
|
06:15
🔗
|
bai |
nice |
|
06:17
🔗
|
SketchCow |
So basically.... an incentive |
|
14:13
🔗
|
DFJustin |
I /thought/ I remembered hearing about a collection like that at the archive |
|
15:26
🔗
|
SketchCow |
Yes |
|
15:26
🔗
|
SketchCow |
It was not indicated to me how unique and interesting it is. |
|
15:26
🔗
|
SketchCow |
Macromedia licensing required sending a copy of the final product to them |
|
15:26
🔗
|
SketchCow |
This is that collection, from Macromedia |
|
15:34
🔗
|
DFJustin |
ohhhh |
|
15:53
🔗
|
SketchCow |
Yes, it is a very big deal. |
|
16:35
🔗
|
db48x |
bai: how can I help with the amiga? |
|
16:40
🔗
|
SketchCow |
Part of it is that it'sa tangly mess of hoobiehoo |
|
16:40
🔗
|
SketchCow |
We're trying two things, I think. Make it work at ALL within Emularity by any means necessary, and then do a round of cleanup so it's not a war crime. |
|
16:57
🔗
|
db48x |
it's pretty unlikely to be a war crime even the first time around |
|
17:00
🔗
|
SketchCow |
I think it depends how much we just bite the bullet and choose a specific machine type for our prototype |
|
17:00
🔗
|
SketchCow |
Also, the creator seemed pretty intent on having 14+ different .js files that are loaded in order,for example. |
|
17:00
🔗
|
db48x |
that's easy to fix :) |
|
17:09
🔗
|
SketchCow |
Yeah, just needs attention. |
|
17:13
🔗
|
db48x |
we'll run it through the google closure compiler and it'll all be sorted |
|
17:13
🔗
|
db48x |
5 minutes work, tops |
|
17:24
🔗
|
SketchCow |
https://archive.org/download/emularity_engine_v1/scriptedamigaemulator.js.gz |
|
18:04
🔗
|
SketchCow |
No idea why the thing isn't blowing to console or anything |
|
18:04
🔗
|
SketchCow |
It just kind of "ends" |
|
18:18
🔗
|
bai |
yeah, it's all loaded but is waiting some functions to be called |
|
18:19
🔗
|
bai |
what we have to do from here is: |
|
18:19
🔗
|
bai |
1) call SAE({cmd: 'init'}) |
|
18:20
🔗
|
bai |
2) set up a bunch of variables in AMIGA.config incliding AMIGA.config.rom.data and AMIGA.floppy.* |
|
18:20
🔗
|
bai |
3) call AMIGA.start() |
|
18:23
🔗
|
SketchCow |
Is that something easily done? |
|
18:23
🔗
|
SketchCow |
Add a little .js? |
|
18:24
🔗
|
bai |
yeah that's what you can help with db48x - we basically need to be able to specify a custom js startup function |
|
18:25
🔗
|
bai |
a way to call some function after everything has finished loading |
|
18:47
🔗
|
db48x |
yea, it calls callbacks.before_run |
|
18:48
🔗
|
db48x |
we'll write an AMIGALoader which sets it |
|
18:48
🔗
|
bai |
oh yeah if we make an AMIGALoader then this is all a lot easier |
|
18:49
🔗
|
db48x |
:) |
|
18:50
🔗
|
bai |
there's a lot of convenience functions that the SAE guy just crams into index.js which we could put into there instead |
|
18:50
🔗
|
bai |
seems to mostly be stuff for setting up the rom and floppy data, but there's other subsystems in there too |