RMQ Engine ... Sheesh!

[Baker]

When I first used the RMQ engine, I noticed a lot of things and I knew a lot of the planned features and a number of them were noticeable in the game, of course.

Maybe just because of the blog, I expected the RMQ engine to be a minimal change mostly to accomodate BSP2, all the GL extensions and use of them.

But there are a hell of a lot of improvements in the engine aside from those changes. I thought I was done pulling in code that I didn't fully understand from another engine just to see the result of a specific set of changes, but I guess I am wrong.

I also see evidence of at least one feature I didn't know could be done. It looks like RMQ *might* have video capture with SDL somehow. I'm also a bit surprised to see demo rewind in the engine. I thought MH hated that, haha.

While talking about MH's code, I was looking at frames per second again (to measure speed improvements) and loaded up a few engines and did timedemos. On this machine, DirectQ 1.9 did the timedemo so fast it was barely on-screen and this Windows 7 laptop isn't fancy at all. It did like 541 frames per second, well over double of the best results with the other engines I tried. It wasn't the 541 fps, it was the "over double" of other engines that was the shocker.

I opened up the RMQ engine to check on 3 things (to see if dynamic light was correct on rotated brushes, it is. I never noticed rotated brushed were improperly lit by dynamic lights in my engine, never thought to check) ... check the model loader for anything of interest ... and I noticed that in particular FitzQuake is really slow switching between fullscreen and windowed and I knew that both RMQ and Quakespasm have some sort of improvement for that.

I hadn't really expected so much of the RMQ engine had been combed over/upgraded. In a way, it also doesn't surprise me ... but the extent of the changes are way more significant than I would have guessed.

[Baker]

[Baker]

[mh]

I do have a tendency to do that - even while working on something specific, if I see something else I don't like or think could be better I fix it up to the way I think it should be. Bit of a bad habit really, too easily side-tracked (and makes keeping any kind of changelog a total bitch).

The series of TGAs was agreed on as the most portable solution. Portable audio is nasty enough; I'm not touching video. It does sync with sound correctly though.

I still hate demo rewind.

[Baker]

RMQ Gem #2

Determine lighting for static entities only once. RMQ does this in CL_ParseStatics (or whatever).

[Note to self: Don't even bother with that if model is marked with noshadow or fullbright rendering like torches].

RMQ also generates a 4x4 matrix here and does it only once.

[Baker]

RMQ/DirectQ Gem #3:

(An oldy to me since I saw this in DirectQ over a year ago).

Generate smax and tmax for brush model surfaces on load rather than recalculate those all the freaking time

RMQ Gem #4

Player skin code includes a lot of software stuff. Kill.

RMQ Gen #5

Most engines appear to keep copies in memory of map texture data it doesn't actually use anywhere.

RMQ Gem #6

Resample 8-bit textures while still 8 bit textures to gain speed. (I wouldn't do this with alpha 8 bit textures, but I think RMQ special handles those).

[mh]

The static entity lighting thing actually sounds like a bug - it should determine the surface and cache that in the entity, because the lightmap data used by the surface may change. A neater way would be to store out the RGBA values used in the lightmap for each surface style on the point it hits, and run those through the standard d_lightstylevalue[surf->styles[maps]] calculation, but that would inflate the entity size a little. Not too much, just a little. dlights should still be determined every frame, of course.

The matrix remains valid.

These two were necessary on account of the huge numbers of static entities in some maps - what was normally not a bottleneck suddenly became significant.

The 8-bit resample expands to 32-bit at the same time so it's just a single pass over the data whether the texture needs to be resampled or just expanded. I thought it would have been an obvious thing to do.

[Baker]

[mh]

Regarding the performance note in your first post, DirectQ can't really be compared fairly to other engines in timedemos because it bases it's damage/bonus flashes on client time rather than realtime. In a timedemo client time is running faster so these last for less (real)time on-screen, and because they have huge fillrate overhead it naturally runs faster as a result.

If you make the same change to any other engine (cl.time - cl.oldtime instead of host_frametime in V_UpdatePalette) you'll get a good speedup in timedemos too. DirectQ will likely still go faster, but not by so large a margin.

The extra performance is maybe 90% achievable by any Quake engine today, but it does require pushing the hardware requirements higher than some people might be comfortable with (I don't think RMQ pushes them high enough). The other 10% comes from D3D-specific things like vertex buffer usage and locking flags that are clear, sensible, and actually work (GL3.x + has these too, but it still goes to hell with UBOs).

It's a very rough rule of thumb but I'd classify gfx hardware into 4 broad categories: prehistoric, ancient, early-modern and modern. There are some oddities (I'd make Voodoo 2 prehistoric on account of the mini-driver and weird texture memory setup for example) and some omissions (I don't classify the likes of GeForce 3 and 4 as they were interim steps on the way to full high-level shader capabilities), but it breaks down like so:

- prehistoric typically has a single texture unit, much missing functionality (such as blend modes), typically a mini-driver, requires lots of hardware-specific hacks.
- ancient is full GL1.2 to 1.5 support, D3D5 to 7, no shaders, may have hardware T&L, usually has multitexture. Full GL ICD.
- early modern is GL2.0 and 2.1 or D3D9 class, shader-model 2 or 3, full capabilities at that level.
- modern is GL3/4, D3D10/11, GPGPU stuff, 1gb of video RAM, etc.

This is a bit of a long-winded preamble leading up to the main observation, which is that if you aim at a specific hardware level, you can usually expect hardware one level up from it to run well enough. Hardware two or more levels up will be highly unlikely to be running at anything near it's full capabilities. Some ad-hoc informal profiling shows me that most Quake engines are doing good if they hit 60% GPU utilization during a typical scene with uncapped framerates and just letting everything run flat-out. There's just so much stuff piled on the CPU, and so much data constantly being sent over the command buffer, that the GPU is spending so much of it's time relatively idle, and when it's not idle it's always stalling so that the CPU can catch up. And it's normally the faster processor that's capable of doing a better job of what's been instead loaded onto the CPU.

DirectQ hits 100% GPU utilization during the same scene, but lately I've been noticing that I may have gone a little too far in that direction - anything extra added (such as my aborted attempt at GPU lightmaps) has tended to have a disproportionate performance impact. There's some better workload balancing to be done here for sure.

[Spike]

the problem with gl, which isn't much of an issue with d3d, is that even moderatly recent intel cards are in the 'ancient' grouping that you mention. which sucks.
if you're okay with it, you can always have alternate paths for different levels of hardware. doom3 supposedly did a chunk of that with its arb/nv/ati specific backends, for instance.

a major major speedup for bigass1 is running particles in clienttime instead of realtime (not just flashes).
another sneaky trick you can potentially do, which fte does, is to throw the audio mixer into another thread. this gave me a 10%ish speedup in fte, but I doubt other engines would see quite the same rise. 10% is a really disturbing amount, but it all builds up.

reaching 100% gpu utilization is bad, if you're running at 640*480.
If they're only reaching 60% at 1920*1080 then that would be really bad.

I remember getting my laptop to render at 1100fps or so. yay for water shaders. add in a lightmap and the fps drops to half of that. the laptop was okay for drawing stencil shadows, but yeah, texture lookups were a killer. the support for rbsp lightmaps I recently added has me worried enough that I refuse to try it on that laptop.
gpu-based lightmaps are even more fun when you're trying to get them working with q3 shaders and a fixed-function gpu. Its really not pretty.

[Baker]

[Baker]

[goldenboy]

It's like the Incredible Hulk of Quake engines.

Hardware requirements, btw, must reflect the hardware that is prevalent among users. What good is DirectX 11 if only 10% of mankind has hardware that can run it.

[taniwha]

What, 600 million isn't enough?

[goldenboy]

Depends what your standards are.

Within the RMQ team itself, not everybody has DirectX11 capable hardware. Not everybody on the team can even run the mod.

I hope this doesn't come as a shock to you.

Developers have to figure these things into their work. You can't just ignore this, or you're developing inside a bubble.