Making the game load faster...

[ChronoSeth]

Whenever I run DirectQ or Darkplaces while using high-resolution model textures, they tend to take a very long time to load. The performance of the game while I'm actually playing, however, seems unaffected. Is there anything I might be able to do to speed it up?

PS: I've noticed that I don't have this problem with the map textures.

[LambentHammerBurst]

Nothing really you can do except to lower the resolution of the textures or change the format to a more compressed (like jpg). However doing that can cause issues as many models do use alpha channels which jpg doesn't support.
If you are using Reforged's textures and are using the ultra, try to high instead and that will help with loading time.

[MH]

Loading most compressed formats (but see below on DDS files) is going to be slower as they need to be changed to an uncompressed representation before they can be uploaded to video memory (they will load faster from disk though, but they need more processing and CPU time in memory). Loading anything from a PK3 file is going to be slower as the data needs to be decompressed first. If you're loading a compressed format from a PK3 file it's going to be doubly-slow as there are two decompression steps involved.

The best thing to do is use something like PakScape to convert PK3 files to PAK files. That will skip one decompression step which may help.

If you're not using PK3 or PAK files for high res textures (i.e. you're loading them from a "textures" subfolder) then putting them into a PAK file can help as in theory file I/O operations at the OS level should be faster.

Additional info.

Many high-res packs contain textures that are - IMO - stupidly high-res. I personally don't believe that replacing what was a 64x64 image with a 512x512 image is reasonable. This image has a surface area 64 times as large as the original, so it will take maybe 64 times longer to load. Resizing them to 256x256 or even 128x128 makes sense here.

Many high-res packs contain textures that are NOT power-of-two sizes (2, 4, 8, 16, 32, 64, etc). If your 3D card doesn't support non-power-of-two texture sizes then most engines must either resample or pad these to sizes that are powers of two (otherwise they may not be supported by your 3D hardware). This is also a slow operation.

I don't know about DarkPlaces but I do know that with DirectQ I used the D3D helper libraries to load most external textures. What goes on behind the scenes with these is outside of my control, but they do have the advantage that you don't need to install extra DLLs on your machine, and I get to provide texture loading for a whole pile of exotic and non-exotic formats in one line of code. So there might be a convenience vs performance tradeoff involved there.

I also do know that both DarkPlaces and DirectQ support loading DDS files which should be the fastest format of all to load. DDS is just a file format and can be loaded by both OpenGL and D3D (Doom3/Q4 uses it too). Advantages are that these files can be compressed to a format natively supported by your graphics hardware and that they can contain pregenerated mipmaps, so they can be loaded more or less directly without going through any intermediate processing or steps. Converting textures to this format might be a good idea if you want the fastest loading.

[Seven]

Hello,

the only thing I want to add to MH�s answer is, that grave_digga discovered it too
and therefore proved that MH�s words are correct.
Read from here on.

Using a .pk3 has a benefit in arranging things, but it doesnt help you fps- and loading-wise.

Just one little thing makes me confused:
MH said here, that the .pk3 support in DirectQ is a little primitive and that its content
must be decompressed in a temp folder first. That needs much time.
Maybe DP handles this process different ? So there MIGHT be a difference/advantage in DP.
I know that DP has .pk3 archive support with compression support using zlib.
This has been written by Elric.
I dont know what is behind it of course. My knowledge ends here.

Anyhow,
a nice topic to discuss and a proof, that .pk3 is NOT always good and has its weaknesses.

Thank you for this thread and information.

Kind regards,
Seven

[LambentHammerBurst]

Yeah I previously ran my Darkplaces with ALL of the content extracted from the pk3's. I did notice a slight increase in loading times (not gameplay FPS).
However this was up until Feb and then removing Rygels textures and installing QRP's and then getting out Reforgeds textures for monsters I've already replaced - it became a mess. So pk3's are great for organization and easy removal.

[MH]

Just one little thing makes me confused:
MH said here, that the .pk3 support in DirectQ is a little primitive and that its content
must be decompressed in a temp folder first. That needs much time.
Maybe DP handles this process different ? So there MIGHT be a difference/advantage in DP.
I know that DP has .pk3 archive support with compression support using zlib.
This has been written by Elric.
I dont know what is behind it of course. My knowledge ends here.

It's possible to decompress directly into memory rather than to a temp file, which can speed things up quite a bit. When I ported my PK3 support from Q3A I didn't complete this final step but took a lazy way out instead, one of the reasons being that some of Quake's file I/O needs rewriting to support it properly.

PAK files are just as good as PK3 for organising stuff by the way, if a little more awkward to create and add files to. But they are always going to be much faster to load from, even leaving aside my temp file thing as a factor.

[ChronoSeth]

Thanks for the tips; I'll play around with stuff a bit.

I should really pay attention to the file sizes of the skins I download... those things were just ridiculous. o_O

[Spike]

just because its a zip doesn't mean the files inside have to be compressed. if they're not compressed then the engine can just read them like it can from a pak. even q3 doesn't compress the roq/video files inside its pk3s as different files can have different compression ratios, and compressing these doesn't give any real benefit.

my personal experience regarding load times is that if an engine is hitting the disk for 36+ files per texture, in 4+ game paths, its going to be slow. FTE achieves this by building a hash table of the available files and then only actually opening them if they really exist. With this feature disabled, it'll try to open stupid numbers of files which always gives terrible performance in windows. Linux doesn't mind nearly as much. It is much much faster with the feature enabled.
Point is, the more paks, gamedirs, etc, you give an engine, the larger your paks are, etc, the more filenames the engine/system needs to scan though to find the file that was asked for. Most quake engines do a simple search when opening files within pak files - a binary search would be more efficient and scale better when you have a lot of different textures/paths/etc in each pak (fodquake implements this). This applies whether the files are actually used by the engine or not, that is, bumps/lumas/links/etc all slow down loading even in an engine that doesn't support them, although the effect is greater with the number of files it tries to open.

[MH]

Yeah, DirectQ does something similar, scanning the gamedirs for available textures and building a list of them, for much the same reason. It's strcmp rather than a hash table, but it does have a table of the first char of each texture name so that it can jump straight to the start of the area to search.

I've reviewed the PK3 code and it actually does use memory instead of disk for it's loader, using the technique described here: It's only temporary - Larry Osterman's WebLog - Site Home - MSDN Blogs - I'd totally forgotten about that!

It does read from the PK3 in 1k blocks however, so it could be sped up by increasing that block size.