(Multi-) Threading

[Baker]

[Spike]

[Baker]

I'm using pthreads http://sourceforge.net/projects/pthreads4w/

And I'm slightly irritated that using the .lib requires using the .dll too. The dll is rather small. I can't think of any good excuse for that I have a rather passionate dislike for having to distribute any .dlls with a binary.

I know I dislike debugging on the Mac, XCode isn't anywhere close to as nice as Visual Studio so using Win32 pthreads would minimize debug time in XCode.

In time this will solve itself.

This may eventually be an alternative: http://locklessinc.com/downloads/winpthreads.h

[revelator]

i do have patches for linking to a static pthread library if you want they are mostly intended for windows but with a little work they could probably be modified to work for other arch's to.
btw .dll ? i hope you mean .so as unix archs are not compatible with the windows dll arch normally.
Oh and yes i didnt mean doom3 specifically just pointed at it for some hrm pointers into how it could be done hehe.

[Baker]

[revelator]

http://sourceforge.net/projects/cbadvan ... z/download

patches for the mingw64 winpthreads library, not sure if they also apply to posix systems but i have the patches for the original pthreads library somewhere on my harddrive so ill upload them as soon as i can find them.

to link statically with pthreads use -DWINPTHREAD_STATIC in cflags and or cxxflags. also rename the libpthreads.dll.a library to libpthreads_s.dll.a to avoid autotools trying to link with the shared version anyway.

same goes for the original pthread library when i dug up the patches.

[revelator]

original pthreads just need to have -DPTW32_STATIC_LIB in cflags and or cxxflags, static support is allready working without patches.

[Baker]

Hehe, yeah. [I did some more reading of docs after you had posted the static lib thing.]

According to the docs, pthreads for win32 has had this maybe 4 years but it doesn't exactly stand out in the docs.

I guess on Unix systems (which seems likely to include the Mac as well) dynamic linking for pthreads is the norm. On Windows, this is highly undesirable since it isn't like Windows is going to have a pthreads dll sitting around.

[revelator]

depends i would say, allthough i seen a rising tendency of the unix guys to prefer dynamic loading because it simplifies linking quite a bit.
Downside is that you end up with having a dependency on a ton of shared objects and if really unlucky dll hell.

[Baker]

[Spike]

personally I use mingw for public builds as a way to ensure the GPL is honoured correctly. they're built from svn, and thus I can ensure that others can build an equivelent build (really just randomized register allocations that might prevent exact builds).
its not strictly needed, but it does guarentee my obligations under the gpl are followed.
when it comes to visual studio, you tend to find that you need to hack everything in order to get stuff to work (like throwing stdint.h into system header locations etc).

dynamic linking is often favoured on linux because it means the system packaging system can provide updates for your dependancies (so eg a buggy libpng is actually fixed without your intervention), it also helps solve the malloc/free mismatch issue when libc is dynamically linked.

static linking is often favoured on windows as a way to avoid the whole msvc runtime installer nightmare (I get 'is not a valid win32 program' ? YOU GAVE ME A VIRUS!). the 'fix' for this forces the malloc/free mismatch issue, and the easy way to solve that is to then just statically link _everything_.
on the other hand, dynamically linking (c api) libraries which correctly provide destructor functions means that calls from different dlls can use the correct TLS entries. This sort of thing can be important when it comes to threads, which is likely why the dll form is favoured enough to require people to compile their own.


dependancies only suck when they're likely to be missing, otherwise dynamically linking to other people's libraries is only ever a good thing.

[revelator]

with gcc i often find this to be an annoyance even though it makes sense.
Problem here is that when an update to the respective libraries finally makes it in they are pretty consistently versioned differently,
so forcing me to recompile every damn package that normally depended on that library's dll to avoid trouble.
Autotool based build systems can actually be coerced to replace the library version with the old version number to get around this,
but it causes problems further down the line because sometimes the exports changed a bit so you cannot guarantee that everything works after.

Newer the less its the right way, but its damn annoying .

Btw you can sometimes get lucky with a mingw built static library by just renaming the libpthreads.a library tp pthreads.lib you can sometimes link directly to it with msvc.
sometime you need to also link to libgcc.a where you can use the same trick by renaming it to gcc.lib, just make sure its from the gcc version that built it.
Debugging will be rather nasty though as the mingw built libraries will have incompatible debugging information (dwarf / sjlj) as compared to msvc's.

[Jay Dolan]

Hm. I've spent a fair amount of time adding threading to Quetoo (previously Quake2World). Let me offer a couple tips based on what I've found:

1) If you're already using libSDL, then you should use SDL_Thread. It provides portable thread creation, non-busy waiting, mutexes, conditions, etc. Very easy to use, and works across the board. Win.

2) BSP recursion (cmodel.c) can be made thread-safe by adding just a few thread-local variables (__thread keyword for GCC and Clang). Basically, there's a global trace state structure that is common to the various tracing functions that are called throughout the recursion. By making that state local to each thread, you're then free to run multiple threads through CM_BoxTrace in parallel. Be careful if your engine has any optimizations that cache things like "current trace number" or "check count" on BSP nodes or leafs, as these will have to be moved into that shared thread-local state bucket in order to be made safe. But by and large, making cmodel.c thread-safe isn't too hard.

3) Parallelizing the Quake engine to any measurable benefit is actually really difficult. This is why tech3 and tech4 use only a pair of threads, and basically run the game in one thread, and draw everything in another. If you parallelize at a more fine-grained level, any benefit is lost on the overhead of coordinating the threads. So the right way to do, as far as I can tell, is to encapsulate an entire renderable scene into a struct, and flip-flop between two instances of that struct at each frame (think of how GL's front and back buffers work). This way, you run a frame of game logic in one thread, populating the scene, and then render the result of that frame asynchronously in the other. Note that doing this safely will require that you literally copy objects into the scene struct. Holding pointers to things that the main thread will modify would be a recipe for crashes

Quetoo doesn't quite work this way as of yet. Instead, I've tried to parallelize scene population itself. So, for example, all particle thinking and entity culling happens in one thread while the main thread recurses the BSP and prepares it for rendering. So far, I've only had minuscule and inconsistent gains using this method. Which is what leads me to believe that 3) above is the way to go.

[revelator]

SDL would definatly be number one for consideration also nice to see someone who has actually tested things a bit with threaded optimization.
I agree with the whole thread coordination issue, its damn hard to get it running to a point where it makes a damn,
if done right it could be worth it though but the ammount of work included might defer some from trying .

[Baker]

I've sure SDL threads are awesome.

But I occasionally write console apps and other miscellaneous things and SDL isn't a good fit for that, for me personally because of this.

The Win32 pthreads is awesome if it didn't require a DLL would be perfect, but as it stands that creative .h file I linked to earlier in the thread may assist me in platform neutralizing to pthreads. I'll probably live with the pthreads .dll until I get some nice stuff using it.

SDL is getting to be quite nice, and was always great for sound (who wants to learn the ins and outs of sound mixing? )