Quakespasm on Steriods

[Baker]

22) "its far easier to just shove a pk3 into the gamedir" -- For a lot of stuff like single player releases, etc. I could see that working ok. For a coop mod that sporadically adds maps (FVF is an example) that sometimes have .lits or skyboxes, it'd be a bit incomplete.

Good solution for some things. Not good solution for others. You'd probably suggest all q1 maps be in a .pk3 like q3 anyway, haha. (If I recall how q3 works, I haven't loaded up a custom q3 map in a long, long time.)

23) bspx -- I thought I saw something about that. But I've been mostly looking at network stuff (and particle stuff).

/ re: other items .... looks like a few things I want to try out and test

[Baker]

23) sv_protocol -15 will unconditionally disable additional extensions, resulting in a compatible demo, for certain definitions of compatible...
The demo that I recorded with R4 worked. Or maybe it had a couple of extra tweaks, but I don't think it did.

Ha! It's perfect

It works just fine. Recorded game in protocol -15 plays back in GLQuake and recorded game in protocol -666 plays back just fine in FitzQuake.

I don't complain about other engines often, but I have always found it annoying that to the best of my knowledge it was impossible to record a Quake compatible demo in DarkPlaces, even if you use the Quake protocol.

This means all the demo editing tools are useless.

This is awesome!

Total backwards and forwards compatibility is the killer feature that any good engine should have. At least as an option.

/Now I head back to acquiring the networking stuff ... I noticed you kept both net_wins.c (winsock) and net_udp.c (non-winsock, obv) and Mark V does not have net_wins.c, so ... slight integration slowdown there. I don't think FTE has net_wins.c either, haha The different between winsock and bsd sockets is pretty thin.

[Spike]

I did wonder about adding pk3 support. it'd be easy enough to add support for non-compressed files, but as soon as there's any compression the rest of the engine wouldn't be able to cope, and rewriting half the engine to not try to do direct file access didn't seem worthwhile.
providing maps inside packages ensures that users will have all the files that they'll need in order to actually use that map properly, without lots of extra random files all over the place.
using pk3s means that any downloads of the file are (typically) pre-compressed, so the engine doesn't need any extra fancy stuff (and the user doesn't need to extract anything etc either, just copy it in place). assuming the engine supports decompressing stuff, anyway.
side note: fte is fine with nesting packages, pk3s in paks in pk3s. which is fun. that said, if you do put a pk3 inside a pk3, its better to 'store' rather than 'deflate', so that it can still seek through the file without needing to decompress the entire thing just to find a single file. solid archives are not good for performance.

[Baker]

Random thoughts:

24) Listing server twice. Can't ipv6 connect to ipv4 just fine always? Otherwise wouldn't a fair chunk of the internet be unavailable to IPv6? You are way more ipv6 knowledgable than myself. But as I've always understood it, the problem is that ipv4 can't connect to ipv6 (without tunneling or such, and I make no claim to know too much about that beyond the very basics).

25A) Is there way to prevent servers that DarkPlaces can't connect to from showing up in DarkPlaces? Random thought. You have far more knowledge of dpmaster than I.

25B) DarkPlaces user gets gibberish if tries to connect to a 666 server. Is way for failure to be a little better? Again, on this topic, your knowledge >+++ mine.

UPDATE to 25B: It appears if I start up a server on FitzQuake 0.85 using protocol 666 and try to connect DarkPlaces, it just sits there with an icon. So I guess DarkPlaces doesn't check the protocol number to see if it supports it. So this may not even be possible.

[Spike]

24) ipv4 and ipv6 are separate and distinct. there is no way to find out if a server is the same one other than by sending one a message and then asking the other one to reply. Its theoretically possible for a server to calculate some unique random number and for the server to advertise that to both ipv4 and ipv6 queries, and then have the client de-dupe, but that's open to possible exploits.
Or in other words, its complicated.
Note that a server can be listed on multiple different ipv4 addresses, even without ipv6 being involved, thanks to the server no longer refusing to use more than one interface. Which is kinda required if you have both a wifi connection and an ethernet connection (at least if they're connected to different networks), or whatever.

25A) Don't subscribe to the master servers claiming to use the same protocol as DP (which it doesn't by default). Use a different port from 26000 so that local broadcasts can't discover it. Both of those are required.
Alternatively, just use a base protocol that DP can understand.

25B) other than staging a coup and taking over the DP source repository, no.
the only other solution here is to not use 666 or 999. FTE-15 will get you quite far in that DP will be treated the same as eg winquake, while not harming qss. However this also affects fitz+vanilla-qs+directq+markv...
That said, if you can find some way to reliably detect DP so that it can be given a more sensible svc_print+svc_disconnect message then you could avoid the gibberish, you could try 'cmd enginecheck $somedpspecificcvar' for that, but there arn't actually cvars any that definitively identify it as DP (many cvars that sound like they might be DP-only might be implemented by FTE too, for instance, although I suppose FTE will advertise itself properly anyway, so meh).

Either way I see it as a bug in DP, rather than something that qss should really care about, it doesn't change the fact that if the client is being sent 666 or 999 data then it won't work when the client doesn't actually support that.
The rest is cosmetic, and users will just end up associating DP with gibberish...

If I didn't care more about vanilla quakespasm than I did for random other engines that someone crazy might be using, then I would have reverted to 15 as the default base protocol, which would avoid all these issues in exchange for also limiting qs+markv+fitz+directq+etc.

[Baker]

I've got ipv6 running on Mark V both as client and as a server to QSS on the first compile, by only taking the ipv6 pieces and (surprisingly) ignoring QSS's net_wins.c entirely which I planned to dissect later on (but Mark V's net_udp.c has #ifdef winsock stuff).

After some comparing and contrasting ...

A) Seems that the default DarkPlaces Win32 build (May 13 2014) + The current DarkPlaces Autobuild doesn't see ipv6 only servers, so servers aren't listed twice.
B) Typing "connect [fe80:0:0:0:189d:cdc1:ddf0:8525%11]" in DarkPlaces doesn't work. Removing the brackets doesn't work either. And removing both the brackets and the %11 doesn't work.

So not entirely sure how DarkPlaces ipv6 support works, wanted to compare and contrast. FTE NQConnect [fe80:0:0:0:189d:cdc1:ddf0:8525%11] worked fine, obv.

/End observations. Probably on to Wave 2 (and 3 and 4?? hehe)

[Baker]

Trying to avoid 2 discussions on 2 different boards about ipv6:

re: "ipv6 isn't hard"

ipv6 isn't hard, but making a "completed/testing implementation" is hard. Or at least time consuming -- mistakes, planning, testing, things that have to be rewritten because you forgot to think about them. Plus while I'm aware of the sockets stuff, it isn't crisp in my head right now (like ah, recvfrom is different on winsock vs. bsd sockets, etc. and all those little differences.)

With Quakespasm Spiked as a reference, I don't have to do any of the time consuming parts of the development process.

I've written ipv6 socket code code before, but with any code the demon is always in the details and the trial and error of doing a completed implementation (and debugging it).

In this instance, I didn't need to avoid any demons. Or very much thinking.

Yeah, single port server --- next on the list.

That one is a masterpiece

[Spike]

fe80::/10 means its a link-local address.
with ipv6, link-local addresses require a 'scope id', this is what the % is. the 'scope id' is basically some system-defined id that tells the socket api which network interface to actually send the packet over. so that's what that's about.

I've no idea why dp wouldn't want to connect... try [::1]:26000 for loopback, if you care.

and yes, both net_wins.c and net_udp.c is annoying (and yes, I just typoed it as net_dup.c).

now you need to make the menus and ip reports pretty... and fix bans...
Be sure to mask ip addresses appropriately, and watch out for mods that will try hacking those masks in themselves then crashing because the offsets are all wrong and the dots arn't even in the address any more. Or totally ignore proquake, that works too. :s

[Spike]

Also, are you making a diff for before+after each of these individual features? one of the things that annoys me about qss is how it ended up as a single patch for everything.
I'm not really disciplined enough to keep stuff separate... :s

[Baker]

Also, are you making a diff for before+after each of these individual features?

Of course

[Baker]

Found a weakness in the network code as I was reviewing it.

Same computer ...
1) Start a listen client
2) Start another client + connect (we'll call this client #1)
3) Start another client + connect (we'll call this client #2) ... connects fine. Client #1 gets disconnected.

Doesn't matter if connections are both via local ip (192.168.1.x) or both via internet (using my external ip).

Looks like an address compare issue somehow. Like the remote port isn't part of the comparison, but the code looks like it does do that. Or at least intends to.

/Frikbots works fine.

/dp ipv6 ... [::1]:26000 ... yeah that works fine. It's also pretty nice that QSS can do "connect 127.0.0.1" ... cba to look through the code to see why doesn't work in GLQuake or any other normal NQ engine.

[Spike]

_Datagram_ServerControlPacket:
ret = dfunc.AddrCompare(clientaddr, &s->addr);
if (ret >= 0)
change that ret>=0 to ret==0

the original nq code did the same. vanilla clients pick a new port on each reconnect, so there is no real way to identify actual reconnects, which leaves stale connections that need to wait for timeout if you spam reconnects. which can be a problem.
damned if you do, damned if you don't. support for multiple people behind a single nat probably trumps the stale connections issue.

regarding localhost, the new code doesn't bind to any specific address, neither in the server nor client, so there's no restrictions on which interface packets can be sent through.
note that if you're resolving ipv6+ipv4 addresses in the same getaddrinfo call, then it'll typically resolve localhost as ::1 instead of 127.0.0.1, which is typically problematic... or would be, if engines actually listened on localhost anyway.

note that ipv6 multicasts generally don't get received by the local host, which can be annoying. should probably both multicast and unicast to ::1 as a way around that.

[Baker]

1) "damned if you do, damned if you don't. support for multiple people behind a single nat probably trumps the stale connections issue" -- imperfections and ye old ProQuake "net_connecttimeout 10" -- which I obv noticed in the code.

2) "the new code doesn't bind to any specific address, neither in the server nor client" --- I was thinking that was why it worked *using INADDR_ANY" instead of the result of gethostbyname. Although --- and I could be wrong about this --- I think the server is binding to a specific address at least on Windows in QSS.

I see gethostname in the Winsock network code (net_wins.c) in initialization (but not in net_udp.c) --- if it doesn't do that, the server can't know its own local ip address. Like if I were to type "status", it would have no idea that the address is 192.168.1.x because just opening a socket without filling that in would get you a localhost (127.0.0.1)

--- Sometimes I kind of wonder if your Mount Olympus levels of network knowledge make listening to my mere mortal level comments/thoughts seem annoying

At least you know I've been paying attention to the stuff you've said over the years and have taken it upon myself to boost my own knowledge up the ladder several rungs by writing network code and doing beej tutorials and working with TCP/UDP in both blocking and non-blocking not just on Windows. I'm no level 100, but level 7 or 8 is way better than level 0.

Anyway, bit piece by piece in the time I have available, I've been disassembling and reassembling (and versioning).

What you have done here ranks pretty high on the all-time awesome scale.

[Baker]

Possible not-ideal code found in R4 I've noticed:

1) sv_main.c: SV_StartParticle .. if (sv.datagram.cursize > MAX_DATAGRAM-16) // Isn't considering client limit
2) sv_main.c: SV_StartSound ... same

There may be some other place that per client limits are put into effect that I haven't found yet, but if so I think that could truncate a specific write.

/Or maybe not sv.datagram looks like it is "all or nothing" depending on if it fits when comes to sending it to the client.

[Spike]

sv.datagram is the unreliable broadcast buffer. if its too big for that client (after entities etc), then the entire unreliable buffer is discarded and the game tries to go on without it.
its not fatal if it overflows, and even if the buffer itself does overflow, then adding it to the client's unreliable message with all the various entities already in it, then that per-client buffer is likely to overflow anyway. so yeah, its not ideal, and the only real way to deal with it properly would be to use multicasts instead.

any attempt you make to properly clean it up will be stomped all over by qc's writebyte+etc builtins stomping all over it without any indication of when a message starts/ends, and you really don't want to have to do the whole preparse crap that FTE does to try to deal with that, sometimes the cure is simply worse than the disease (I can justify it in FTE thanks to the nq<>qw translation that it also does).