Breaking down the networking mechanism

[Baker]

The networking aspect of NetQuake and of Quakeworld is bit difficult for me to breakdown. Any good books, web links, internet reference materials or docs to disassemble this stuff.

I get what is happening with all other parts of the Quake engine (memory, rendering, model loaders, video, input, largely understand the sound/audio), but I'm trying to get a foothold into breaking down the network code --- sockets, datagrams, packets, reliable vs. unreliable messages --- even the stuff like sign-on buffer and the handshake.

[Spike]

the 'netchan' is the chunk of code which ensures delivery of reliable or unreliable datagrams to the other end. qw merges unreliable and reliable datagrams into a single udp packet, while nq (even dp7) sends both separately. this is handled by the net_*.c

Socket support is handled internally by the netchan supposedly, net_win/net_wins/net_udp handle this.

net_dgram is the datagram code... as opposed to the magic single-other-person serial/modem code, which is completely different and deserves to die.


the engine itself cares only about reliable and unreliable chunks of svc/clc messages/sizebufs. the demo code is basically an alternative to the netchan.

so input goes into an unreliable sizebuf, it gets sent to the netchan and the netchan forwards it into the dgram code. the dgram code adds sequence and ident info and submits it to the socket code, the socket code sends it to the operating system.
on the server, the engine polls the netchan which polls the datagram which polls the sockets. the packet is checked to make sure its for quake, its sequence is checked to count packetloss and ignore duplicates or packets out of sequence, and its dumped into net_message which is then parsed for its clc payloads via MSG_Read*().

its the dgram code which contains all the bugs really, as its the dgram code which contains the connection process (which is evil and blocks because it is not able to poll iteratively), and the code responsible for separate sockets for each client (nq was written a bit like ftp, with relilable modem/serial links - unreliable just means drop it if you have old unreliable to send and thus it would be really really easy to use tcp instead of udp, with NAT/firewall all the issues that go with it - see ftp nat/firewall rules).
The dgram code also contains the server polling and replies. While these are not 'buggy'

Quakeworld has rewritten netchan for a very good reason.
single sockets fixes nat, and simplifies waiting for activity
combined reliables+unreliables reduces packet rates
challenges on connections means you can't flood servers from spoofed ip addresses.
Quakeworld has no explicit 'dgram' layer as everything is a datagram.
connectionless packets are mostly human-readable.
DP has an lhnet.c which basically provides an nq or qw netchan equivelent. I'm not sure on code compatibility, but it does provide network compatibility if configured that way, as well as far superior server queries.

signon buffers are created each frame and are just blocks of svcs like baselines, static sounds, static ents, initial lightstyles. map stuff that is sent to all. prebuilt and broadcast to all. each one should be kept smaller than a udp packet or reliable packet.

the connection handshake is basically that the client sends a 'gimme a connection' packet to the server. the server directly replies with 'this is the ip:port I will use for you, send your stuff there'. The server then begins to send the serverinfo svc to the source of the connection request from the socket at the address:port it gave. Note that it'll retry multiple times, and send keepalives.

backbuffers are specific to quakeworld, and are 'extra storage' for reliable messages which are longer than the max length. They are flushed in order, and this prevents breaking due to routers with broken udp fragmentation, and without large reliables stalling datagram messages on slow links. They are limited to the max reliable packet size of 1024 bytes, but you get lots at a time, so the total reliable data can be much larger, just with no large individual svcs.

any of that help?

[Baker]

Yeah, it is a good start and probably enough information for me to start dissecting. In the past, I've tried to implement lhnet but failed ... but I have more experience now so maybe I'll get greater traction.

And I guess much of the rest is taking lhnet.c and Quakeworld's "net_*.c" files and looking through them.

[Baker]

[Baker]

[Baker]

Not 100% related to exactly this thread per se ...

In the past, reading more recent DarkPlaces code [last 6 years versions +/-] was a bit challenging [but rewarding].

More recently I've found DarkPlaces code to far more easy to read and understand. I guess there is a certain amount of general "around the engine knowledge" you have to absorb and after you start to get some depth in understanding those areas, you pretty much know what is going on, you just want to see the fine details.

I think it is very fortunate to have so many engine topics with people such as Spike and MH and others digging down into the technical guts of the engine and some of it sinks in later even if it takes a couple of years to understand some of these conversations.

[Spike]

[mh]

[Baker]

[Spike]

[Baker]

[mh]

[Spike]

[Baker]

Back to an earlier post, thanks Spike for the comprehensive descriptive overview of, in particular, the Quakeworld protocol and the DP comments. It is helping me understand what I seeing in the code more quickly.

In the past, I mostly was for the most part only able to dissect others engine work to explain how it worked or summarize the changes but I'm now on the verge of being able to make my own contributions and --- well --- also being able to dissect and explain more advanced modifications.

[Baker]