packet fragmentation on modern routers

[qbism]

Is packet fragmentation over wifi still a big issue as it was 5 or 10 years ago? I've been reading about the long-standing 1400byte +/- packet limit and wondered if it is still relevant. For example, do most new routers support frame aggregation and/or 'jumbo frames' of 3000 bytes?

[Spike]

the larger a packet, the more likely you'll get interferance corrupting it, resulting in loss.
one way around this is to just repeat the packet multiple times. of course, this will increase packetloss on wired links.
both FTE and DP have network protocols that resend current state on loss. the server needs a journal of what was sent in each packet and the protocol needs a way to track loss somehow (a gap in ack sequences or something), but the reduction in data can be significant while large scenes can send entity data round-robin or so (as the previous frame is not instantly bad). on the other hand, the quakeworld protocol will keep resending data until its acked, and may have the best performance on a lan, but as it deltas from the last-known-received state, both ends need quite extensive state logs.

the ethernet packet limits remain as de-fragmentation can be used for ddos attacks.
in theory, routers are not meant to need to defrag packets, only the final receiver does. however, any port numbers are present only in the first fragment, and thus NAT boxes *MUST* have some way of tracking that, and the cheesy way to handle that breaks if you have packets coming from multiple sources/routes.
With wide-spread carrier-grade NAT on the horizon (combined with home/private NAT boxes), expect the worst.

minimum ipv4 fragment size is something around 576(-headers) bytes. the standard permits silently dropping any fragments smaller than that regardless of how they arrived, and fully blaming the sender for it never arriving. hurrah. any link that requires fragmentation smaller than that *MUST* support defragmenting before sending over ethernet or whatever (typically such extra fragmentation happens at the hardware/driver layer rather than the router's kernel).
if an application generates packets larger than this limit, there is a chance that such packets will be fragmented and a chance that the receiver/NAT will drop all fragments.
ipv6 has a higher minimum, but I don't remember it.

[qbism]

Thanks, I'm starting to understand the difference between the MTU (1500 bytes for ethernet), the minimum reassembly size (576 bytes IPV4 and 1500 bytes IPV6), and the absolute minimum datagram size (64 bytes).

Some protocols use various types of compression to squeeze as much as possible into the limit but I'm not clear on whether one scene/frame can be split into smaller chunks before sending. 'Not instantly bad' - referring to large messages that are likely fragmented? Or are these frames chopped into multiple packets by the server?

[Spike]

in terms of quake payloads 1448 is the lowest I've seen needed with quakeworld servers (1446 for qw clients).
1448 should also apply to both nq reliables and unreliables (either direction).
which is 16 lower than the theoretical ethernet cap

1500(ethernet payload limit)-20(ip)-8(udp)-8(quake) = 1464
additional costs:
pppoe(oa) = 8
ipv4 options = 40 (max)
vpn = 36-80

so this 1448 may have been caused nested pppoe headers, or pppoe combined with some ipv4 option. or something else, I have no idea. all I really know is that 1450 (vanilla quakeworld's default) wasn't enough.
for those few packets where the server was trying to resend 1450 bytes of data, the connection died.

[Baker]

There is a minimum size?

That seems counter-intuitive ... the sign-on handshake sends very small amounts of data.

If so, how does a client that sends only a small amount of data (wish velocity, buttons, impulse, angles) work? 99% of the time this is going to be under 50 bytes or very nearly so.

Or am I misinterpreting this? Or is something hidden going on or something I didn't notice happening (padding somewhere? An unnoticed delay somewhere?).

[qbism]

[Spike]

ethernet runt packets are not a real concern. they should common enough that hosts will pad as required.

if you're trying to keep things aligned, I'd aim for 43-byte alignment as that's the ATM cell size.
10 sounds like quakeworld/q2's netchan protocol size - two 32bit sequences and a 16bit 'qport' value for client->server packets.

[Jay Dolan]

Sorry for necro-posting, but I've actually been experimenting with these values in my engine (Quetoo) just recently.

Interestingly, I've found that my own home network (ADSL NAT'ed behind Apple Airport) is happy to transparently fragment and reassemble UDP packets all the way to Rackspace hosting.

To test this, I increased MAX_MSG_SIZE to 16KB, and added some goofy logic to the game module that spawned hundreds of giblets, and deployed a dedicated server on Rackspace. I then connected from my LAN, and proceeded to blast the giblets with the BFG, which produces BFG tracer lasers and causes the giblets to bounce and fly all over the map.

In my tests, messages as large as 12KB (the largest I could generate) were handled completely gracefully. I didn't expect that. Crazy, right?

So where does this leave us? The IP spec claims that the transport layer _should_ do this transparent fragmenting for us. Is it finally safe to assume that, in most environments, it will?

[Jay Dolan]

Just as a follow-up, I even connected two separate clients from my Macbook to the remote server, to ensure that qport was working correctly, and indeed, everything held up for the same test of 200 giblets + BFG nonsense. Food for thought!

[Spike]

there'll always be someone with a connection that can't handle fragments.
typically with multiple levels of router that needs different mtus - 8-byte fragments are probably going to trip all sorts of DOS protection and paranoia.
there's a lot of paranoia out there, and when it works just fine for tcp you'll always have someone complain that facebook works so your game should too, despite their router being badly configured like having udp completely firewalled or whatever...

so it works for you. that's great... but that's really all your test demonstrates.

[Jay Dolan]

Sure. I saw this same experiment fail 10 years ago when I tried it behind a Linksys WRT54 on cable.

Nothing works 100% of the time, not even gravity

You're not really offering much in the way of solutions. And even the caveats you call out contradict the net code we've all been using for 20 years: 1450 > 576, yet QW and Q2 do not handle internal fragmentation.

I guess the point I was making is: Perhaps it's time to revisit acceptable limits for MAX_MSG_SIZE? And as an aside, in older engines, where the legacy behavior is to immediately drop any client requiring a message > 1400 bytes anyway, frankly what's the harm in attempting to deliver a larger payload? It'll either get there (yay!) or it won't, and the client will timeout or drop on a malformed / out of sequence message. What's the difference?

[Spike]

that's because any time I try replying in detail, this forum logs me out and swallows my posts in the process.
1450 comes from ethernet, which is very abundant.

fte+q3 support their own netchan fragmentation. dp only does for reliables, it depends upon prioritisation and packet rates for unreliables.
fte+dp use some kind of round-robin or prioritisation in order to limit packet sizes somewhat. both use nack-based networking. both also run at more than 10fps too, of course...

imho clients should be willing to accept large payloads, but servers should try to avoid needing them. its probably better to try than to kick outright. users will feel that it was malicious either way, but an error message saying that its not personal can help a little on a psychological level.
if you're unsure what a client supports, then either you're stuck updating q2's ents at only 5 or 2.5 fps, or you're depending on fragmentation. Neither are ideal, but at least fragmentation can be 'solved' by getting the user to buy decent hardware/connectivity.