To see this working, head to your live site.

Edited: Mar 18, 2021

UDP out-of-order

My first post here and first of all, l want to say hello to everyone and of cause to Tom!

PAN-OS 8.1.10-h1

We have a weird issue where F5 keep shifting UDP packets for a given flow and sends them toward PA.

Below the clean and expected flow sent from the client:

It looks like, PA is doing it's best to receive, apply NAT and send packets in the correct order but still fails. Not all packets are sent in exactly the same order (based on the client-side capture above):

Checking the KB below, it looks like they all should be sent in the correct order:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClWeCAK

Does this mean PA is doing best efforts or that feature is broken?

Thanks,

myky

6 answers4 replies

Comments (10)

myky

Mar 22, 2021

I see, thanks Tom!

Under normal circumstances, the network device should not shuffle packets. Just trying to understand why they arrive out-of-order in the first place.

myky

Mar 22, 2021

I haven't seen this before, hence some doubts.

My assumption that for the given flow sender will be incrementing initially chosen IP ID by 1 for every packet (fragmented will have the same IP ID).

myky

Mar 22, 2021

Hello guys,

Just wonder if l am applying the correct logic for UDP in this case.

Packets are not fragmented but sent in a specific order with equally increment IP IDs.

Now, just wondering if this is even relevant. 🤔

Thanks,

myky

Reaper

Mar 22, 2021

Replying to

as far as i can interpret the kb article, out-of-order packets should be rearanged into proper order without a requirement or dependency on fragmentation. i could see fragmentation causing a hurdle as rearranging packets that require reassembly first may flood the buffer

Reaper

Mar 18, 2021

More things to test: check packet buffer usage Show running resource-monitor If the buffer is filling up, see if packet buffer protection is enabled on your zones. See if disabling that helps (disable zone protection while you're at it)

myky

Mar 18, 2021

Replying to

Good tips as always! Will do

Reaper

Mar 18, 2021

i don't know, i'm hoping the global counters point that out

don't forget to check the software pools, maybe you're depleting them because the packets are so scrambled

debug dataplane pool statistics

myky

Mar 18, 2021

Replying to

Thanks for the good next steps.

I can image what the results would be if there was no PA in between. That connection would miserably fail.

I will get back here with my findings.

Thanks,

myky

Reaper

Mar 18, 2021

@myky so glad to see you here! welcome! :D

Have you tried to trace the global counters for this session? do you happen to know the relative time between the 26629 cluster and the 26648 cluster? it could be that the palo is handling frames of packets and reordering them but the distance between some is too large to properly reorder the entire chain

the global counters may shed some light on that if the packets on the 'fringe' of the firewall's capability/buffer might get discarded every so often

myky

Mar 18, 2021

Replying to

I will have to check deeper the time stamps tomorrow, but it was 3 simultaneous captured during the 60 seconds UDP iPerf test. Global counters also might be the next step.

It's essential for that application for all packets to arrive in exact same order (the order they were sent out from the client).

Not too sure why F5 is messing so much with the packet order (that will be another investigation).

Do you happen to know the max time gap that PA will wait to reorder out-of-order packets before sending them out for a given flow? Is there such thing at all or am l making something up?

Thanks,

myky

Forum: Forum