Phase 1 & 2 are up but no traffic flowing

Unfortunately it did not work, but it's very much appreciated anyway :)

Your help was very useful anyway, and this forum is definitely the way to go when it comes to PA. :), I will recommend it on my blog https://bluenetsec.com/blog/ which is more dedicated to Cisco security topics, at least for now :)

Good to hear you found the solution! I was starting to get worried I overlooked something obvious ;) Thanks for ordering the book! I hope you find it useful :)

The issue is fixed now. It was caused by the missing configuration on the ISP router which was denying NAT-T traffic, this is why it was not working when I tried to enable NAT-T on the PA. After opening port 4500/udp on the ASA's ISP router and after I enabled NAT-T for this tunnel, everything start working as expected. I learned a lot during this troubleshooting journey through, and I am now looking more forward to receive your book that I ordered on Amazon:

https://www.amazon.co.uk/Mastering-Palo-Alto-Networks-industry-leading/dp/1789956374/ref=sr_1_3?dchild=1&keywords=palo+alto+networks&qid=1596997780&quartzVehicle=109-1385&replacementKeywords=palo+networks&sr=8-3

Thank you for all your help on this.

Yes Debug dataplane packet-diag clear log log

Is it possible to clear the pan_packet_diag.log file content before I start again?

Debug dataplane packet-diag clear filter all

Ok, how to clear all the filters so I can start again?

Also, you don't need to define all the filters, your first filter is from the Palo to the ASA but you set an ingress interface while this is outbound traffic, so leave out the interface and focus on the source/dest IPs

After which step shall I apply the filter on?

Don't forget to turn on the filters Debug dataplane packet-diag set filter on

I must have done something wrong, the log file was showing a bunch of details not only related to the filters I sat. I followed these steps:

debug dataplane packet-diag set filter match source <PA-endpoint> destination <ASA-endpoint> ingress-interface <PA-internal-int>

debug dataplane packet-diag set filter match ingress-interface <PA-external-int> source <ASA-public-ip>

debug dataplane packet-diag set filter match source <ASA-public-ip> destination <PA-external-private-ip>

debug dataplane packet-diag set log feature flow basic

debug dataplane packet-diag set log on

Ping from endpoint behind PA to the one behind the ASA

debug dataplane packet-diag set log off

debug dataplane packet-diag aggregate-logs

less mp-log pan_packet_diag.log

Are those steps correct?

True Ok So, flow basic Make sure to set filters first Between the client and server on the inside Between the external interface and remote public IP Between the remote public IP and the external ip Debug dataplane packet-diag set log feature flow basic Debug dataplane packet-diag set log on Run a test *Set log off Wait a minute Debug dataplane packet-diag aggregate-logs Wait a few more minutes Less mp-log pan_packet-diag.log Find the interface packets are being pushed into and if there's any communication between the endpoints

I see, but I can't see a good value in running the capture in this case since it would not show me what being passed through the tunnel interface.

you can set all kinds of filters, was just pointing out you can only pick ingress-interface where egressinterface would have been a nice filter option Last step is to enable flow basic (debug dataplane packet-diag set log feature flow basic) but this gets extremely deep dive-y...

Is it possible to do a real time capture on PA?

If the capture is going to be only on the ingress interface it would not help much since I know the traffic goes through the ingress interface. That's because when I try to ping the remote endpoints behind the ASA, I see the tunnel being triggered on the PA.

I am running out of ideas of how to fix this interesting issue.

You can set up packetcapure and simultaneously follow global counters Unfortunately the only 'interface' option is on ingress To set filters Debug dataplane packet-diag set filter match <tab for filters> To set captures Debug dataplane packet-diag set capture stage receive|transmit|drop file <filename> Then turn both on Debug dataplane packet-diag set filter on * Set capture on Or via GUI monitor > packet capture Show counter global filter delta yes packet-filter yes (do one before you start to set the delta)

I did without the condition keyword since we have only one peer configured on the ASA, I could not spot anything wrong. I am still thinking that the issues are more related to routing rather than the tunnel itself.

Is there any way on PA to capture that is being sent out of the tunnel interface? on the ASA that is possible, but unfortunately the ASA in this case has an old code that does not support that feature.

Have you tried debug crypto condition peer <peer IP> Debug crypto ipsec 255

This is what I see on the ASA:

xxx-xx(config)# sh vpn-sessiondb de l2l | i Tx|Rx

Bytes Tx : 0 Bytes Rx : 0

Bytes Tx : 0 Bytes Rx : 0

Pkts Tx : 0 Pkts Rx : 0

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

I'm still thinking traffic is flowing into the tunnel but Cisco is rejecting it. I'm not a Cisco guru, any fancy debug commands that can verify the amount of packets received vs the amount of packets decrypted? (Or any deep dive error messages that could indicate any communication issues with the remote end)

Same exact result, both Phase 1 & 2 came up successfully, but the traffic does not flow.

Just set it to none, the remote end should figure out routing on their end

Do I need to define the next hop or shall I set it as None?

Set it to source interface and set your trust interface, destination is the remote subnet, forwarding is the tunnel interface You may need to set an IP on the tunnel interface but that can be anything

Sorry, not something I've done before, how it should be configured? What would be the source zone, address, destination address, and egress interface?

Ok You could try a policy based forwarding rule to force sessions down the pipe, at least that will validate if routing is an issue or not

Ok You could try a policy based forwarding rule to force sessions down the pipe, at least that will validate if routing is an issue or not

The debug does not seem to have anything curious, and it is identical of when it worked by initiating the traffic from the ASA side.

I do still believe there is something wrong in terms of routing from the PA side, but not sure how to verify that? Why I am thinking that is because when I initiate the traffic from PA I don't see any encaps/decaps on the ASA, both show zeros, as if the traffic coming from PA does not land into the ASA tunnel.

Aggressive mode would not be a preferable option.

Hm no sorry I was assuming they were identical... Is the debug outputting anything curious? If all else fails I'd try for NAT-T in aggressive mode with FQDN IDs...

The PA and ASA outside interfaces are in the same subnet but they are different, the PA is .254 and the ASA is .2.

Do you think I should still try with a loopback interface?

Yes, that would allow you to shift the IP so it won't overlap with the cisco external IP, just in case that is causing this weird behavior

Setting up a loopback on the PA and use it for the IKE gateway interface?

On the PA side, just trying to check all the boxes Any luck on aggressive mode with FQDN identifiers? Or setting up a loopback on a different IP?

On the PA side, just trying to check all the boxes Any luck on aggressive mode with FQDN identifiers? Or setting up a loopback on a different IP?

Which side you mean, ASA or PA? the PA has only a PAT rule, the ASA instead has a PAT rule as well as the identity NAT rule for the VPN.

Interesting... Did you make sure you're not hitting a NAT rule on the outbound?

Well, here is the interesting thing, if I initiate the traffic from the ASA side everything is working. This leads me again to believe that if I initiate the traffic from the PA side, the PA does not seem to route it properly. However, when it gets that traffic from the ASA it seems to be able to route it back?

I will try now to initiate the traffic from the ASA side.

IKE debug is enabled on the PA, and I am using the command "tail follow yes mp-log ikemgr.log" to review the debug.

The tunnel interface I created for this tunnel is in trusted zone, similar to the other one for the other tunnel that is working. Is this an issue?, if so, why is working with the other tunnel then? :)

Ah, you could try setting aggressive mode for that. Are you able to send traffic from the cisco to the PA? You could `debug ike gateway 'asa' on debug` and then `tail follow yes mp-log ikemgr.log` while you send packets (like a ping) from the ASA It may also be useful while sending packets but the receiving end will usually get the most details of failed connections. Another alternative, if you can use any IP in the external private subnet, is to create a loopback interface on a different IP and bind the ikegw to that (You did set the tunnel interface to a different zone than all others, right?)

Sorry, was my bad, I'd not select FQDN for the peer ID before. Now I sat both ends with their FQDN but still no luck, I get this error on the PA:

[PERR]: { 2: }: Expecting IP address type in main mode, but FQDN.

I did run debug on the ASA, and last thing I could see was PHASE 2 COMPLETED.

The proxyIDs are set correctly, mirroring each other.

When I tried to set the peer local ID on the PA I get this error: <IKE-Gateway> -> peer-id -> id 'xxx.xxx.xxx' is invalid. the peer ID string is invalid. <IKE-Gateway> -> peer-id -> id is invalid

Are you able to run debug on the ASA? The encap means packets are being sent into the tunnel so that part works. Are the proxyID subnets identically configured on both ends? You could try setting the identifiers on both ends to a FQDN instead of an IP to prevent an issue where the IP causes some sort of collision (or bind the tunnel to the internal interface so the ID changes to the internal IP)

I am using remoted ID with the ASA private IP address only, if I don't use that it won't work and the tunnel does not come up. Using the local ID did not make any difference.

Yes seeing the encaps means that the traffic being tunneled, but I have the feeling that the PA does not route that traffic to the remote end, how can I verify that?

The proxyIDs are different, no overlapping at all. The only thing is that my PA outside interface has a private IP in the same subnet as the remote ASA outside private IP, but not sure if this would cause any issue?

As I defined the proxyIDs on PA for this tunnel, it should work in policy-based mode. I have another tunnel with another ASA (with multiple proxyIDs) and that is working just fine. The only difference between the two ASAs is that the one that is working is not behind a NAT device!

have access to both firewalls. I could not see anything weird on the ASA side, it says PHASE 2 COMPLETED, and on PA it says ====> IPSEC KEY INSTALLATION SUCCEEDED <====.

Did you use localID and remoteID in the ikegw? What are the proxyIDs: if there are multiple, are you sure all of them are up? As you see encap, you are sending things into the tunnel but the remote end is not reporting to decap, pointing to a negotiation issue. The PA is a route based vpn so will send everything you tell it into the tunnel while the cisco is policy based which means if it receives something inside the tunnel it does not like, it will not decap and discard instead. Is the remote end able to run debug? If not, are they able to initiate from their end, that may give you the opportunity to debug and see where the mismatch is. As the tunnel is coming up, I'd double-check the proxyIDs, maybe there's a subnet mismatch.