Hi Friends,
In my previous MPLS posts I have covered some basics about MPLS now moving to MPLS VPN concepts, lets see how you can wrap your head around it. For mpls vpn configuration please check this blog..
If you are a guy who comes across MPLS VPN issue once in couple of months then it can be challenging even if you are well versed with the technology( Yeah! rusting is a thing). So it is always a good idea to have a method or notes of your own on the basis of your prior experiences. Below is a simplest method that I keep handy while working with MPLS. Here I am explaining simple MPLS VPN path tracing and method to check if it is working fine or not. Yes, many times you may need to fight to prove the issue is not with MPLS VPN.
Let me share the topology first..
In below topology CE1 is trying to reach 222.222.222.222(loopback on CE2) from source 111.111.111.111(Loopback on CE1). Below is the traceroute and screenshot depicting forward and reverse path so you get the understanding of the overall network.
Forward Path:
CE1#traceroute 222.222.222.222 sou lo 1
Type escape sequence to abort.
Tracing the route to 222.222.222.222
1 10.10.10.6 8 msec 12 msec 4 msec
2 20.20.20.6 [MPLS: Labels 204/623 Exp 0] 76 msec 96 msec 76 msec
3 30.30.30.2 [MPLS: Labels 309/623 Exp 0] 92 msec 96 msec 96 msec
4 40.40.40.2 [MPLS: Labels 501/623 Exp 0] 104 msec 100 msec 100 msec
5 10.10.10.9 [MPLS: Label 623 Exp 0] 92 msec 80 msec 76 msec
6 10.10.10.10 92 msec 108 msec 132 msec
Reverse Path:
CE2#traceroute 111.111.111.111 sou lo 1 timeout 1
Type escape sequence to abort.
Tracing the route to 111.111.111.111
1 10.10.10.9 8 msec 20 msec 16 msec
2 50.50.50.1 [MPLS: Labels 510/115 Exp 0] 80 msec 80 msec 100 msec
3 40.40.40.5 [MPLS: Labels 413/115 Exp 0] 92 msec 80 msec 92 msec
4 30.30.30.5 [MPLS: Labels 207/115 Exp 0] 104 msec 92 msec 96 msec
5 10.10.10.6 [MPLS: Label 115 Exp 0] 100 msec 84 msec 80 msec
6 10.10.10.5 108 msec 112 msec 112 msec
Tracing Forward Path:
Below is the simple three step process which will tell you if the forward path is fine and also provide you some very crucial and necessary information which you need to know in order to pursue the issue further. Before you get into the below outputs see the forward path traceroute from CE1 above.
Step 1: Check the route to destination on CE router. CE1#sh ip route 222.222.222.222 Routing entry for 222.222.222.222/32 Known via "bgp 100", distance 20, metric 0 Tag 200, type external Last update from 10.10.10.6 00:09:35 ago Routing Descriptor Blocks: 10.10.10.6, from 10.10.10.6, 00:09:35 ago -----> Next hop is PE2 Route metric is 0, traffic share count is 1 AS Hops 2 Route tag 200 Step 2: Checking on PE2 if the route is learnt from PE3 or not. PE2#sh ip bgp vpnv4 vrf R3-C 222.222.222.222 BGP routing table entry for 3.3.3.3:1:222.222.222.222/32, version 33 Paths: (1 available, best #1, table R3-C) Advertised to update-groups: 1 300, imported path from 8.8.8.8:1:222.222.222.222/32 8.8.8.8 (metric 5) from 8.8.8.8 (192.168.2.1)------> Learnt from 8.8.8.8(PE3) Origin IGP, metric 0, localpref 100, valid, internal, best Extended Community: RT:8.8.8.8:1 mpls labels in/out nolabel/623-------> outgoing VPN label Similarly you need to go to PE3 and ensure the MPLS VPN label is matching. The label which is showing as "out label" here should be "in label" other side for the route prefix. PE3#sh ip bgp vpnv4 vrf R7-C 222.222.222.222 BGP routing table entry for 8.8.8.8:1:222.222.222.222/32, version 26 Paths: (1 available, best #1, table R7-C) Flag: 0x820 Advertised to update-groups: 2 300 10.10.10.10 from 10.10.10.10 (222.222.222.224) Origin IGP, metric 0, localpref 100, valid, external, best Extended Community: RT:8.8.8.8:1 mpls labels in/out 623/nolabel Step 3: Once we know from where the route is learnt, the source should also be reachable over VPN. PE2#ping mpls ipv4 8.8.8.8/32 Sending 5, 100-byte MPLS Echos to 8.8.8.8/32, timeout is 2 seconds, send interval is 0 msec: Codes: '!' - success, 'Q' - request not sent, '.' - timeout, 'L' - labeled output interface, 'B' - unlabeled output interface, 'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch, 'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry, 'P' - no rx intf label prot, 'p' - premature termination of LSP, 'R' - transit router, 'I' - unknown upstream index, 'X' - unknown return code, 'x' - return code 0 Type escape sequence to abort. !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 48/65/76 ms PE2#traceroute mpls ipv4 8.8.8.8/32 Tracing MPLS Label Switched Path to 8.8.8.8/32, timeout is 2 seconds Codes: '!' - success, 'Q' - request not sent, '.' - timeout, 'L' - labeled output interface, 'B' - unlabeled output interface, 'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch, 'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry, 'P' - no rx intf label prot, 'p' - premature termination of LSP, 'R' - transit router, 'I' - unknown upstream index, 'X' - unknown return code, 'x' - return code 0 Type escape sequence to abort. 0 20.20.20.5 MRU 1500 [Labels: 204 Exp: 0] L 1 20.20.20.6 MRU 1500 [Labels: 411 Exp: 0] 12 ms L 2 30.30.30.6 MRU 1500 [Labels: 501 Exp: 0] 28 ms L 3 40.40.40.6 MRU 1504 [Labels: implicit-null Exp: 0] 60 ms ! 4 50.50.50.2 48 ms
That is all, if you can ensure getting above information then I think the L3 VPN control plane is sorted. Now unless there is anything wrong with the device like bug or hardware issue, the end to end reachability should be fine in terms of forward path. Now lets dwell into reverse path.
Tracing Reverse path:
Similar to the method of forward path tracing we should also trace the backward path. The goal is to verify both sides reachability so that we can point out something unexpected in the network. Before you get into the below outputs see the reverse path traceroute from CE2 above.
Step 1: Check the route to destination on CE router. CE2#sh ip route 111.111.111.111 Routing entry for 111.111.111.111/32 Known via "bgp 300", distance 20, metric 0 Tag 200, type external Last update from 10.10.10.9 01:52:39 ago Routing Descriptor Blocks: 10.10.10.9, from 10.10.10.9, 01:52:39 ago -----> Shows PE3 as next hop Route metric is 0, traffic share count is 1 AS Hops 2 Route tag 200 Step 2: Checking on PE3 if the route is learnt from PE2 and PE1 or not. Notice that we have learnt routes from both the PEs but the first route is preferred PE3#sh ip bgp vpnv4 vrf R7-C 111.111.111.111 BGP routing table entry for 8.8.8.8:1:111.111.111.111/32, version 17 Paths: (2 available, best #1, table R7-C) Advertised to update-groups: 1 100, imported path from 3.3.3.3:1:111.111.111.111/32 3.3.3.3 (metric 5) from 3.3.3.3 (3.3.3.3) -----> Learnt from PE2 Origin IGP, metric 0, localpref 100, valid, internal, best Extended Community: RT:3.3.3.3:1 mpls labels in/out nolabel/115 --------------> Notice the VPN label 100, imported path from 2.2.2.2:1:111.111.111.111/32 2.2.2.2 (metric 5) from 2.2.2.2 (192.168.1.1) Origin IGP, metric 0, localpref 100, valid, internal Extended Community: RT:2.2.2.2:1 mpls labels in/out nolabel/33 Similarly you need to go to PE2 and ensure the MPLS VPN label is matching. The label which is showing as "out label" here should be "in label" other side for the route prefix. PE2#sh ip bgp vpnv4 vrf R3-C 111.111.111.111 BGP routing table entry for 3.3.3.3:1:111.111.111.111/32, version 2 Paths: (1 available, best #1, table R3-C) Advertised to update-groups: 2 100 10.10.10.5 from 10.10.10.5 (111.111.111.114) Origin IGP, metric 0, localpref 100, valid, external, best Extended Community: RT:3.3.3.3:1 mpls labels in/out 115/nolabel Step 3: Now we only need to ensure that the MPLS reachability between the PE3 and PE2 is fine, for which mpls ping and trace will help us. PE3#ping mpls ipv4 3.3.3.3/32 Sending 5, 100-byte MPLS Echos to 3.3.3.3/32, timeout is 2 seconds, send interval is 0 msec: Codes: '!' - success, 'Q' - request not sent, '.' - timeout, 'L' - labeled output interface, 'B' - unlabeled output interface, 'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch, 'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry, 'P' - no rx intf label prot, 'p' - premature termination of LSP, 'R' - transit router, 'I' - unknown upstream index, 'X' - unknown return code, 'x' - return code 0 Type escape sequence to abort. !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 52/60/68 ms PE3#traceroute mpls ipv4 3.3.3.3/32 Tracing MPLS Label Switched Path to 3.3.3.3/32, timeout is 2 seconds Codes: '!' - success, 'Q' - request not sent, '.' - timeout, 'L' - labeled output interface, 'B' - unlabeled output interface, 'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch, 'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry, 'P' - no rx intf label prot, 'p' - premature termination of LSP, 'R' - transit router, 'I' - unknown upstream index, 'X' - unknown return code, 'x' - return code 0 Type escape sequence to abort. 0 50.50.50.2 MRU 1500 [Labels: 510 Exp: 0] L 1 50.50.50.1 MRU 1500 [Labels: 413 Exp: 0] 8 ms L 2 40.40.40.5 MRU 1500 [Labels: 207 Exp: 0] 20 ms L 3 30.30.30.5 MRU 1504 [Labels: implicit-null Exp: 0] 8 ms ! 4 20.20.20.5 48 ms
With this , we have verified the end to end path with labels.
Summary :
This is the method you can use anytime working with MPLS L3 VPN. Having able to trace the end to end path with these three steps will reveal you a very good amount of clear data which will definitely help to solve the issue very quickly. The benefit of practicing this method is multifold, I am going to apply this method in troubleshooting 6VPE as well. Stay tuned for my next blog on the same. Thanks for reading..have a nice time.