GNS3 LAB : Cisco MPLS troubleshooting trouble ticket #1

In Continuation to my previous MPLS articles, in this article I will discuss a rather real life example of issue and troubleshooting. You can check my previous posts on MPLS here.

1- MPLS Basic configuration and MPLS QOS

2- MPLS QOS

3- MPLS Basics and Beyond Concepts…

Problem :

In the below network MPLS VPN between between PE1 and PE3 is not coming up.

Approaching solution by breaking down to components

We know that for MPLS VPN to work we must have end to end MPLS up and for MPLS to work we first need IGP to facilitate the FIB information. So first step to troubleshoot MPLS is to check if the IGP is fine. Hence, lets check and troubleshoot MPLS reachability between router PE1 and PE3, for some reason MPLS ping and trace are not working. 

PE1#ping mpls ipv4 8.8.8.8/32
 Sending 5, 100-byte MPLS Echos to 8.8.8.8/32,
      timeout is 2 seconds, send interval is 0 msec:
 Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
   'L' - labeled output interface, 'B' - unlabeled output interface,
   'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
   'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
   'P' - no rx intf label prot, 'p' - premature termination of LSP,
   'R' - transit router, 'I' - unknown upstream index,
   'X' - unknown return code, 'x' - return code 0
 Type escape sequence to abort.
 BBBBB
 Success rate is 0 percent (0/5)

PE1#traceroute mpls ipv4 8.8.8.8/32
 Tracing MPLS Label Switched Path to 8.8.8.8/32, timeout is 2 seconds
 Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
   'L' - labeled output interface, 'B' - unlabeled output interface,
   'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
   'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
   'P' - no rx intf label prot, 'p' - premature termination of LSP,
   'R' - transit router, 'I' - unknown upstream index,
   'X' - unknown return code, 'x' - return code 0
 Type escape sequence to abort.
   0 20.20.20.1 MRU 1500 [Labels: 201 Exp: 0]
 L 1 20.20.20.2 MRU 1500 [Labels: 302 Exp: 0] 20 ms
 B 2 30.30.30.2 MRU 1504 [No Label] 12 ms
 B 3 30.30.30.2 MRU 1504 [No Label] 20 ms
 B 4 30.30.30.2 MRU 1504 [No Label] 20 ms
 B 5 30.30.30.2 MRU 1504 [No Label] 16 ms
 B 6 30.30.30.2 MRU 1504 [No Label] 20 ms
 B 7 30.30.30.2 MRU 1504 [No Label] 24 ms
 B 8 30.30.30.2 MRU 1504 [No Label] 12 ms
 B 9 30.30.30.2 MRU 1504 [No Label] 12 ms
 B 10 30.30.30.2 MRU 1504 [No Label] 24 ms
<Output SNIP>

Looking at mpls code(B) it seems that the packet is hitting a router which is sending packet to an unlabeled interface, I guess the unlabeled interface is the interface where LDP is not enabled. But I have used mpls ldp autoconfig on most of the routers(as far as I remember) so if ospf is working then MPLS should also work. Let us check if we are able to do ipv4 ping to the IP 8.8.8.8 or not and to my surprise ipv4 ping is working perfectly .

PE1#ping 8.8.8.8
 Type escape sequence to abort.
 Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
 !!!!!
 Success rate is 100 percent (5/5), round-trip min/avg/max = 52/65/72 ms
 PE1#

We know that the IGP is working between PE1 and PE3, so what is next ? What next..we will have to take a step back and check the traceroute lets see where exactly it is breaking. 

 Type escape sequence to abort.
   0 20.20.20.1 MRU 1500 [Labels: 201 Exp: 0]
 L 1 20.20.20.2 MRU 1500 [Labels: 302 Exp: 0] 20 ms
 B 2 30.30.30.2 MRU 1504 [No Label] 12 ms

Looking at the traceroute it seems that the node 30.30.30.2 doesn’t assign label. We should look at 30.30.30.2 for hints.Router P2 holds the IP address 30.30.30.2 and looking at the ldp neighbor we do not see all of them in neighbor table. As per the config there should be three neighbors but we see only two. 

P2#sh mpls ldp neighbor
     Peer LDP Ident: 4.4.4.4:0; Local LDP Ident 5.5.5.5:0
     TCP connection: 4.4.4.4.646 - 5.5.5.5.37957
     State: Oper; Msgs sent/rcvd: 225/223; Downstream
     Up time: 03:03:06
     LDP discovery sources:
       GigabitEthernet1/0, Src IP addr: 30.30.30.1
         Addresses bound to peer LDP Ident:
           20.20.20.2      20.20.20.6      30.30.30.1      30.30.30.5
           4.4.4.4
     Peer LDP Ident: 6.6.6.6:0; Local LDP Ident 5.5.5.5:0
     TCP connection: 6.6.6.6.18062 - 5.5.5.5.646
     State: Oper; Msgs sent/rcvd: 227/225; Downstream
     Up time: 03:03:04
     LDP discovery sources:
       GigabitEthernet3/0, Src IP addr: 40.40.40.10
         Addresses bound to peer LDP Ident:
           30.30.30.6      40.40.40.10     6.6.6.6         40.40.40.5

P2 should also have 7.7.7.7 as neighbor but we dont see it. However we see ospf adjacency up with 7.7.7.7.

P2#sh ip osp neighbor
 Neighbor ID     Pri   State           Dead Time   Address         Interface
 6.6.6.6           1   FULL/DR         00:00:31    40.40.40.10     GigabitEthernet3/0
 7.7.7.7           1   FULL/DR         00:00:32    40.40.40.2      GigabitEthernet2/0
 4.4.4.4           1   FULL/BDR        00:00:35    30.30.30.1      GigabitEthernet1/0

Till here we are able to narrow down that the problem is between P2 and P4 and also with LDP, even though the IGP is fine between them. All across the network MPLS neighbourship is formed over lo0 so all the loopbacks should be reachable with each other. As IGP is up hence we expect this to be reachable lets check if P4’s loopback0 is reachable from P2.

P2#ping 7.7.7.7 sou lo 0
 Type escape sequence to abort.
 Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
 Packet sent with a source address of 5.5.5.5
 …..
 Success rate is 0 percent (0/5)
 P2#

Well, I think this is where the problem lies.We are not able to reach to P4’s loopback0 .Lets see if we are advertising it on P4 at first place. And looking at the configs we now know that we missed to advertise loopback 0 on P4 which is creating problems. 

router ospf 1
  mpls ldp autoconfig
  router-id 7.7.7.7
  log-adjacency-changes
  network 40.40.40.2 0.0.0.0 area 0
  network 40.40.40.6 0.0.0.0 area 0
  network 50.50.50.1 0.0.0.0 area 0

Lets put the network back and see if this works. 

router ospf 1
  mpls ldp autoconfig
  router-id 7.7.7.7
  log-adjacency-changes
  network 40.40.40.2 0.0.0.0 area 0
  network 40.40.40.6 0.0.0.0 area 0
  network 50.50.50.1 0.0.0.0 area 0
  network 7.7.7.7 0.0.0.0 are 0

Our LDP has come up between P2 and P4 and also the mpls ping seems to work fine between PE1 and PE3. Also MPLS VPN comes up fine. 

PE1#ping mpls ipv4 8.8.8.8/32
 Sending 5, 100-byte MPLS Echos to 8.8.8.8/32,
      timeout is 2 seconds, send interval is 0 msec:
 Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
   'L' - labeled output interface, 'B' - unlabeled output interface,
   'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
   'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
   'P' - no rx intf label prot, 'p' - premature termination of LSP,
   'R' - transit router, 'I' - unknown upstream index,
   'X' - unknown return code, 'x' - return code 0
 Type escape sequence to abort.
 !!!!!
 Success rate is 100 percent (5/5), round-trip min/avg/max = 32/45/68 ms
 PE1#

Thanks for reading, please follow the blog for more articles like this.

Leave a Reply