For MPLS VPN to work (end-to-end) we need Route-Distinguisher, Route-Target, Transport Labels and VPN labels. But these four entities are not dependent on each other rather without any one of them the MPLS VPN is not complete.

In this article I will explain the significance of RD & RT only (I am not covering LDP in this article). I would like to show a couple of behavior of them which I came across during my preparation.

Now let’s decode RD & RT.

What is Route-Distinguisher aka RD

This is a 64bit value. When it is appended to IPv4 address/route, it creates a globally unique VPNv4 address. This helps in having overlapping IPv4 address from different customers yet unique when converted to VPNv4 address. Because of this, when the PE routers exchange the NLRI to each other, they get unique routes (VPNv4 address).

For example:-

RD:-  10:100 & IPv4 route:-  100.100.100.0/24
Then VPNv4 route is 10:100:100.100.100.0/24

 

 

 

What is Route-Target aka RT

This is again a 64bit value. This value is carried with NLRI as an attribute value or additional field as part of BGP extended community. The RT value determines which routes will be exported & imported to the VPNv4 address table. It works like another identifier when exported from one VRF and then imported by any other interested VRF.

If one exports its routes with RT value (eg 10:11) & another PE is interested to learn those routes, it has to import that RT value (10:11) into its VRF table.

Now we have two different values associated with VPNv4. But what is the significance of each of them?

Let’s discuss that in depth.

(*** This topic is only dedicated to RD & RT. I will only discuss about them. In my topology I had configured IGP and LDP in background to complete the topology for exchanging VPNv4 routes.)

mpls-rd-rt

Topology:

PE1 (1.1.1.1), PE2 (2.2.2.2), PE3 (3.3.3.3), PE4 (4.4.4.4) & P (5.5.5.5) are ISP routers in AS 10 and running MPLS.

PE3 is acting as VPNv4 RR (Route Reflector server) and PE2 & PE4 are RR clients for PE3.

PE1 is a normal iBGP neighbor to PE2. (I have created this on purpose to show the issue in the design and the different solutions for that).

All PEs are running ebgp towards their respective CEs on different VRFs.

PE2 is learning (from eBGP)

Route 100.100.100.100/32 on eth1/0 part of VRF A. (RD 10:100, RT export 10:101, RT import 10:111)
Route 200.200.200.200/24 on eth1/1 part of VRF B. (RD 10:200, RT export 10:201, RT import 10:21)

PE3 is is learning (from eBGP)

Route 101.101.101.101/32 on eth1/0 part of VRF A. (RD 10:10, RT export 10:11, RT import 10:101)
Route 201.201.201.201/32 on eth1/1 part of VRF B. (RD 10:20, RT export 10:21, RT import 10:201)

PE1 is learning (from eBGP)

Route 105.105.105.105/24 on eth1/0 part of VRF A. (RD 10:1000, RT export 10:1001)

PE4 is learning (from eBGP)

Route 205.205.205.205/24 on eth1/0 part of VRF B. (RD 10:2000, RT export 10:2001)

Now let’s look at the vpnv4 RIB on PE2

 PE2#show bgp vpnv4 unicast all
BGP table version is 21, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
 r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
 x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

      Network        Next Hop     Metric LocPrf Weight Path
Route Distinguisher: 10:10
 *>i 101.101.101.101/32 3.3.3.3    0       100   0 1011 i
Route Distinguisher: 10:20
 *>i 201.201.201.201/32 3.3.3.3    0       100   0 1021 i
Route Distinguisher: 10:100 (default for vrf A)
 *> 100.100.100.100/32 150.1.32.1   0            0 1001 i
 *>i 101.101.101.101/32 3.3.3.3     0      100   0 1011 i
Route Distinguisher: 10:200 (default for vrf B)
 *> 200.200.200.0 150.2.32.1        0            0 1002 i
 *>i 201.201.201.201/32 3.3.3.3     0      100   0 1021 i
PE2#

Did you notice the routes highlighted in color green?

These are the routes PE2 received from PE3 as we imported the respective RTs.

PE3 advertised the routes to PE2 as 10:10:101.101.101.101/32 and 10:20:201:201:201:201/32.

Those routes are installed to the respective VRFs (who imported them) inside their vpnv4 vrf table. When VRFs install any VPNv4 route, the remote RD (part of VPNv4 route) gets dropped. Now those routes are installed in the VRF RIB with the local RD value.

Have a look at 101.101.101.101/32 in VRF A (RD 10:100) and 201.201.201.201/32 in VRF B (10:200).

Route Distinguisher: 10:100 (default for vrf A)
 *>  100.100.100.100/32
                       150.1.32.1               0             0 1001 i
 *>i 101.101.101.101/32
                       3.3.3.3                  0    100      0 1011 i
Route Distinguisher: 10:200 (default for vrf B)
 *>  200.200.200.0    150.2.32.1               0             0 1002 i
 *>i 201.201.201.201/32
                       3.3.3.3                  0    100      0 1021 i

Let’s prove that from the debug logs

*Dec 31 08:45:56.372: %BGP-5-ADJCHANGE: neighbor 3.3.3.3 Up 
*Dec 31 08:45:56.396: BGP: nbr_topo global 3.3.3.3 VPNv4 Unicast:base (0xB293DC60:1) rcvd Refresh Start-of-RIB
*Dec 31 08:45:56.396: BGP: nbr_topo global 3.3.3.3 VPNv4 Unicast:base (0xB293DC60:1) refresh_epoch is 2
*Dec 31 08:45:56.402: BGP(4): Incoming path from 3.3.3.3
*Dec 31 08:45:56.402: BGP(4): 3.3.3.3 rcvd UPDATE w/ attr: nexthop 3.3.3.3, origin i, localpref 100, metric 0, merged path 1011, AS_PATH , extended community RT:10:11
*Dec 31 08:45:56.402: BGP(4): 3.3.3.3 rcvd 10:10:101.101.101.101/32, label 1006
*Dec 31 08:45:56.403: BGP(4): Incoming path from 3.3.3.3
*Dec 31 08:45:56.403: BGP(4): 3.3.3.3 rcvd UPDATE w/ attr: nexthop 3.3.3.3, origin i, localpref 100, metric 0, merged path 1021, AS_PATH , extended community RT:10:21
*Dec 31 08:45:56.403: BGP(4): 3.3.3.3 rcvd 10:20:201.201.201.201/32, label 1007

<Truncated>

*Dec 31 08:45:56.465: BGP(4): Revise route installing 1 of 1 routes for 101.101.101.101/32 -> 3.3.3.3(A) to A IP table
*Dec 31 08:45:56.465: BGP(4): Revise route installing 1 of 1 routes for 201.201.201.201/32 -> 3.3.3.3(B) to B IP table
PE2#

Still confused? Check the vpnv4 address table for route 201.201.201.201/32 on PE2

vpnv4-rd-local-rd

We see from the above logs, the VPNv4 routes are installed to the respective VRFs with help of RT. If RT is not there then the PE router will not be able to decide under which VRF RIB the route needs to be installed.

Now think about PE4.

PE4 is learning (from eBGP)

Route 205.205.205.205/24 on eth1/0 part of VRF B. (RD 10:2000, RT export 10:2001)

How the VPNv4 RIB will look like on PE4?

Remember PE4 is RR client for PE3 and PE3 is exporting VPNv4 routes under RT 10:11 and 10:21 to its neighbors.

Let’s have a look at PE4 vpnv4 address table.

PE4#show bgp vpnv4 unicast all summary 
BGP router identifier 4.4.4.4, local AS number 10
BGP table version is 2, main routing table version 2
1 network entries using 152 bytes of memory
1 path entries using 80 bytes of memory
2/1 BGP path/bestpath attribute entries using 304 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
1 BGP extended community entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 584 total bytes of memory
BGP activity 1/0 prefixes, 1/0 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
3.3.3.3 4 10 426 410 2 0 0 06:08:54 0
155.2.32.1 4 1010 249 248 2 0 0 03:43:24 1
PE4#
PE4#show bgp vpnv4 unicast all 
BGP table version is 2, local router ID is 4.4.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
 r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
 x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

 Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10:2000 (default for vrf B)
 *> 205.205.205.0 155.2.32.1 0 0 1010 i
PE4#

It is not receiving anything from PE3.  It has only one route (learned from CE-B3) inside its VPNv4 address table.

Is PE3 not advertising anything to PE4? What went wrong here?

Nothing is wrong here. PEs (except RR) won’t accept/install routes if they don’t need it. In other words if they don’t accept the routes if the corresponding RT value is not imported by any of its VRFs.

Time to check that from the debug output.

*Dec 31 10:25:15.661: %BGP-5-ADJCHANGE: neighbor 3.3.3.3 Up 
*Dec 31 10:25:15.662: BGP: nbr_topo global 3.3.3.3 VPNv4 Unicast:base (0xB4335CB0:1) rcvd Refresh Start-of-RIB
*Dec 31 10:25:15.662: BGP: nbr_topo global 3.3.3.3 VPNv4 Unicast:base (0xB4335CB0:1) refresh_epoch is 2
*Dec 31 10:25:15.680: BGP(4): (base) 3.3.3.3 send UPDATE (format) 10:2000:205.205.205.0/24, next 4.4.4.4, label 907, metric 0, path 1010, extended community RT:10:2001
*Dec 31 10:25:15.687: BGP(4): 3.3.3.3 rcvd UPDATE w/ attr: nexthop 3.3.3.3, origin i, localpref 100, metric 0, merged path 1011, AS_PATH , extended community RT:10:11
*Dec 31 10:25:15.687: BGP(4): 3.3.3.3 rcvd 10:10:101.101.101.101/32, label 1006 -- DENIED due to: extended community not supported;
*Dec 31 10:25:15.687: BGP(4): 3.3.3.3 rcvd UPDATE w/ attr: nexthop 2.2.2.2, origin i, localpref 100, metric 0, originator 2.2.2.2, clusterlist 3.3.3.3, merged path 1001, AS_PATH , extended community RT:10:101
*Dec 31 10:25:15.687: BGP(4): 3.3.3.3 rcvd 10:100:100.100.100.100/32, label 1206 -- DENIED due to: extended community not supported;
*Dec 31 10:25:15.687: BGP(4): 3.3.3.3 rcvd UPDATE w/ attr: nexthop 3.3.3.3, origin i, localpref 100, metric 0, merged path 1021, AS_PATH , extended community RT:10:21
*Dec 31 10:25:15.687: BGP(4): 3.3.3.3 rcvd 10:20:201.201.201.201/32, label 1007 -- DENIED due to: extended community not supported;
*Dec 31 10:25:15.690: BGP(4): 3.3.3.3 rcvd UPDATE w/ attr: nexthop 2.2.2.2, origin i, localpref 100, metric 0, originator 2.2.2.2, clusterlist 3.3.3.3, merged path 1002, AS_PATH , extended community RT:10:201
*Dec 31 10:25:15.690: BGP(4): 3.3.3.3 rcvd 10:200:200.200.200.0/24, label 1208 -- DENIED due to: extended community not supported;
*Dec 31 10:25:15.690: BGP: nbr_topo global 3.3.3.3 VPNv4 Unicast:base (0xB4335CB0:1) rcvd Refresh End-of-RIB

Have a look at

*Dec 31 10:25:15.687: BGP(4): 3.3.3.3 rcvd 10:10:101.101.101.101/32, label 1006 — DENIED due to:  extended community not supported;

It says that PE3 sent the route update towards PE4 but PE4 denied it. Here Cisco has to modify the error log to reflect the error correctly. The error log “DENIED due to:  extended community not supported;” does not mean that extended community is not enabled, rather interpret that as, the RT value which is part of extended community is not imported/listed in this router hence it won’t accept it.

But remember PE3 which is Route Reflector will have all the VPNv4 routes advertised by its neighbors even if it does not have RT import statement for all of those routes.

Now think about PE1.

PE1 is learning (from eBGP)

Route 105.105.105.105/24 on eth1/0 part of VRF A. (RD 10:1000, RT export 10:1001)

As you might have already guessed that the VPNv4 address table on PE1 will also be having only one route (105.105.105.0/24 under VRF A) like that of PE4.

If you said that you are absolutely correct.

Now add the import statement under VRF A and try to import RT 10:2001 (it is exported from PE4).

PE1#show bgp vpnv4 unicast all     

<Truncated>

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10:1000 (default for vrf A)
 *>  105.105.105.0/24 155.2.31.1               0             0 1005 i
PE1#

Even after configuring route-target import 10:2001 under vrf B on PE4 & clearing the vpnv4 session, it is still not learning the route.

Is PE2 not advertising anything to PE1? What went wrong here?

Let’s debug on PE1 and check what updates it is receiving from PE2.

*Dec 31 15:21:19.426: BGP: nbr_topo global 2.2.2.2 VPNv4 Unicast:base (0xB28CEAD0:1) rcvd Refresh Start-of-RIB
*Dec 31 15:21:19.426: BGP: nbr_topo global 2.2.2.2 VPNv4 Unicast:base (0xB28CEAD0:1) refresh_epoch is 2
*Dec 31 15:21:19.432: BGP(4): 2.2.2.2 rcvd UPDATE w/ attr: nexthop 2.2.2.2, origin i, localpref 100, metric 0, merged path 1001, AS_PATH , extended community RT:10:101
*Dec 31 15:21:19.432: BGP(4): 2.2.2.2 rcvd 10:100:100.100.100.100/32, label 1206 -- DENIED due to: extended community not supported;
*Dec 31 15:21:19.433: BGP(4): 2.2.2.2 rcvd UPDATE w/ attr: nexthop 2.2.2.2, origin i, localpref 100, metric 0, merged path 1002, AS_PATH , extended community RT:10:201
*Dec 31 15:21:19.433: BGP(4): 2.2.2.2 rcvd 10:200:200.200.200.0/24, label 1207 -- DENIED due to: extended community not supported;
*Dec 31 15:21:19.433: BGP: nbr_topo global 2.2.2.2 VPNv4 Unicast:base (0xB28CEAD0:1) rcvd Refresh End-of-RIB
PE1#

PE1 is only sending its local originated routes under RD 10:100 & 10:200. It is because of the BGP split-horizon rule which says routes learned from one iBGP neighbor will not send to other iBGP neighbor. This is the reason it only sends its local routes.

In order to get the routes of RT 10:2001 on PE1, PE1 has to build neighborship which has those routes.

Remember few minutes ago I told about Route-Reflector behavior?

Yes. You got it. In practical scenarios, we need to get the desired VPNV4 routes from the Route-reflector as it stores all the routes from its clients in its global VPNv4 RIB.

To solve this issue we have two solutions.

Solution 1:

We can make PE1 route-reflector client for PE3. Then it will learn the route from PE3. This solution works well for a medium scale design where we will have a single Route-Reflector.

Solution 2:

We can make PE2 as the second Route-Reflector server in the ISP AS. And PE3 and PE2 will be client to each other. Then all PEs can be made route-reflector client to either one or both PE2 & PE3. Here we have resiliency over Route-Reflector. If one goes down then other will be still acting as RR in the ISP AS.

I implemented solution no.2 and after that PE1 successfully received the desired route.

PE1#show bgp vpnv4 unicast all     
BGP table version is 8, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10:1000 (default for vrf A)
 *>  105.105.105.0/24 155.2.31.1               0             0 1005 i
 *>i 205.205.205.0    4.4.4.4                  0    100      0 1010 i
Route Distinguisher: 10:2000
 *>i 205.205.205.0    4.4.4.4                  0    100      0 1010 i

Route-reflector config of VPNV4 address-family on PE2 & PE3:

PE2

router bgp 10
address-family vpnv4
 neighbor 1.1.1.1 activate
 neighbor 1.1.1.1 send-community both
 neighbor 1.1.1.1 route-reflector-client
 neighbor 3.3.3.3 activate
 neighbor 3.3.3.3 send-community both
 neighbor 3.3.3.3 route-reflector-client
 exit-address-family

PE3

router bgp 10
address-family vpnv4
 neighbor 2.2.2.2 activate
 neighbor 2.2.2.2 send-community both
 neighbor 2.2.2.2 route-reflector-client
 neighbor 4.4.4.4 activate
 neighbor 4.4.4.4 send-community both
 neighbor 4.4.4.4 route-reflector-client
 exit-address-family

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s