EVPN multi-homing support for L3 services

1.1. Problems with unicast load-balancing from core to CE

The layer-2 hashing performed by CE over its LAG port means that its possible for only one service PE to populate its ARP/ND cache. Take for example PE1 and PE2 from Figure 1. If CE1 ARP/ND response happens to always hash over I1 towards PE1, then PE2 ARP/ND table remains empty. Since unicast traffic from remote PEs can be received by either service PE, traffic that reaches the service PE2 does not find an ARP entry matching the host IP address and traffic is dropped until its ARP/ND table is updated.¶

If the CEs hash implementation always calculates the ARP/ND response towards PE1, the resolution on PE2 never succeeds and traffic load balanced to PE2 is permanently dropped.¶

The route sync solution is described in Section 2.5¶

1.2. Problems with multicast from core to CE

Like the unicast behavior above, multicast IGMP/MLD join messages from CE to LAG link may always hash to a single PE.¶

When PIM runs on both redundant layer-3 PEs, both serving multicast for the same access segment, PIM hello messages [RFC7761] issued by I1 (Figure 1) are not received by I2, and, vice versa; PIM hello messages issued by I2 are not received by I1. This is due to the CE not being able to switch traffic between the two members of the same LAG. Both PEs therefore become PIM Designated Router (DR). The PIM DR is responsible for tracking local multicast listeners and forwarding traffic to those listeners. The PIM DR is also responsible for sending local Join/Prune messages towards the RP or source. However, due to the CE hashing, a particular IGMP join for a given multicast group is received by only one of the PEs. Only that PE programs the multicast route for the group and issues a PIM join message.¶

The multicast route sync solution is described in Section 2.6¶

1.3. Problems with IGP adjacencies over the LAG port

A layer-3 CE device/router that connects to the redundant PEs may establish an IGP adjacency on the bundle port. In this case, the adjacency is formed to one of the PEs and IGP customer route(s) is only present on that PE.¶

This prevents the load-balancing benefits of redundant PEs from supporting this use case, as only one PE is aware and advertising the customer routes to the core.¶

                  <---------+
                            | IGP Adj
    +-------+               |
    |       | 1.1.1.1/24    |
    | PE1   +-----------+   |
    |       |           |   |
    |       |           |   +
    +-------+           |
                        |
        +               |  +------+
  RT5   |             L |  | CE1  +------>H1
  Sync  |             A +->+      |
        v             G |  |      |
                        |  |      +------>R1
    +-------+           |  +------+
    |       |           |    1.1.1.2/2
    | PE2   +-----------+
    |       | 1.1.1.1/24
    |       |
    +-------+

Figure 2 provides an example of this use case, where CE1 forms an IGP adjacency with PE1 (example: ISIS or OSPF), and advertises its H1 and R1 routes into the IP-VRF of PE1. PE1 may then redistribute this IGP route into the core as an L3 service. Any remote PEs are only aware of the service from PE1, and cannot load balance through PE2 as well.¶

Further study is required to support the case of BGP PE to CE protocols.¶

A solution to this is described in Section 2.7¶

1.4. Problems with supporting multiple subnets on same ES in all active mode

In the case where the L3 service is L3VPN such as [RFC4364], it is likely the CE device could be a layer-2 switch supporting multiple subnets through the use of VLANs. In addition, each VLAN may be associated with a different customer VRF.¶

When ARP/ND routes are synchronized between the PEs for ARP proxy support using RT-2, a similar problem is encountered as described by Section 1.1 of [I-D.ietf-bess-evpn-ac-aware-bundling]. The PE receiving RT-2 is unable to determine which sub-interface the ARP/ND entry is associated with.¶

When IGMP/MLD routes are synchronized between the PEs using RT-7 and RT-8, a similar problem is encountered as described by Section 1.2 of [I-D.ietf-bess-evpn-ac-aware-bundling]. The PE receiving RT-7 and RT-8 is unable to determine which sub-interface the IGMP join is associated with.¶

This document proposes to use the solution defined by Section 4 of [I-D.ietf-bess-evpn-ac-aware-bundling] to solve both these cases. All route sync messages (RT-2, RT-5, RT-7, RT-8) carry an Attachment Circuit Identifier Extended Community to signal which sub-interface the routes were learnt on.¶

This document focuses on configuration models over access-facing interfaces with L3 sub-interfaces. Models with both L2 and L3 sub interfaces on a interface are left for future study.¶

1.5. Acronyms

BD:

Broadcast Domain¶

BE:

Bundle Ethernet¶

DF:

Designated Forwarder¶

DR:

Multicast Designated Router¶

EC:

BGP Extended Community¶

ES:

Ethernet Segment. When a customer site (device or network) is connected to one or more PEs via a set of Ethernet links, then that set of links is referred to as an 'Ethernet Segment'.¶

ESI:

Ethernet Segment Identifier. A unique non-zero identifier that identifies an Ethernet Segment is called an 'Ethernet Segment Identifier'.¶

ESI-LAG:

This refers to multi-homing scenario where peering PEs, connected to same CE, are two, three or more.¶

ETAG:

Ethernet Tag. An Ethernet tag identifies a particular broadcast domain, e.g., a VLAN. An EVPN instance consists of one or more broadcast domains.¶

EVI:

An EVPN instance spanning the Provider Edge (PE) devices participating in that EVPN. It is used to assist a L3 VRF for route synchronization.¶

GRT:

Global Routing Table¶

ICL:

Inter Chassis Link¶

IGMP:

Internet Group Management Protocol¶

IGP:

Interior Gateway Protocol¶

IP-VRF:

A VPN Routing and Forwarding table for IP routes on an PE. The IP routes could be populated by EVPN and IP-VPN address families. An IP-VRF is also an instantiation of a layer 3 VPN in an PE.¶

L3AA

All-Active Redundancy Mode for Layer 3 services. When all PEs attached to an Ethernet segment are allowed to forward known unicast traffic to/from that Ethernet segment for a given VLAN, then the Ethernet segment is defined to be operating in All-Active redundancy mode.¶

MAC-VRF:

A Virtual Routing and Forwarding table for Media Access Control (MAC) addresses on a PE. A MAC-VRF is also an instantiation of an EVI in a PE¶

MC-LAG:

Multi-Chassis Link Aggregation Group (MC-LAG).¶

MLD:

Multicast Listener Discovery.¶

PE:

Provider Edge.¶

PIM:

Protocol Independent Multicast.¶

RD:

Route Distinguisher used in BGP.¶

RP:

Multicast Rendezvous Point.¶

RT:

Route-Targets used in BGP¶

RT-2:

EVPN route type 2, i.e., MAC/IP advertisement route, as defined in [RFC7432].¶

RT-5:

EVPN route type 5, i.e., IP Prefix route, as defined in Section 3 of [RFC9136].¶

RT-7:

EVPN route type 7, i.e., Multicast Join Synch Route, as defined in Section 9.2 of [RFC9251].¶

RT-8:

EVPN route type 8, i.e., Multicast Leave Synch Route, as defined in Section 9.3 of [RFC9251].¶

1.6. Requirements

1. The multi-homing solution MUST support Layer-3 access interface¶

2. The multi-homing solution MUST support Layer-3 access sub-interface¶

3. The solution MUST support unicast and multicast VPN services¶

4. The solution SHOULD support IGP synchronization¶

5. The solution SHOULD support unicast and multicast global routing services¶

6. The solution MUST support all-active load-balancing mode¶

7. The solution MAY support single-active load-balancing mode¶

8. The solution MUST support port-active load-balancing mode¶

2.1. Usage of L3VRF route target

The synchronization of information between peering PEs is done via various EVPN route types. For instance, adjacencies in ARP/ND tables are synchronized by leveraging EVPN route type-2. When dealing with Layer-3 interface, basic principles described in [RFC9136] are leverage. By default, any routes used for synchronization are advertised with IP-VRF route targets.¶

Alternatively, EVPN routes may be advertised with ES-import route targets along with EVI-RT EC equal to associated IP-VRF route target. This allows BGP to distribute the route(s) to only the PEs attached to the associated ESI, and also allows routes to be applied to the respective IP-VRF(s) at receiving end.¶

In the example Figure 3, route synchronization from CUST1 has IP-VRF1 RT(s) and CUST2 has IP-VRF2 RT(s). As an optimization, route synchronization uses ES-import RT(s). On top of that, CUST1 has EVI-RT BGP Extended Community (EC) with IP-VRF1 RT(s), and CUST2 EVI-RT BGP Extended Community (EC) has IP-VRF2 RT(s).¶

2.2. Usage of EVPN instance

[RFC7432] eases the auto-generation of BGP constructs such as route-distinguisher and route targets per MAC-VRF, based on a unique value for the Broadcast Domain that, in this document, we referred to as EVI. Similarly as in [RFC9136], the usage of EVI is not required when dealing with L3VPN multi-homing scenarios. The RD may be auto-generated locally with a unique Id and associated RT(s) may be taken from the IP-VRF¶

The synchronization over GRT is different. In that specific situation, an EVPN instance may be assigned to support non-VPN layer-3 services. The assignment is only serving the purpose of providing route targets as requested by [RFC7432]; where RT(s) are mandatory per EVPN route.¶

EVPN enhances the multi-homing layer 3 service with the following synchronization routes:¶

• ARP / ND¶

• IGMP / MLD¶

• IP (for customer subnets learned from IGP adjacency)¶

2.3. Mapping for L3 Interface to ESI

The ESI represents the L3 LAG interface between PE and CEs. This ESI is signaled using RT-4 with the ES-Import Route Target as described in Section 8.1.1 of [RFC7432] so that the service PE peers can discover each other's common ES.¶

In the example Figure 3, route-syncs from interface BE1 have IP-VRF RT(s) or ES-Import RT and EVI-RT EC with ESI 1 as an optimization.¶

2.4. Mapping for L3 Sub-Interface to Attachment Circuit ID

The Attachment Circuit ID represents the sub-interface subnet on the L3 LAG interface between PE and CEs. The AC-ID is signaled using RT-2, RT-5, RT-7 and RT-8 by attaching Attachment Circuit ID Extended community as described in Section 6.1 of [I-D.ietf-bess-evpn-ac-aware-bundling].¶

In the example Figure 3, route-syncs from sub-interface BE1.1 (VLAN1) have Attachment-Circuit-ID EC with ID 1¶

2.5. Route sync for ARP/ND

This document proposes solving the issue described in Section 1.1 using RT-2 IP/MAC route sync as described in Section 10 of [RFC7432] with a modification described below.¶

2.6. Route sync for IGMP/MLD

This document proposes solving the issue described in Section 1.2 using RT-7 and RT-8 route sync as described by [RFC9251].¶

Local IGMP/MLD join and leave triggers a RT-7/8 route sync to peer PE.¶

2.7. Customer Subnet Route sync using Route type-5

Section 3 of [RFC9136] provides a mechanism to synchronize layer-3 customer subnets between the PEs in order to solve problem described in Section 1.3.¶

Using Figure 2 as example, if PE1 forms the IGP adjacency with CE, it is the only PE with knowledge of the customer subnet R1. BGP on PE1 advertises R1 to remote PEs using L3-VPN signaling, either based on [RFC4364] IP-VPN routes or [RFC9136] EVPN IP Prefix routes.¶

Although PE2 has the same ES connection to the CE, and could provide load balancing to remote PEs, since it has not formed an IGP adjacency with CE, it is not aware of the customer subnet R1.¶

This is solved by PE1 signaling R1 to PE2 using a RT-5 synchronization route. PE2 can then advertise this customer subnet R1 towards the core as if it was locally learned through IGP, and provide load-balancing from the remote PEs. There are two possible options to achieve synchronization:¶

1. ESI based approach.¶

2. IP Gateway based approach.¶

2.8. Mapping for VLAN to ETAG

[I-D.ietf-bess-evpn-ac-aware-bundling] proposes the use of an Attachment Circuit ID Extended Community to carry specific VLAN identification. To avoid the usage of EC, the Ethernet-tag field may be used to signal VLAN/sub-interface identification between service PE peers in RT-2, RT-5, RT-7 and RT-8 as opposed to the Attachment Circuit Extended Community.¶