BFD for VXLAN: Bidirectional Forwarding Detection (BFD) for Virtual eXtensible Local Area Network (VXLAN)
Using fast liveness detection to quickly discover broken tunnel paths in VXLAN overlay networks.
Pallagatti, S., Ed., Mirsky, G., Ed., Paragiri, S., Govindan, V., and M. Mudigonda, "Bidirectional Forwarding Detection (BFD) for Virtual eXtensible Local Area Network (VXLAN)", RFC 8971, DOI 10.17487/RFC8971, December 2020, <https://www.rfc-editor.org/info/rfc8971>.
@misc{rfc8971,
series = {Request for Comments},
number = 8971,
howpublished = {RFC 8971},
publisher = {RFC Editor},
doi = {10.17487/RFC8971},
url = {https://www.rfc-editor.org/info/rfc8971},
author = {S. Pallagatti and G. Mirsky and S. Paragiri and V. Govindan and M. Mudigonda},
title = {{Bidirectional Forwarding Detection (BFD) for Virtual eXtensible Local Area Network (VXLAN)}},
pagetotal = 9,
year = 2020,
month = dec,
} RFC text
Contents
Internet Engineering Task Force (IETF) S. Pallagatti, Ed.
Request for Comments: 8971 VMware
Category: Informational G. Mirsky, Ed.
ISSN: 2070-1721 ZTE Corp.
S. Paragiri
Individual Contributor
V. Govindan
M. Mudigonda
Cisco
December 2020
Bidirectional Forwarding Detection (BFD) for Virtual eXtensible Local
Area Network (VXLAN)
Abstract
This document describes the use of the Bidirectional Forwarding
Detection (BFD) protocol in point-to-point Virtual eXtensible Local
Area Network (VXLAN) tunnels used to form an overlay network.
Status of This Memo
This document is not an Internet Standards Track specification; it is
published for informational purposes.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8971.
Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction
2. Conventions Used in This Document
2.1. Abbreviations
2.2. Requirements Language
3. Deployment
4. Use of the Management VNI
5. BFD Packet Transmission over VXLAN Tunnel
6. Reception of BFD Packet from VXLAN Tunnel
7. Echo BFD
8. IANA Considerations
9. Security Considerations
10. References
10.1. Normative References
10.2. Informative References
Acknowledgments
Contributors
Authors' Addresses
1. Introduction
"Virtual eXtensible Local Area Network (VXLAN)" [RFC7348] provides an
encapsulation scheme that allows the building of an overlay network
by decoupling the address space of the attached virtual hosts from
that of the network.
One use of VXLAN is in data centers interconnecting virtual machines
(VMs) of a tenant. VXLAN addresses the requirements of the Layer 2
and Layer 3 data-center network infrastructure in the presence of VMs
in a multi-tenant environment by providing a Layer 2 overlay scheme
on a Layer 3 network [RFC7348]. Another use is as an encapsulation
for Ethernet VPN [RFC8365].
This document is written assuming the use of VXLAN for virtualized
hosts and refers to VMs and VXLAN Tunnel End Points (VTEPs) in
hypervisors. However, the concepts are equally applicable to non-
virtualized hosts attached to VTEPs in switches.
In the absence of a router in the overlay, a VM can communicate with
another VM only if they are on the same VXLAN segment. VMs are
unaware of VXLAN tunnels, because a VXLAN tunnel is terminated on a
VTEP. VTEPs are responsible for encapsulating and decapsulating
frames exchanged among VMs.
The ability to monitor path continuity -- i.e., perform proactive
continuity check (CC) for point-to-point (p2p) VXLAN tunnels -- is
important. The asynchronous mode of BFD, as defined in [RFC5880], is
used to monitor a p2p VXLAN tunnel.
In the case where a Multicast Service Node (MSN) (as described in
Section 3.3 of [RFC8293]) participates in VXLAN, the mechanisms
described in this document apply and can, therefore, be used to test
the continuity of the path between the source Network Virtualization
Endpoint (NVE) and the MSN.
This document describes the use of the Bidirectional Forwarding
Detection (BFD) protocol to enable monitoring continuity of the path
between VXLAN VTEPs that are performing as VNEs, and/or between the
source NVE and a replicator MSN using a Management VXLAN Network
Identifier (VNI) (Section 4). All other uses of the specification to
test toward other VXLAN endpoints are out of scope.
2. Conventions Used in This Document
2.1. Abbreviations
BFD: Bidirectional Forwarding Detection
CC: Continuity Check
FCS: Frame Check Sequence
MSN: Multicast Service Node
NVE: Network Virtualization Endpoint
p2p: Point-to-point
VFI: Virtual Forwarding Instance
VM: Virtual Machine
VNI: VXLAN Network Identifier (or VXLAN Segment ID)
VTEP: VXLAN Tunnel End Point
VXLAN: Virtual eXtensible Local Area Network
2.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
3. Deployment
Figure 1 illustrates a scenario with two servers: each hosting two
VMs. The servers host VTEPs that terminate two VXLAN tunnels with
VNI number 100 and 200, respectively. Separate BFD sessions can be
established between the VTEPs (IP1 and IP2) for monitoring each of
the VXLAN tunnels (VNI 100 and 200). Using a BFD session to monitor
a set of VXLAN VNIs between the same pair of VTEPs might help to
detect and localize problems caused by misconfiguration. An
implementation that supports this specification MUST be able to
control the number of BFD sessions that can be created between the
same pair of VTEPs. This method is applicable whether the VTEP is a
virtual or physical device.
+------------+-------------+
| Server 1 |
| +----+----+ +----+----+ |
| |VM1-1 | |VM1-2 | |
| |VNI 100 | |VNI 200 | |
| | | | | |
| +---------+ +---------+ |
| VTEP (IP1) |
+--------------------------+
|
| +-------------+
| | Layer 3 |
+---| Network |
+-------------+
|
+-----------+
|
+------------+-------------+
| VTEP (IP2) |
| +----+----+ +----+----+ |
| |VM2-1 | |VM2-2 | |
| |VNI 100 | |VNI 200 | |
| | | | | |
| +---------+ +---------+ |
| Server 2 |
+--------------------------+
Figure 1: Reference VXLAN Domain
At the same time, a service-layer BFD session may be used between the
tenants of VTEPs IP1 and IP2 to provide end-to-end fault management;
this use case is outside the scope of this document. In such a case,
for VTEPs, the BFD Control packets of that session are
indistinguishable from data packets.
For BFD Control packets encapsulated in VXLAN (Figure 2), the inner
destination IP address SHOULD be set to one of the loopback addresses
from 127/8 range for IPv4 or to one of IPv4-mapped IPv6 loopback
addresses from ::ffff:127.0.0.0/104 range for IPv6.
4. Use of the Management VNI
In most cases, a single BFD session is sufficient for the given VTEP
to monitor the reachability of a remote VTEP, regardless of the
number of VNIs. BFD control messages MUST be sent using the
Management VNI, which acts as the control and management channel
between VTEPs. An implementation MAY support operating BFD on
another (non-Management) VNI, although the implications of this are
outside the scope of this document. The selection of the VNI number
of the Management VNI MUST be controlled through a management plane.
An implementation MAY use VNI number 1 as the default value for the
Management VNI. All VXLAN packets received on the Management VNI
MUST be processed locally and MUST NOT be forwarded to a tenant.
5. BFD Packet Transmission over VXLAN Tunnel
BFD packets MUST be encapsulated and sent to a remote VTEP as
explained in this section. Implementations SHOULD ensure that the
BFD packets follow the same forwarding path as VXLAN data packets
within the sender system.
BFD packets are encapsulated in VXLAN as described below. The VXLAN
packet format is defined in Section 5 of [RFC7348]. The value in the
VNI field of the VXLAN header MUST be set to the value selected as
the Management VNI. The outer IP/UDP and VXLAN headers MUST be
encoded by the sender, as defined in [RFC7348].
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Outer Ethernet Header ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Outer IPvX Header ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Outer UDP Header ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ VXLAN Header ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Inner Ethernet Header ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Inner IPvX Header ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Inner UDP Header ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ BFD Control Packet ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Outer Ethernet FCS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2: VXLAN Encapsulation of BFD Control Packet
The BFD packet MUST be carried inside the inner Ethernet frame of the
VXLAN packet. The choice of destination Media Access Control (MAC)
and destination IP addresses for the inner Ethernet frame MUST ensure
that the BFD Control packet is not forwarded to a tenant but is
processed locally at the remote VTEP. The inner Ethernet frame
carrying the BFD Control packet has the following format:
Ethernet Header:
Destination MAC: A Management VNI, which does not have any
tenants, will have no dedicated MAC address for decapsulated
traffic. The value 00-52-02 SHOULD be used in this field.
Source MAC: MAC address associated with the originating VTEP.
Ethertype: This is set to 0x0800 if the inner IP header is IPv4
and set to 0x86DD if the inner IP header is IPv6.
IP header:
Destination IP: This IP address MUST NOT be of one of tenant's IP
addresses. The IP address SHOULD be selected from the range
127/8 for IPv4 and from the range ::ffff:127.0.0.0/104 for
IPv6. Alternatively, the destination IP address MAY be set to
VTEP's IP address.
Source IP: IP address of the originating VTEP.
TTL or Hop Limit: MUST be set to 255, in accordance with
[RFC5881].
The destination UDP port is set to 3784 and the fields of the BFD
Control packet are encoded as specified in [RFC5881].
6. Reception of BFD Packet from VXLAN Tunnel
Once a packet is received, the VTEP MUST validate the packet. If the
packet is received on the Management VNI and is identified as a BFD
Control packet addressed to the VTEP, then the packet can be
processed further. Processing of BFD Control packets received on a
non-Management VNI is outside the scope of this specification.
The received packet's inner IP payload is then validated according to
Sections 4 and 5 in [RFC5881].
7. Echo BFD
Support for echo BFD is outside the scope of this document.
8. IANA Considerations
IANA has assigned a single MAC address of the value 00-52-02 from the
"Unassigned (small allocations)" block of the "IANA Unicast 48-bit
MAC Addresses" registry as follows: the "Usage" field is "BFD for
VXLAN". The "Reference" is this document.
9. Security Considerations
Security issues discussed in [RFC5880], [RFC5881], and [RFC7348]
apply to this document.
This document recommends using an address from the internal host
loopback addresses 127/8 range for IPv4, or an IP4-mapped IPv6
loopback address from the ::ffff:127.0.0.0/104 range for IPv6, as the
destination IP address in the inner IP header. Using such an address
prevents the forwarding of the encapsulated BFD control message by a
transient node, in case the VXLAN tunnel is broken, in accordance
with [RFC1812].
| A router SHOULD NOT forward, except over a loopback interface,
| any packet that has a destination address on network 127. A
| router MAY have a switch that allows the network manager to
| disable these checks. If such a switch is provided, it MUST
| default to performing the checks.
The use of IPv4-mapped IPv6 addresses has the same property as using
the IPv4 network 127/8. Moreover, the IPv4-mapped IPv6 addresses'
prefix is not advertised in any routing protocol.
If the implementation supports establishing multiple BFD sessions
between the same pair of VTEPs, there SHOULD be a mechanism to
control the maximum number of such sessions that can be active at the
same time.
10. References
10.1. Normative References
[RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers",
RFC 1812, DOI 10.17487/RFC1812, June 1995,
<https://www.rfc-editor.org/info/rfc1812>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection
(BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010,
<https://www.rfc-editor.org/info/rfc5880>.
[RFC5881] Katz, D. and D. Ward, "Bidirectional Forwarding Detection
(BFD) for IPv4 and IPv6 (Single Hop)", RFC 5881,
DOI 10.17487/RFC5881, June 2010,
<https://www.rfc-editor.org/info/rfc5881>.
[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
10.2. Informative References
[RFC8293] Ghanwani, A., Dunbar, L., McBride, M., Bannai, V., and R.
Krishnan, "A Framework for Multicast in Network
Virtualization over Layer 3", RFC 8293,
DOI 10.17487/RFC8293, January 2018,
<https://www.rfc-editor.org/info/rfc8293>.
[RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
Uttaro, J., and W. Henderickx, "A Network Virtualization
Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
DOI 10.17487/RFC8365, March 2018,
<https://www.rfc-editor.org/info/rfc8365>.
Acknowledgments
The authors would like to thank Jeff Haas of Juniper Networks for his
reviews and feedback on this material.
The authors would also like to thank Nobo Akiya, Marc Binderberger,
Shahram Davari, Donald E. Eastlake 3rd, Anoop Ghanwani, Dinesh Dutt,
Joel Halpern, and Carlos Pignataro for the extensive reviews and the
most detailed and constructive comments.
Contributors
Reshad Rahman
Cisco
Email: rrahman@cisco.com
Authors' Addresses
Santosh Pallagatti (editor)
VMware
Email: santosh.pallagatti@gmail.com
Greg Mirsky (editor)
ZTE Corp.
Email: gregimirsky@gmail.com
Sudarsan Paragiri
Individual Contributor
Email: sudarsan.225@gmail.com
Vengada Prasad Govindan
Cisco
Email: venggovi@cisco.com
Mallik Mudigonda
Cisco
Email: mmudigon@cisco.com
AI ELI5
On this page
§1VXLAN Tunnels and Failure Detection
VXLAN is a tunneling protocol that encapsulates Ethernet frames inside UDP packets so that virtual networks can overlay a Layer 3 network. A VXLAN Tunnel End Point (VTEP) is a device (usually a hypervisor) that encapsulates traffic from virtual machines and sends it as a VXLAN tunnel to another VTEP. A VNI (VXLAN Network Identifier or segment ID) identifies which virtual network the traffic belongs to.
The challenge is detecting when a tunnel path fails. If the underlay network loses connectivity between two VTEPs, higher-layer protocols may not notice for minutes. Applications relying on the tunnel suffer a long outage. Bidirectional Forwarding Detection (BFD) is a lightweight protocol where two endpoints exchange hello messages (called Control packets) at regular intervals. If one endpoint stops receiving hellos from the other within a few hundred milliseconds, BFD declares the session down, signaling a failure. BFD is designed to be fast and stateless on the endpoint, making it ideal for monitoring tunnel health.
A virtual machine sends data through a tunnel. If the tunnel breaks, nobody upstream notices for a while. BFD is like two people on opposite ends of the tunnel sending quick hand signals. If one stops signaling, the other knows something is wrong within a second.
§2Encapsulation and the Management VNI
BFD packets are encapsulated in VXLAN exactly like any other traffic, but with a specific encapsulation:
The outermost headers are Ethernet, IP, and UDP (the VXLAN underlay). Next is the VXLAN header with the Management VNI field set to a designated management segment ID (typically VNI 1, but operator-configured). Inside the VXLAN envelope are inner Ethernet, IP, and UDP headers, followed by the BFD Control packet.
The inner Ethernet destination MAC SHOULD be set to 00-52-02, a dedicated MAC address for BFD on VXLAN (assigned by IANA). The inner IP destination address SHOULD be from the loopback address range 127/8 for IPv4 or ::ffff:127.0.0.0/104 for IPv6. This ensures that if the VXLAN tunnel is broken and the packet is accidentally forwarded by a router, it will be discarded instead of creating a loop.
The inner source IP is the VTEP's own IP address. The destination UDP port is 3784, the standard BFD port. All packets on the Management VNI are processed locally at the receiving VTEP and MUST NOT be forwarded to tenants.
The BFD hello packet is wrapped in layers: VXLAN layer outside, then loopback IP address inside so it gets processed right away and not forwarded anywhere if something goes wrong.
- Outer EthernetCarries the underlay network packets
- Outer IP/UDPRoutes packets through the underlay network
- VXLAN headerIdentifies the Management VNI
- Inner EthernetDestination MAC 00-52-02 for BFD
- Inner IP/UDPLoopback address, port 3784
- BFD Control packetThe hello message
§3BFD Sessions and Multiple Tunnels
A separate BFD session can be established for each tunnel between a pair of VTEPs, or a single BFD session can monitor all tunnels between them. When a BFD session goes down, all tunnels it monitors are presumed to have failed. Implementations MUST be able to limit the number of BFD sessions between the same pair of VTEPs to prevent resource exhaustion.
On transmission, a VTEP encapsulates a BFD Control packet as described above, ensuring that BFD packets follow the same forwarding path as VXLAN data packets through the sender system. On reception, the receiving VTEP validates the packet: it must have arrived on the Management VNI and be identified as a BFD Control packet addressed to the VTEP itself. The inner IP payload is then validated according to the single-hop BFD specification (RFC 5881). Echo mode BFD (where a VTEP echoes packets back to the sender) is outside the scope of this document.
Both ends of the tunnel send hello packets on a special management channel. If both sides receive hellos at regular intervals, the tunnel is okay. If one side stops receiving hellos, it knows the tunnel is broken.
§4Deployment in Virtual and Physical Networks
The RFC applies to both virtualized hosts in hypervisors (the primary use case) and non-virtualized hosts connected to physical VTEPs in switches. In a data center, a hypervisor hosts virtual machines across multiple VNI segments and terminates multiple VXLAN tunnels. BFD sessions monitor the health of the tunnels themselves, independent of tenant traffic.
When a BFD session detects a tunnel failure, the VTEP can trigger failover: if multiple paths exist to a remote VXLAN segment, traffic is rerouted. If a Multicast Service Node (MSN) participates in VXLAN to replicate multicast traffic, BFD can monitor the path from the source Network Virtualization Endpoint (NVE) to the MSN. The RFC does not address service-layer BFD between tenants, which would be indistinguishable from normal data packets and is outside the scope.
In a data center with many virtual machines, the hypervisor monitors multiple tunnel connections. When it detects a broken tunnel with BFD, it can reroute traffic to another path.