Time-Sensitive Networking for Real-Time Ethernet Communication

Abstract

Real-time applications require guaranteed data delivery within a limited amount of time. Such applications may not tolerate the latency in the retransmission of lost data. Time-Sensitive Networking (TSN) is a set of protocols that allow Ethernet networks to guarantee data delivery with low data loss and bounded latency. Network’s setup and configuration are automated, including resource allocation and scheduling of data transmission in such a manner that it guarantees data delivery within a bounded latency. TSN consists of many protocols that are standardized and continuously enhanced. Thus, TSN is adopted widely by industry and implemented in many application domains. This paper introduces the TSN, explains how TSN provides deterministic communication for Ethernet networks. It also shows TSN advantages, some application domains, and the current challenges.

Keywords: Time-Sensitive Networking, TSN, Audio Video Bridging, AVB, real-time Ethernet communication, deterministic networking, IEEE 802.1, deterministic Ethernet, stream reservation, TSN applications, TSN challenges

Introduction

The current advances in computing allowed us to operate autonomous vehicles and establish a multi-domains Internet of Things (IoT) networks such as smart cities, manufacturing, and healthcare. All these applications are time-critical and require reliable communication. Other time-critical applications are audio and video applications, such as vehicular entertainment systems, theatres equipment, and live video streaming. In typical Ethernet networks, data latency and retransmission for lost data may not meet the time requirement for time-critical applications.

Time-Sensitive Networking (TSN) enables bridged networks to exchange data with guaranteed data delivery, bounded latency, and low data loss [802.1-12b] [wikipedia01]. It allows devices to send and receive many types of data (media and non-media) at the same time [McCarthy16]. Also, TSN employs automatic setup and configuration of the network, reducing the network administration complexity with a large number of devices [Bhattacharjee18]. Many organizations have collaborated to standardize TSN, such as the Institute of Electrical and Electronics Engineers (IEEE), the Avnu Alliance, and the Industrial Internet Consortium [McCarthy16] [ni19].

From a historical perspective, TSN was built mainly for audio and video applications [McCarthy16] [ni19]. The work on supplying time-sensitive audio and video streams started in 2005 under the IEEE 802.1 Audio Video Bridging (AVB) Task Group [802.1-12a] [802.1BA-11 ]. With the continuous enhancement, it expanded to cover broader domains such as industrial and automotive applications. To reflect the expansion, IEEE renamed AVB in 2012 to Time-Sensitive Networking (TSN) [802.1-12b]. In 2015, IEEE merged the TSN Task Group and the Internetworking Task Group.

This paper is introductory to TSN and intends to answer the following questions. What are the types of devices in the network that comply with TSN? How are these devices connected? What are the protocols and mechanisms that enable deterministic communication? Where can TSN be used? What are the challenges in developing TSN in a variety of application domains?

Network Components

A time-sensitive network consists of bridges, which manage the exchange of frames, and endpoints (or host nodes), which execute applications [Zurawski17]. Host nodes themselves are connected directly or indirectly through one or more bridges. Bidirectional links connect bridges and endpoints through incoming and outgoing physical ports. Figure 1 shows an example of a bridged network, where A, B, C, D, and E represent bridges, and 1, 2, 3, 4, 5, 6, and 7 represent endpoints.

Figure 1: Example of Time-Sensitive Network

Ethernet frames are the only allowed data to be exchanged [Zurawski17]. Figure 2 shows the format of the Ethernet frame. For managing the transfer of data between endpoints, bridges need to perform switching and traffic shaping. For switching, the bridge decides which port to forward the received frame to, based on the MAC (Media Access Control) destination address and the VLAN (Virtual Local Area Network) identifier. For forwarding, bridges perform traffic shaping by sequencing the Ethernet frames using the Priority Code Point information in the VLAN tag. The Priority Code Point is three bits size, so eight priorities can be implemented. Before data exchange, the network should be set up, and endpoints should agree on specific configurations [Bhattacharjee18].

Figure 2: TSN Ethernet Frame

Setup and Configuration

In TSN, the network setup and configuration are automated employing Software-Defined Networking (SDN) [Bhattacharjee18]. SDN is an efficient network management approach that allows dynamic and programmable network configuration [wikipedia03]. TSN uses YANG (Yet Another Next Generation) data modeling language, which defines the format of configuration messages and status reports exchanged between nodes [P802.1Qcp-18]. Before any exchange of data frames, bridges should reserve the required bandwidth and agree on the time synchronization mechanism, scheduling mechanism, and QoS (Quality of Service) metrics. The approaches used for setup and configuration have many advantages.

TSN Benefits

TSN has many advantages. First, the automatic setup and configuration of TSN reduce the management complexity [electronicdesign01] [Intel18]. It can also improve the security of the network eliminating the attack surface introduced by human errors. Second, devices from different vendors can exchange media and non-media data using different protocols without being limited to one vendor’s devices and protocols. Thus, TSN increases the connectivity of devices. Third, the high compatibility between devices decreases in the use of proprietary protocols. As a result, TSN lessens the overall cost, including network administrator personnel, the number of devices handling multiple kinds of data, and the use of proprietary protocols. Fourth, TSN is a developing open standard, which applies to many domains. Finally, the resource allocation mechanism described in the next section will show how TSN can enhance the scalability of the network.

TSN Components

As mentioned earlier, before any exchange of data frames and as part of the configuration, bridges should reserve the required bandwidth and agree on the time synchronization mechanism, scheduling mechanism, and QoS metrics. This section discusses how the bandwidth is reserved and the time synchronization mechanism. It also describes how reliability and bounded latency are maintained.

Bandwidth Reservation

For reserving the bandwidth for a stream of data, a listener (destination) endpoint firstly indicates what streams it expects to receive [Zurawski17] [wikipedia02]. Also, a talker (source) endpoint advertises the streams it will send, which should be supported by the connecting bridges. The talker announces the streams via a message that indicates the specifications of the streams, such as quality of service requirements, and the maximum latency. The bridge then measures the resources necessary and propagates the message to the next bridge until it reached the listener. The listener then replies with “listener ready” signal, which propagates back to the talker. The “listener ready” message means that all connecting bridges have reserved required bandwidth and can guarantee the QoS requirements. If a bridge is not able to provide the bandwidth needed, it raises a “talker failed” message.

Stream Reservation Protocol (SRP) is the associated protocol to bandwidth reservation in endpoints to guarantee the QoS level agreed on [Zurawski17]. A stream can be terminated to deallocate the bandwidth. The automated bandwidth reservation configuration adds flexibility to the network that uses TSN and enhances the scalability of the network [electronicdesign01] [TTTech] [Bhattacharjee18]. Adding more endpoints does not compromise the network performance, and allows topologies to be optimized better according to the reserved bandwidth. Another component that contributes to providing deterministic communication is time synchronization.

Time Synchronization

In TSN, all interconnected devices share an identical time, synchronized to the clock of one node, either a bridge or an endpoint [Zurawski17]. This node is named the grandmaster, and it is selected using the Best Master Clock Algorithm (BMCA). BMCA receives “announce messages” send by capable nodes to be a grandmaster and decides the grandmaster node based on predefined criteria. The time is maintained between nodes using a synchronization spanning tree, where the grandmaster is the root of the tree and broadcasts its time periodically. For applications that need only a shared clock, nodes can synchronize their clocks from an external reference like Global Positioning System (GPS) [Bhattacharjee18].

In addition to time synchronization, the grandmaster node is responsible for measuring forwarding delays in bridges [802.1-12a] [ni19]. The communication delays on each communication link in the network are also measured, and it is called “peer delay” mechanism. Additionally, the relative difference of the clock rates is also measured and should be within ±0.1 parts per million. Shared clock and bounded delays are critical for the scheduled frames to be received at the expected time. An enhancement to the protocol suggests implementing a redundant grandmaster as a clock synchronization fault-tolerance mechanism. Although this mechanism would enlarge the application domain of TSN, it would be too simple for some safety-critical applications to consider.

In summary, a grandmaster node is responsible for distributing its time and measuring the delays periodically to maintain the schedule. Fault-tolerance mechanisms for time synchronization are under consideration, and it should be sophisticated enough to include broader application domains. Bandwidth reservation and time synchronization are not enough for providing deterministic communication. TSN should also minimize data loss.

Reliability

In any network, frame loss is a possibility. However, time-sensitive networks may not tolerate the latency resulted from the retransmission of lost frames [Zurawski17]. TSN reliability defines the mechanisms that prevent the loss of data and hence guarantee data delivery at the expected time [802.1-12a]. In TSN, delivering data is as important as delivering it at the specified [youtube01]. In this section, we introduce the main protocols associated with reliable data delivery.

802.1Qca (Path Control and Reservation) allows the same frame to be sent over multiple paths concurrently to ensure that the frame is delivered. It uses IS-IS (Intermediate System to Intermediate System) for setting up the parallel paths. At the destination node or certain bridges, 802.1CB (Frame Replication and Elimination for Reliability) defines the mechanisms for eliminating the redundant copies according to the sequence number attached to each frame. [802.1-12a] [Zurawski17]

802.1Qci (Per-Stream Filtering and Policing) protocol is responsible for detecting and mitigating disruptive transmissions by other systems in a network through policing and filtering [802.1-12a] [P802.1Qci-17]. Detecting such behavior in the network is crucial because there are no standards that enforce each system in the network to comply with certain configurations for the exchange of frames. Unexpected behavior affects the whole network, not just the misbehaved stream.

In summary, TSN allows redundant frames to be sent through multiple paths to guarantee data transfer at the scheduled time. However, endpoints should delete the redundant copies. TSN empowers bridges with detection and resolution mechanisms for any incompliance of the agreed configurations. To bound latency, TSN offers many traffic shaping mechanisms.

Bounded Low Latency

As mentioned earlier, bridges prioritize frames based on the value of the Priority Code Point in the frame’s VLAN tag [Zurawski17]. As the Ethernet frame allows eight priorities, bridges commonly implement eight queues on each ongoing port. When many frames are ready to be sent from the same port, each one is added to its respective queue and sent one by one. This mechanism is called the strict-priority. This section discusses the limitations of the strict-priority along with alternative approaches.

In strict-priority traffic-shaping, priority loses its significance if the number of high priority messages is large enough to make the frames wait in the queue [Zurawski17]. Also, if higher priority messages are always there, the network may not serve low priority applications for a significant amount of time. These applications may not be able to function efficiently. Therefore, more complicated mechanisms are necessary to guarantee the delivery of different priority frames within their bounded latency.

TSN provides mechanisms for traffic shaping to provide bounded low latency [Zurawski17]. Standardized approaches are Credit Based Shaper, Time-Aware Shaper, and Asynchronous Traffic Shaping [wikipedia01]. Time-critical traffic implements several priority classes. A credit-based shaper implements mainly two priority classes A and B. The sharper assign Class A the highest priority and dedicates it to stream reservation. The destination endpoint must receive Class A traffic within 125 microseconds with a maximum latency of 2 milliseconds across a maximum of seven hops.  The destination endpoint must receive Class B traffic within 250 microseconds with a maximum latency of 50 milliseconds across a maximum of seven hops. The maximum bandwidth agreed on must not be exceeded. Control traffic in the credit-based shaper has the lowest priority.

The time-aware shaper introduces a new priority class called CDT (Control Data Traffic). It is for real-time control data and command streams and has higher precedence than Class A traffic. The destination endpoint must receive the CDT traffic within 0.5 milliseconds with a maximum latency of 100 microseconds across a maximum of five hops. The time-aware shaper is believed to be significantly stable. Asynchronous traffic shaper implements different approaches for time synchronization and scheduling priority classes, which this paper will not discuss.

In summary, the strict priority traffic-shaping mechanism is not adequate for real-time applications. More sophisticated approaches are Credit-Based Shaper, Asynchronous Traffic Shaper, and Time-Aware Shaper. These mechanisms implement different traffic classes and priorities. Traffic-shaping is the last component to discuss.

Summary

Before the exchange of the data frame, bridges dynamically reserve the required bandwidth for each stream. Bridges also share a synchronized clock and allow the sending of redundant copies of frames in multiple paths. Additionally, bridges schedule traffic to deliver different traffic classes within a limited amount of time. Many application domains need these requirements. However, deploying such requirements in some applications is challenging.

Application Domains and Challenges

As mentioned earlier, TSN is adopted widely in many application domains. However, TSN has many challenges to be addressed. This section demonstrates some use cases of TSN and highlights some of the present issues.

Application Domains

As mentioned earlier, TSN was built mainly for audio and video media applications such as in-car entertainment, audio and video professional equipment, and automotive infotainment [Zurawski17]. Besides audio and video media applications, TSN is useful for non-media applications such as industrial measurement, control, and automation [McCarthy16] [ni19]. Gaj et al. mention that offering low-cost devices with the lowest configuration is a goal for TSN to deploy plug-and-play functionality. Also, Bhattacharjee stated that TSN is designed to provide the low-level, real-time, dynamic, virtualized, and distributed architecture for wired cyber-physical systems [Kovácsházy18].

By simulation, TSN is found to be suitable for specific in-car communications [Gaj17]. Plug-and-play functionality is not useful for in-car communication as the network infrastructure installation is fixed. However, the support of multiple streams decreases the complexity of the network infrastructure. Gaj et al. survey the work on TSN amendments evaluation in in-car communication and concluded that the current redundancy techniques lack flexibility in the stream reconfiguration and automatic stream reservation. Also, implementing TSN raised the network configuration overhead. Finally, it was hard to find fully functioning implementations of TSN to simulate.

Challenges

Although TSN can reduce the attack surface of human errors, it does not promise security. TSN relies on the IEEE 1588 protocol to achieve time synchronization. So, TSN is subject to IEEE 1588 vulnerabilities [Bhattacharjee18]. All connected devices should implement adequate measures to secure the clock synchronization process. Other security challenges are implementing wire-speed cryptographic security, authentication mechanisms, and integrity check since high-priority scheduled traffic should be immediately transmitted. Lack of authentication or adequate analysis before resending the data allows the injection of unauthenticated malicious traffic.

Many devices implementing TSN are currently available and deployed by leading companies such as Intel, Cisco, National Instruments, and Hirschmann [McCarthy16] [ni19]. Previously deployed devices without TSN can still implement time synchronization. However, not all of these devices can have full deterministic communication functionalities depending on the complexity level of the network and platform setup of the application.

Summary

TSN was intended for audio-video applications and then extended to include non-media applications such as time-critical measurement, control, and safety-critical applications. Vendors should consider implementing extra security measures, and future enhancement should also consider overcoming the existing vulnerabilities. Adding TSN functionalities to existing networks is application-specific.

References

• [Zurawski17] Richard Rawski, "Industrial Communication Technology Handbook," CRC Press, 2017, ISBN: 1138071811, 9781138071810 (Safari Book)
• [McCarthy16] Bob McCarthy, "Sound Systems: Design and Optimization: Modern Techniques and Tools for Sound System Design and Alignment," CRC Press, 2016, ISBN: 1317911091, 9781317911098 (Safari Book)
• [Bhattacharjee18] Sravani Bhattacharjee, “Practical Industrial Internet of Things Security,” Packt Publishing, 2018, ISBN: 9781788832687, 9781788832687 (Safari Book)
• [Gaj17] Piotr Gaj, Andrzej Kwiecień, and Michał Sawicki, "Computer Networks: 24th International Conference, CN 2017, Lądek Zdrój, Poland, June 20–23, 2017, Proceedings," Springer, 2017, ISBN: 3319597671, 9783319597676 (Springer Book)
• [Kovácsházy18] Tamás Kovácsházy, "Distributed Architecture for Real-time Cyber-physical System, Time-Sensitive Networks," 19th International Carpathian Control Conference (ICCC), IEEE, 2018, pp. 1-6, https://ieeexplore.ieee.org/document/8399588
• [Intel18] "Time-Sensitive Networking: From Theory to Implementation in Industrial Automation," https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/wp/wp-01279-time-sensitive-networking-from-theory-to-implementation-in-industrial-automation.pdf [Intel white paper on TSN]
• [ni19] "Time Sensitive Networking (TSN) Frequently Asked Questions,” https://www.ni.com/en-us/innovations/white-papers/18/time-sensitive-networking--tsn--frequently-asked-questions.html [National Instruments white paper on TSN]
• [TTTech] "Time Sensitive Networking (TSN)," https://www.tttech.com/technologies/time-sensitive-networking-tsn/ [TSN overview, key benefits and product line by TTTech companies]
• [wikipedia01] "Time-Sensitive Networking," https://en.wikipedia.org/wiki/Time-Sensitive_Networking  [Standards of time-sensitive transmission of data over deterministic Ethernet networks]
• [wikipedia02] “Stream Reservation Protocol,” https://en.wikipedia.org/wiki/Stream_Reservation_Protocol [Explanation of stream reservation protocol and used mechanisms]
• [electronicdesign01] "What’s the Difference Between Ethernet and Time-Sensitive Networking?," https://www.electronicdesign.com/industrial-automation/what-s-difference-between-ethernet-and-time-sensitive-networking [TSN overcome Ethernet limitations in terms of scalability and latency]
• [youtube01] “Deterministic Networking for Real-Time Systems (Using TSN),” https://www.youtube.com/watch?v=oxURD2rr4Y4&t=535s [A video explaining TSN deterministic networking for real-time systems]
• [802.1-12a] “Introduction to IEEE 802.1: Focus on the Time-Sensitive Networking Task Group,” http://www.ieee802.org/1/files/public/docs2018/tsn-farkas-intro-0318-v01.pdf [An introductory presentation to Time-Sensitive Networking standards]
• [802.1-12b] "Time-Sensitive Networking (TSN) Task Group," https://1.ieee802.org/tsn/ [Base and published TSN Standards and ongoing TSN Projects]
• [802.1BA-11] “802.1BA - Audio Video Bridging (AVB) Systems,” http://www.ieee802.org/1/pages/802.1ba.html [Introducing the standard along with lists of its drafts, and associated presentations]
• [P802.1Qci-17] “P802.1Qci – Per-Stream Filtering and Policing,” https://1.ieee802.org/tsn/802-1qci/ [P802.1Qci protocol definition, current status, presentations, and archive documents]
• [P802.1Qcp-18] “P802.1Qcp – Bridges and Bridged Networks Amendment: YANG Data Model,” https://1.ieee802.org/tsn/802-1qcp/ [P802.1Qcp protocol definition, current status, presentations, and archive documents]
• [wikipedia03] “Software-defined networking,” https://en.wikipedia.org/wiki/Software-defined_networking [description of Software-defined networking technology along with applications]

Appendix

List of Acronyms

AVB  Audio Video Bridging

BMCA Best Master Clock Algorithm

CDT Control Data Traffic

GPS Global Positioning System

IEEE The Institute of Electrical and Electronics Engineers

IS-IS Intermediate System to Intermediate System

IoT Internet of Things

MAC Media Access Control

QoS Quality of Service

SDN Software-Defined Networking

SRP Stream Reservation Protocol

TSN Time-Sensitive Networking

VLAN Virtual Local Area Network

YANG Yet Another Next Generation