Telecommunication Systems 8 (1997) 293–301
293
Simulation of deep space network traffic under various loading conditions Victor Yeeman Lo Jet Propulsion Laboratory, California Institute of Technology, M/S 161-241, 4800 Oak Grove Dr., Pasadena, CA 91109, USA
This is a follow-up report on the end-to-end data link performance of the NASA deep space network. The focus of this report is the effect of traffic loading on data delivery. Architecture of the existing network is modeled via COMNET. The contribution of the data latency to the overall acquisition time is simulated as a function of network traffic conditions. The results show steady-state latencies in the range of tenths of milli-seconds under a mild loading condition. During heavy traffic, a delay rate of over a hundred seconds of increased delay per hour of network operation time is observed. This is primarily due to the queueing delays associated with large volume of data stored in the buffers throughout the link.
1.
Introduction
In this report, in-depth simulation analyses are performed on the impact of network traffic loading on the latency of end-to-end data delivery for various deep space and near Earth missions [1]. The architecture of the existing communication network is presented in section 2. This is followed by a description on the features available for network simulation under the COMNET1 environment. These features are ultilized to construct a COMNET model of the deep space network (DSN) in section 3. Next, a network performance under mild and heavy traffic loading is investigated in section 4. Finally, a summary of the findings is shown in section 5. 2.
DSN ground communication system
The local area networks (LAN) at various deep-space communication complexes are 10BASE-T ethernets using carrier sense multiple access with collision detection (CSMA/CD) algorithms. Following the LAN at each station, the telemetry data received from a spacecraft flows through a T1 transmission line relatively unobstructed. Next, the downlink signal is routed through either a high-speed or low-speed pipe according to the preset algorithm in a Cisco Router2 . In most applications, a wider bandwidth associated with a high speed pipe is used to support high rate telemetry. 1 2
COMNET III is a communication network simulation software by CACI Products Company. Cisco Router is a network product manufactured by Cisco Systems, Inc.
J.C. Baltzer AG, Science Publishers
294
V.Y. Lo / Simulation of deep space network traffic
A wide area network (WAN) covers the DSN world-wide data transport from remote stations. Data packets with labels designating Jet Propulsion Laboratory (JPL) are routed through the JPL T1 circuit before entering the JPL LAN. The telemetry and monitor data are then processed by the ground interface facility (GIF) servers where records of data files are written. 3.
Network simulation via COMNET
Similar to the SPW3 simulation used in the front-end acquisition analysis, COMNET is a block simulation program with a layered architecture. The software is designed to accommodate a wide variety of network topologies and routing algorithms. These include LAN, WAN, SUBNET and TRANSIT-NET in the topology library. Specific network protocols like X.25 and 10BASE-T ethernet can be used. Links with specific bandwidths like T1, ISDN or any other user-defined values are supported. Nodes, switches, gateways, bridges and buffer models are also available for interconnection. The router menu has built-in generic as well as commercial router models (e.g., Cisco). Global and local transport protocols can be set with known TCP/IP, UDP/IP or user-defined protocols. Global and local routing tables can be used to implement minimum hop or minimum penalty routing. Frame, packet and header sizes can all be set for specific telemetry modes for various missions. Traffic control like sliding window and enhanced sliding window with flow control or back-off algorithms are available. Finally, network performance at various nodes can be monitored at real-time or for post-processing. 3.1. COMNET model on the Deep Space Network (DSN) Considering the ground system architecture described in section 2, modifications are made on the generic subnet and transit-net COMNET models to construct the three deep- space complexes, the ground WAN as well as the space flight centers at JPL and Goddard. In the next layer of this network hierarchy, data sources at stations and data sinks at JPL and GSFC are installed. Bandwidths of all transmission circuits are then set according to the capability of the existing ground link. Details of COMNET DSN model are described next. 3.2. Modeling of DSN communication system A COMNET simulation model on the DSN ground communication system has been constructed (figure 1). The architecture is based on existing DSN configuration. All stations have high-rate and low-rate pipes for data transport (figure 2). The station at Goldstone/USA (GDSCC) has six telemetry groups (figure 3) while Canberra/Australia (CDSCC) and Madrid/Spain (MDSCC) each have five. Each supports 3
Signal Processing WorkSystem (SPW) is the registered trademark of the Alta Group.
V.Y. Lo / Simulation of deep space network traffic
295
Figure 1. COMNET model of the DSN communication system.
Figure 2. Architecture of the existing network.
a combination of deep-space as well as near-Earth missions. These structures are embedded in the second layer of the station blocks. A set of sample missions has been chosen. This includes CASSINI, MGS, AXAF, SOHO, GEOTAIL at Goldstone, MPF, TOPEX and ERBS in Canberra, and RADARSAT, DSPSE, STRV, UARS and NOAA
296
V.Y. Lo / Simulation of deep space network traffic
Figure 3. Models of the Telemetry and Command Groups at GDSCC.
in Madrid. Additionally, there are eight command modulation assemblies (CMA) in each station servicing uplink commands. Both incoming telemetry and outgoing command traffic are spread out over JPL, Goddard, the station complexes and the GCF cloud. The telemetry packets (transport frames) are transported via TCP/IP protocol and then processed by GIFs and simulation subsystems (SIMs). By choosing a mixture of high and low-rate data at various transport frame sizes, this simulation provides a good estimate on the data transport latency across DSN under various loading conditions. 4.
Network performance under mild and heavy loading
To evaluate the impact of network traffic on near-real-time telemetry, we simulate both mild-load by choosing mostly low data rates from the missions and heavy-load by selecting high data rates. Since there is a large number of JPL supported deep-space and near-Earth missions, the objective of this simulation study is to examine the worst case network condition rather than to document all possible network behaviors. 4.1. Network traffic under mild loading condition By assuming mostly low rate data (aggregate rate < 200 kbps) from both nearEarth and deep-space missions, a light traffic condition is set for the simulation. The delays at each local area network for different deep-space communication complexes (DSCC) are shown in figure 4. Due to the additional telemetry group, GDSCC suffers
V.Y. Lo / Simulation of deep space network traffic
297
Figure 4. Frame delays at various DSN station complexes.
Figure 5. Itemized transmission delay between GDSCC and JPL.
a slightly higher latency than the LANs in the other two complexes. In general, the 10BASE-T ethernet is more than adequate to handle the mild traffic load. In figure 5, transmission delays from Goldstone to JPL are itemized. The low-speed pipe, the high-speed pipe and the JPL T1 line present themselves as potential bottlenecks in the
298
V.Y. Lo / Simulation of deep space network traffic
Figure 6. Cassini telemetry and command data delay under mild traffic condition between Goldstone to JPL.
ground data transmission. This observation will be confirmed in the next section. With the duplex circuit, there are also command data flowing in the opposite direction into the station complexes. Under this mild network loading configuration, the telemetry and command transmission and delivery delay for Cassini is shown in figure 6. The data delivery delay is normally larger than the data transmission delay because of the waiting time associated with the acknowledgment. An average of 32 msec at 142 kbps is negligible compared to the latency in other DSN subsystems. 4.2. Network traffic under heavy loading condition Under the heavy network loading condition (aggregate rate > 1.48 Mbps), the itemized link utilization report of various DSN groundlink circuits is summarized in table 1. By comparing percentage ultility from mild to heavy loading condition, we can see a dramatic increased link utilization at the JPL T1 line. The percentage of link utilization is an indicator on the average spectrum usage of the available bandwidth. This confirms our earlier observation of a possible bottle-neck at the JPL T1 line. Furthermore, a high link utilization induced queueing delays of packets stored in the router buffers. This affects the overall transmission delay. For high rate Cassini telemetry, the frame transmission delay is shown in figure 7. The contrast between figure 6 and figure 7 is the linearly increasing delays (144 seconds increased delay per hour of network time) at heavy traffic as compared to a stable small latency at light traffic. As more packets are waiting in queue, the buffer usage increases. The rate of buffer usage is a monotonic function of the link utilization. This relationship is illustrated in figure 8 on various buffers.
V.Y. Lo / Simulation of deep space network traffic
299
Table 1 Channel utilization on dsn links under mild and heavy loading. Link GDSCC.GDSCC-ETHERNET CDSCC.CDSCC-ETHERNET JPL.Link6 MDSCC.MDSCC-ETHERNET GCF.GDSCC 1024 kbps FROM GCF.Router G1 FROM GCF.Router G2 GCF.CDSCC 672 kbps FROM GCF.Router C1 FROM GCF.Router C2 GCF.MDSCC 672 kbps FROM GCF.Router M1 FROM GCF.Router M2 GCF.MDSCC T1 FROM GCF.Router M1 FROM MDSCC.Gateway@GCF GCF.CDSCC T1 FROM GCF.Router C1 FROM CDSCC.Gateway@GCF GCF.JPL T1 FROM GCF.Router JPL FROM JPL.Gateway@GCF GCF.WAN GCF.GSFC 1024 kbps FROM GCF.Router GSFC FROM GCF.Router GSFC GCF.GDSCC T1 FROM GCF.Router G1 FROM GDSCC.Gateway@GCF GCF.GDSCC 448 kbps FROM GCF.Router G1 FROM GCF.Router G2 GCF.CDSCC 64 kbps FROM GCF.Router C1 FROM GCF.Router C2 GCF.MDSCC 64 kbps FROM GCF.Router M1 FROM GCF.Router M2 GCF.GSFC 200 kbps FROM GCF.Router GSFC FROM GCF.Router GSFC GCF.GSFC T1 FROM GCF.Router GSFC FROM GSFC.Gateway@GCF GSFC.Link6
Mild loading % utility
Heavy loading % utility
3.56 0.48 2.14 0.32
9.98 5.37 14.95 6.02
35.88 14.78
98.55 24.24
6.85 4.46
78.83 38.08
4.52 3.14
91.18 52.85
1.60 2.13
24.55 41.74
2.21 3.19
18.09 36.83
14.17 4.02 9.10
99.70 32.67 42.02
22.35 14.23
54.37 35.46
10.12 24.09
16.62 66.52
0.59 0.92
2.82 1.56
4.71 6.24
56.25 34.36
3.62 5.53
44.25 34.36
2.61 3.34
37.67 22.71
15.24 9.92 2.22
41.15 26.49 5.88
300
V.Y. Lo / Simulation of deep space network traffic
Figure 7. Cassini telemetry and command transmission delay.
Figure 8. JPL, GDS, MDS and CDS buffers buildup.
5.
Summary
The simulation shows that at moderate traffic when most of the missions are running at low to medium data rates, the delays through the existing GCF are in the range of tenths of milli-seconds. However, during high-rate traffic, congestion induces a delay rate of over a hundred seconds per hour of network operation time. This
V.Y. Lo / Simulation of deep space network traffic
301
heavy traffic condition will eventually lead to data loss after the buffers in the link are filled up. From a network management perspective, there are different ways to handle this: One approach is to store most high-rate telemetry data on tapes at the deep-space complex; the tapes can be sent back at a later time when the network traffic is light. Another option is to deliver the tapes to JPL via commercial courier service. This is a common practice for the transport of science data including those from the very long baseline interferonmetry (VLBI) and the radio science. The priority here is to keep the data path open for emergency mode real-time telemetry. For missions supported by the DSN stations, sufficient time delays should be allocated for the delivery of science and non-emergency engineering data back to JPL so as to avoid congestion of the network. The COMNET simulation results also reveal the weakness of the existing DSN ground network. Depending on future mission requirements on real-time data transport, various degrees of network upgrades may be needed to meet future demands. A recent proposal on the Advanced Communication Service will improve the network traffic by adding T1 lines in the bottle-neck circuits. There is also a proposal to install an ATM backbone for high-speed links between deep-space complexes and JPL. Another low-cost approach is to set up a deep space intranet utilizing virtual private circuits on the internet. Data security and reliability can be accomplished by encryption and IP filters. To determine the impacts of these upgrades on the quality, quantity, continuity and latency (QQCL) of the end-to-end mission data transport system, more network simulations will have to be performed. These simulation results can be verified by comparing to the captured traffic profile from a network sniffer connected to one of the existing test strings. Thus, network performance can be evaluated before any proposed upgrades are implemented. Acknowledgement This paper presents the result of research carried out at the Jet Propulsion Laboratory, California Institute of Technology, under contract with the National Aeronautics and Space Administration. References [1] V.Y. Lo, Characterization and simulation of end-to-end data link performance of the NASA deep space network, in: Proc. ICC ’97, Montreal, Canada.
