| Internet-Draft | IP in Deep Space | November 2024 |
| Blanchet, et al. | Expires 7 May 2025 | [Page] |
Deep space communications involve long delays (e.g., Earth to Mars is 4-20 minutes) and intermittent communications, because of orbital dynamics. The IP protocol stack used on Internet is based on assumptions of shorter delays and mostly uninterrupted communications. This document describes the architecture of the IP protocol stack tailored for its use in deep space. It involves buffering IP packets in IP forwarders facing intermittent links, signaling buffer storage near capacity, adjusting transport protocols configuration and application protocols timers. This architecture applies to Moon, Mars or in general interplanetary networking.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 7 May 2025.¶
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Deep space communications involve long delays (e.g., Earth to Mars is 4-20 minutes) and intermittent communications, because of orbital dynamics. Up to now, communications have been done on a layer-2 point to point basis, with sometimes the use of relays, but no layer-3 networking has been in use. [RFC4838] reports an assessment done around 25 years ago concluding that the IP protocol stack was not suitable for deep space networking. This result lead to the definition of a completly new protocol stack based on a store-and-forward paradigm implemented in the Bundle Protocol(BP) [RFC9171] and its various components, such as convergence-layer adapters([RFC9174], [RFC7122]) and BP Security(BPSEC)[RFC9172].¶
More recently, space agencies are planning to deploy IP networks on celestial bodies, such as Moon[ioag] or Mars[ioag-mars], surface, and orbital vicinity, using layer2 technologies such as WIFI or 5G. On the surface, it is planned to have high density of network nodes of space stations and habitats.¶
Mission concepts are also based on a cluster of multiple network nodes in close proximity at Langrange points.¶
A previous document[I-D.many-deepspace-ip-assessment] revisited the initial assessment of not using IP and concluded that the IP stack is in fact viable in deep space, given the IP stack evolution that happened since the initial assessment. This document defines an architecture to use IP in deep space networking. IP in deep space means running IP over deep space layer-2 links, a reliable transport over IP, applications protocols over that transport and applying proper routing, security and network management on that IP network. Reusing the whole IP stack in deep space enables the reuse of all protocols, tools and software currently used on Internet. However, as one might already argue, many components of the IP stack can not be used as is and therefore requires careful configuration and deployment considerations that are discussed in this document.¶
The keyword Delay-Tolerant Networking (DTN), also expanded to Delay and Disruption-Tolerant Networking, has been used to identify the problem space and given that up to now, the solution was based on the Bundle protocol, DTN was also associated with Bundle protocol. This document tries to solve the DTN problem using the Internet Protocol stack. Therefore, in this document, the DTN keyword is used to name the problem space, not the Bundle protocol solution.¶
Since Moon is a few light seconds away from Earth, it is possible to somewhat configure and run various IP based protocols and applications to make it "work". Mars with a much longer delay is more difficult. Therefore, this document uses Mars as the base example, knowing that if it works for Mars, a much harder problem, it could be replicated easily for Moon, or for other networks made with relays around a celestial body. This framework shall also work for longer delays, such as reaching Jupiter or the whole Solar System Internet(SSI), but it is not specifically discussed. This document uses "deep space" extensively in order to differentiate with "space" which often includes Earth orbiting communications, which is not discussed in this document, even if the definition of deep space per ITU does not include Moon, while this document is applicable to IP networks on the Moon.¶
It should also be noted that DTN and BP were also designed for non-space use cases. While this document focuses on the deep space use case, it shall work for the other use cases of BP, but these use cases are out of scope of this document.¶
Space missions are typically planned many years in advance and are long-lived, spanning over many years even decades. Spacecrafts are controlled from Earth and therefore should always be manageable from Earth. Given the remoteness and the difficulty to physically access the spacecraft, software upgrades and configuration changes are avoided whenever possible.¶
As with Bundle protocol, this framework proposes to use IP in deep space with a similar store-and-forward paradigm. Therefore, the IP layer has to deal with the fact that a destination may not be currently reachable and that IP packets should be stored for an unusual amount of time, such as minutes or hours or days, in the forwarding device waiting for a new link up opportunity. The transport layer should be able to work with long and variable delays, including intermittent communications. The application protocols and application themselves should be properly set to wait a longer time than on Internet to receive a response to a query. Finally, all network services such as routing, security, naming and network management should also be adapted in this new context. This document is structured around these layers.¶
The source of this document is located at https://github.com/marcblanchet/draft-deepspace-ip-architecture. Comments or changes are welcomed using a PR or an issue.¶
This subject should be discussed on the [email protected] mailing list.¶
Multiple countries are developing systems aimed for a sustained lunar presence combining manned and robotic missions, within several years. IP has been included in the stack for the International Deep Space Interoperability Standards[idsis], and the LunaNet Interoperability Specification[lnis]. There is a general intention to extend and reuse systems developed for lunar use to later Mars use¶
Separate space agencies and private companies are deploying lunar space stations, orbiters, landers, rovers, habitats, crewed mission elements, and other assets. Due to pervasive use and support of IP in modern computing systems, it also is naturally used onboard many space systems, and between co-located systems. As more-and-more IP-enabled assets become deployed in lunar vicinity, it will increasingly create opportunities to interconnect them. In fact, internetworking of lunar (and future Mars) systems is becoming essential, as plans call explicitly for cooperation between mission elements and communications/navigation system assets operated by different space agencies and/or private companies acting as service providers. There are expected to be several different lunar network service providers (LNSPs) offering different varieties of relayed and/or direct services. It may be expected that lunar IP networks should over time become united into larger aggregates, and even into a single interoperable network (as intended within the LunaNet conception).¶
Routing within and between dynamically connected network elements may be handled in various ways, as the practices develop and specific systems are deployed. There will likely be coordination between LNSPs and some methods to share and control pre-determined time-varying static routes (similar to PCE architecture), traffic volume expectations, security policies, etc. The early steps in lunar networking may not actually interconnect the global Internet directly with the lunar surface, though later on this may become reasonable.¶
The examplary network for this document is where deep space links are using IP over CCSDS space links[IPoverCCSDSSpaceLinks] and that on and around a celestial body, a connected IP network is established with local network infrastructure and services. Orbiters around the celestial body provides connectivity to and from Earth to the devices on the celestial body. As an example, a rover on the Mars surface is connected to a Mars surface IP network which receives intermittent connectivity from a few orbiters with an average of 6 hours per orbit. Some of those orbiters have circular orbits, other elleptical. The latter means that the overpass are not at a fixed frequency. The orbiters are connected to Earth ground station while they are in line of sight with Earth. Earth and Mars have variable distance from 4 to 20 minutes light seconds. That one way delay however change "slowly" as the planets are orbiting around the Sun.¶
When an orbiter has direct line of sight with Earth, it can receive from and transmit packets to Earth. During that window, it may have connectivity using another radio or laser to devices on the orbit or on the surface, but very often it does not. Therefore, during those periods where only one segment of the path is up, the orbiter must buffer the received packets until the next segment becomes available when it forwards the packets and flushes the buffer.¶
While the target scenarios for Moon or Mars are way more complex than what is deployed today, it is useful to know what has been deployed already. This section describes the Mars communications infrastructure as it relates to its layer 2¶
On Mars, there are a few devices such as rovers or helicopters that are in use. There are currently 5 active orbiters that provide communications relay services between Mars surface and Earth. On Earth, various antennas such as the Deep Space Network(DSN)[DSN] are used for communicating with Mars. The MAROS project at the Jet Propulsion Laboratory is a broker software enabling missions to enter data about the communications capabilities such as frequencies, bandwidth, window of communication time, ... so that rover missions can schedule the available communications windows for transmitting and receiving. Most orbiters are used and scheduled in MAROS. One of the Mars orbiters is Mars Reconnaissance Orbiter(MRO)[mro]. It was launched in 2005 and has a single 40Mhz processor but over 100G of solid state memory. As demonstrated by a study[marscommstudy] on Mars communication windows, the communication windows seem of having a constant frequency, but the reality shows that they are pretty variable, which means a very large range of resulting round-trip time (RTT) for communications from Earth to Mars and back. For example, within 3 months in 2024, the RTT varied from 30 minutes to 170 hours.¶
The Interagency Operations Advisory Group(IOAG)[ioag] has defined the communications architecture for Moon and Mars. On the celestial body surface, it is planned to use 3GPP and IEEE 802.* link layer protocols. The IP protocol suite is expected to be used over these link layers.¶
Deep space links typically use the Consultative Committee for Space Data Standards(CCSDS)[CCSDSWEB] standards for link layers, such as Telecommand(TC)[CCSDS_TC], Telemetry (TM)[CCSDS_TM], Advanced Orbiting Systems(AOS)[CCSDS_AOS], Proximity1(Prox1)[CCSDS_PROX1]or the Unified Space Data Link Protocol (USLP)[CCSDS_USLP]. CCSDS has defined a generic encapsulation mechanism for the payloads for all these link layer protocols which defines IP as an encapulated protocol[IPoverCCSDSSpaceLinks][SANAIPEHeaderRegistry]. Therefore, IP packets can be transported over any CCSDS link layers.¶
On celestial body orbits, IOAG has planned the use of CCSDS link layer protocols. However, as on Earth, it may be possible to use 6G-NTN technology around celestial bodies, such as Moon or Mars orbits. 6G-NTN technologies use IP as its layer3 technology.¶
IPv4 or IPv6 packets can be carried as is over long delays and disruptions, as IP itself has no notion of time. Originally, the Time To Live(TTL) field of IPv4 was defined based on time[STD5], but it has been effectively implemented as an hop count, which was renamed as "Hop Count" in IPv6[STD86]. Nothing needs to be changed to the IP protocol or its packet format.¶
For deep space applications, an IP packet may need to be stored temporarily over much longer periods than are typical for the Internet when the next hop is currently unreachable or undefined, which can happen due to orbital dynamics. This is commonly referred to as "store-and-forward" and bears no relationship to the same term when used in reference to switch architectures.¶
This store and forward mechanism may be implemented at layer 2 as is currently done by the Mars orbiters. In this case, the frames are stored, independent of payload type. In this case, IP packets are unaware of the store-and-forward and no changes are needed in the IP forwarding function. The L2 network is just behaving as a point-to-point link with a large and variable latency.¶
If an IP router has an interfaces on an intermittent link, then a queueing discipline should be used to store packets. This might be implemented as a deep queue with active queue management(AQM)[RFC7567]. When the link to the next hop is up again, maybe minutes or hours later, the stored packets can then be transmitted.¶
This store-and-forward mechanism requires proper provisioning of storage at each forwarding interface for the target deployment and usage. Mechanisms such as ECN (Section 6.4) can be used to signal storage congestion can be used to request that end nodes throttle their transmissions. If the appropriate queue is full, then the interface MAY drop packets and MAY send ICMP error messages to the source, so that the transport protocol can recover by resending the dropped packets. The choice of which packets to drop during a queue congestion event depends on the queuing policies configured in the node, which may be a function of diffserv/traffic class, source or destination IP addresses, flow label, or other parameters. An example is described in [I-D.blanchet-tvr-forwarding].¶
Deep space links are point-to-point links and bandwidth in space is very valuable, so header compression is very useful. Static Context Header Compression (SCHC) [I-D.ietf-schc-architecture] is a header compression technique that relies on rules in a static context, and is therefore more efficient for deep space. SCHC should be considered on a deep space point-to-point link or a string of L2 links.¶
Existing routing protocols have a requirement for proof of liveness between protocol partners that is implemented through the periodic exchange of packets between partners. This is impractical on long-delay or intermittent links, so a PCE [RFC4655] based approach seems appropriate for those domains possibly supplemented by contact plan schedules [I-D.blanchet-tvr-contactplan] [I-D.ietf-tvr-schedule-yang]. Interconnection between domains can still be done with BGP [RFC4271], but long-delay or intermittent links should be avoided. Domains straddling such links will need to provide proxy advertisements for prefixes reachable across such links.¶
Optimal routing for domains with intermittent links is out of scope for this document.¶
On the surface of celestial bodies and in proximal orbit, traditional protocols are applicable and should be used (e.g., [I-D.li-arch-sat]).¶
The IP address space is a hierarchical namespace where ranges of addresses are encoded as "prefixes". Individual domains advertise a set of prefixes to the broader Internet and assign these addresses internally. Prefixes may be aggregated into less-specific prefixes, which makes the routing subsystem more efficient by decreasing overhead.¶
Space networks provide a unique opportunity to provide extremely efficient routing by assigning a unique prefix or block of addresses per celestial body and its proximal orbits. Management of the IP address space is currently documented in [RFC7020], but this only covers continental regions and does not provide for addressing for space.¶
Address space for outer space should be managed by a single Regional Internet Registry (RIR) and blocks of address space should be allocated for each celestial body of interest. Space service providers should use prefixes assigned by this RIR.¶
UDP[RFC768] has no notion of time, therefore can be used as is in deep space. Protocols using UDP transport can therefore be used in space as is, if they do not rely on time or if they can be configured with timeouts appropriate in deep space.¶
QUIC[RFC9000] like most IP transports implements congestion control mechanisms, which, based on various metrics such as calculated delays or packet loss, pace the rate of sending packets at the source node to decrease the perceived congestion in the network. QUIC supports many new features suitable and useful in deep space such as 1 RTT for connection establishment and security, mobility, 0RTT, streams, user-space, ....¶
Current implementations of QUIC typically set various transport configuration parameters suitable for the Internet environment, with RTT in the hundreds of milliseconds and relatively always connected network. Therefore, QUIC stacks using default configurations will not work in deep space. However, studies and simulations[quic-sim] showed that with proper transport configuration parameters, QUIC stacks support delays and disruptions in deep space. [I-D.many-deepspace-quic-profile] describes how to properly configure a QUIC stack for deep space application, where the QUIC transport is unaware of disruptions. If the transport is aware of the disruptions, then further optimizations may be done.¶
The ability to have multiple streams and applications within a single QUIC connection is valuable and useful for deep space. A ground station may setup the initial QUIC connection with a spacecraft and then carry all needed applications and streams over that same connection for the whole duration of the mission.¶
Session key and certificate lifetime together with certificate validation and trust chain anchors need to be carefully configured and handled.¶
QUIC proxies[I-D.ietf-masque-quic-proxy] can be used at space edge to isolate, apply policies or to optimize trafic at ingress/egress to a celestial body network.¶
Other transports such as TCP[RFC9293], SCTP[RFC9260], DCCP[RFC4340] and others were not investigated for their suitability in space.¶
Explicit Congestion Notification(ECN)[RFC3168] enables a network forwarder to signal to the sources to pace down in the context of congestion. In deep space where forwarders will be buffering packets, the actual congestion is when the storage is approaching its full capacity. ECN enables the forwarders to signal that issue. Given delays and disruptions which means the reception by the source may take longer time, the forwarders should be pro-active and preempt based on the types of links and actual rate of storage usage. IP clients with QUIC should process the received ECN signals appropriately.¶
HTTP by itself has no notion of time. An HTTP request and response may take minutes or hours to be completed. However, current infrastructure and software on Internet have various time-related configurations that will not work as is in the deep space context.¶
HTTP headers containing time, such as Cache-Control and Expires [RFC9111], should not be used or if used be set to large enough values to cover the longest delay so that expiration does not happen before the actual data arrives at the destination. As with any HTTP application and content on Internet, these headers should be set properly based on the deployment use case, which is ever more important for deep space. Similarly, when continuous content transfer is used, as with 100-Continue [RFC9110], proper values for headers should be set.¶
HTTP clients and servers typically have default timeouts that shall be modified. For example, curl [curl] has the "-m" option for this use case. Similarly, HTTP server implementations have various timeouts configuration variables to be set properly. Testing with HTTP client Curl and HTTP server nginx and an introduced network delay of 20 minutes showed that HTTP communications work just fine with very basic configuration changes.¶
HTTP applications themselves must be developed using an asynchronous pattern and if they have timeouts, they should be adjusted appropriately.¶
Internet Web sites are designed with the assumption of hundred of milliseconds delay and relatively always connected, where pages contain multiple queries to further get resources, media, queries to web services and downloading additional code and frameworks. This could work in theory in this context of space, but it will not be optimal, as multiple queries will be generated and therefore taking multiple RTT before the whole page is received complete. This issue can be mitigated by using various techniques such as Web Assembly [wasm] or pre-caching. Moreover, tt could be possible to have very basic HTML pages with zero or very few href and no media content unless locally cached to be used. An example would be a rover on Mars presenting an HTTP server with a base and bare HTML page to offer basic info on its status (maybe all in text) and some additional detailed pages, most likely also in base html text. However, it is foreseen that most applications based on QUIC-HTTP transport in deep space would be using REST or similar asynchronous patterns and not typical web browsing.¶
Caching should be used extensively on celestial bodies networks to maximize local fetching. Preemptive caching by pre-populating caches with data that shall be used locally on the celestial body network shall be done as much as possible to provide better response time on the local celestial body network.¶
QPACK [RFC9204] should be considered for higher bandwidth efficiency.¶
At a small scale, one can use IP addresses directly or can use static names to IP address mappings such as /etc/hosts. However, this does not provide easy dynamic updates, scaling by hierarchy, service discovery, authentication of records, ... Therefore, the Domain Name System (DNS) shall be considered early on in the space deployment. However, naming hierarchy and infrastructure has to be carefully designed to avoid name resolution over deep space links, given that answers may come after minutes or hours. There are clear advantages of having the space name hierarchy anchored to the current Internet root, as it enables DNSSEC using the same security infrastructure currently used and deployed. Using the same root also does not require new policies. A new TLD or a new root is way more complicated and does not bring any significant value compared to using the current domain tree.¶
Care must be taken to manage key lifecycles and resource record lifetimes. [I-D.many-dnsop-dns-isolated-networks] discusses the various methods and naming hierarchy that can be used in space.¶
NETCONF[RFC6241] and RESTCONF[RFC8040] shall be used with proper configuration values to avoid timeouts and appropriate transport. NETCONF over QUIC transport[I-D.ietf-netconf-over-quic] or RESTCONF over HTTP over QUIC transport shall be configured with appropriate QUIC transport parameters as discussed in Section 6.2.¶
While being declared historic in IETF, SNMP[RFC1157] runs over UDP and have no notion of time. Therefore, with proper configuration of client timeout, it can be used as is to manage nodes and services in deep space.¶
This memo includes no request to IANA.¶
Using the current IP protocol stack in deep space inherit all the work on privacy, cryptography, key management, firewalls and relative high scrutiny of protocols that is deployed on Internet. As an example, TLS has been way more scrutinized than almost any other secure transport protocol. Moreover, given that no changes are made in the protocols, this architecture does not bring new security issues. Obviously, the deep space security requirements are different than on Earth Internet, but nothing has been found to prevent the conformance of the IP protocol stack to those requirements.¶
As it is currently planned, the deep space network shall be isolated from the current Internet by "air gap", to disable any direct communications from Internet to deep space. Moreover, destination IP prefixes filtering shall be used to restrict the traffic to only the relevant one for each link. Note that this shall also be implemented in the routing control plane, but additional security might be appropriate to further protect the deep space links.¶
Each celestial network edge device shall have firewall rules to disable non-useful trafic to go through deep space links. If communications from Mars may only occur to Earth, but not Moon, then appropriate filtering based on destination IP prefixes shall be used. Given the air gap on Earth for Internet, there shall be no default route advertised in space that could for example point to Earth Internet.¶
This work started by reassessing the use of the whole IP stack in the context of deep space. Soon, QUIC was identified as the key technology for this endeavour. Christian Huitema was very helpful in not only confirming the ability to use QUIC but also took the time and effort to test and modify its picoquic stack[picoquic] to confirm the initial hypothesis[picoquic-poc]. Its involvement and confirmation are the key for the launch of this work. Then, Martin Thompson has been also kind to take time to answer initial questions on QUIC, further confirming the possibility of using QUIC for deep space. Since then, many individuals have provided significant comments and perspectives on this subject.¶
This document and its underlying work has been reviewed and discussed by many, who have provided valuable feedback and comments, including disagreements, and made an overall more solid document. These people are, in no specific order: TBD.¶