Preface

With the explosion of the Internet, the TCP/UDP/IP protocol suite has become the underlying framework upon which all Ethernet communications are built. Their success attests to the generality and power of these protocols. However, these transport-level protocols are too low level to be used directly by any but the simplest applications. Consequently, higher-level protocols such as HTTP, FTP, DHCP, DCE, RTP, DCOM, and CORBA have emerged. Each of these protocols provides well-tuned functionality for specific purposes or application domains, but none of them is suitable for real-time distributed applications, where developers need some methods to control data traffic on bus. For example, none offers deterministic communications, time-aware notifications, heartbeats, transparent hot-swap substitution, or quality of service control. Modification of publish-subscribe protocol (Real-Time Publish-Subscribe) adds parameters, that offers application developers an easy way to manage communication on bus with different deadline requirements.

Communication architectures

Distributed application developers have several choices for writing communication. There are three main communication architectures (Point-to-Point, Client-Server, Publish-Subscribe).

Client-Server (CS) : data exchange is done between group of client nodes and one server node. It works well when all data are located on central server (database applications). CS architecture is inefficient when data are produced by many nodes and are consumed by multiple nodes (two transactions are needed), which is typical for distributed application.

Publish-Subscribe (PS) : nodes "subscribe" to data they need and "publish" data they produce. Messages are transferred directly between communication nodes. Data are beeing sent without exact destination address, they are received by all nodes and each node decides itself (with respect to topic identification) whether it is interested in this message or not. PS architecture not suited for request/response traffic, such as file transfers.

Real-Time Publish-Subscribe (RTPS)

Real-time applications require more functionality then the one provided by traditional publish-subscribe architectures. RTPS adds publication and subscription timing parameters and properties so the developer can control different types of data flows and achieve their application's performance and reliability goals.

Figure 8-3 illustrates how RTPS uses the deadline, minimum separation, strength and persistence properties to provide network communication for real-time applications and simplify system design. Features shown on the figure are:

Subscription Issue data flow: If a new issue does not arrive by the deadline, the application is notified.

Redundant Issue hot-swap: During persistance period ORTE stack accepts issues from publiccations with equal or higher strength. After persistance, it accepts the first issue, regardless of the publication's strength.

ORTE architecture

The OCERA Real-Time Ethernet (ORTE) will be open source implementation of RTPS communication protocol. This protocol is being to submit to IETF as an informational RFC and has been adopted by the IDA group. Figure 8-4 shows the network stack layers. Non Real-Time applications, which are using standard protocols such as HTTP, FTP, DCOM etc., are running on top of standard TCP or UDP stack. ORTE is new application layer protocol, which is build on top of standard UDP stack. Since there are many TCP/IP stack implementations under many operating systems and ORTE protocol does not have any other special HW/SW requirements, it should be easily ported to many HW/SW target platforms. It doesn't use or require TCP, so it retains control of timing and reliability.

The ORTE is composed from three main components, as shown on Figure 8-5:

Database: stores parameters describing both local as well as remote node's objects.

Processes: perform message processing, serialization/deserialization and communication between objects.

API: application interface.

Database implementation

Status of all objects is stored in ORTE database. The database is set data structures containing parameters of all known objects. There are not only data describing local node's objects, but also description of all known remote nodes' objects. Since access to this database should be granted to both ORTE stack processes as well as to applications, there should be implemented some access control algorythm allowing concurrent access. Applications use an API call to access the database.

Database contains two types of information. The first type is description of all known objects without any specification of relationship among them. This information is stored using a binary tree structure with object's GUID used as the key. The second type of information describes relationship among objects. Such relationship is for example association between Manager and ManagedApplications or between publishers and subscribers. This information is stored as a (double) linked list of GUID of objects which belong to certain owner. As an example, such owner is a publisher, the linked list contains list of GUIDs of its subscribers. The communication objects can be represented as the branch of a tree. Each of the nodes in the branch and its descendants represents an element of the complex data-structure.

• there is local manager Manager 1

• Manager O2 is local copy of parameters describing Manager 2, which is located on remote node Node 2

• there is local managed application ManagedApp 1

• there is local copy of remote application ManagedApp O2

• there are publishers Publisher 1 and Publisher 2

• there is remote subscriber Subscriber O1 which is subscribed to data published by Publisher 1

• there is currently no subscription to Publisher 2

Note: letter O, which is placed before numerical index of an object means, that this database object represents object which belongs to other node than local. For example node 1's object ManagedApp O2 is local representation of node 2's object ManagedApp 2.

ORTE processes implementation

The ORTE system spawns several processes during its lifetime as shown on Figure 8-7. Generaly there are two groups of processes. The first group represents processes involved in real-time application's data exchange between local and remote nodes. These processes are responsible for publishing of local application's data if there are any remote subscribers and also for receiving of remote application's data if there are any local subscribers. Another group are processes involved in network management.

Process Subscription which is responsible for processing of all local subscriptions, i.e. it receives all application data from all remote publishers to which local applications want to be subscribed. It performs whole processing of received data including storing of all data changes into database and, if requested by application, also calling an application's callback functions. This process can listen on more UDP ports where each port belongs to one local subscription, or on a single UDP port, which is common to all subscriptions. In any case, this process will never listen on default NDDS port 7400.

Processes Publication1, Publication2, ..., PublicationN are responsible for proper publication of local data to all its subscribers across the network. Each process is associated with queue Pqueue1, Pqueue2,... PqueueN. Data changes are stored (using proper API call) into certain queue. Process which is associated with this queue reads its content and sends ISSUE message to all subscribers and waits for their ACK. Each process in this group use its own UDP port, they will never use default NDDS port 7400.

Process NodeManagement is responsible for whole management traffic. It is the only process which uses default NDDS port. Since management traffic does not have real-time requirements, it is designed as single process which processes both incomming as well as outgoing requests. During evaluating phase it may be split to two separate processes, one for incomming requests and second for outgoing requests.

API implementation

First implementation will be designed like one application running in user space using standard Linux 2.4 kernel. Since the main purpose of this version will be to test this implementation against another commercially available implementation, there will be no any standard API provided. The real-time capabilities will not be focused during this phase too. The next version will be written as a Linux kernel module. Interface between this module and an application will use standard ioctl function and read/write operation. The function will be divided into three categories:

• Administration - create and destroy database (InitDB, DestDB, SetParamDB, GetParamDB,...)

• Publish - create and destroy publishers, sending data (CreatePub, DestroyPub, SndData,...)

• Subscribe - create and destroy subscribers, receiving data (CreateSub, DestroySub, RecDataPoll, @RecDataCallBack, ...)