Performance Evalution of an Erlang based Implementation of DDS

Performance Evalution of an Erlang based Implementation of DDS

Wafa Helali, Khaled Barbaria and Belhassen

Université de Tunis El Manar,

Faculté des Sciences de Tunis, Laboratoire LIP2.

2092, Tunis, Tunisie

AbstractWhile middleware technologies solve many design and development issues related to distribution, they usually introduce performance and scalability overheads. DDS (Data Distribution Service) is a middleware standard designed to support many requirements of quality of service. The way this standard is interpreted by the developers of middleware solutions impacts the ability of the middleware to efficiently develop distributed applications with stringent requirements of performance and scalability. In this paper, we compare two solutions to support the DDS standard. The first is based on the CORBA middleware, while the second uses the built-in message passing mechanism that comes with Erlang. While the CORBA based solution is easier to design and develop, the Erlang based solution generally presents better performances and scalability aspects.

KeywordsMiddleware, DDS, Erlang, Message Passing Model.

1. INTRODUCTION .

   Building distributed applications is a complex activity specially when we talk about real-time large scale distributed systems where extreme time constraints and non-functional requirements must be met (performance, extensibility, quality of service parameters, robustness, scalability). Middleware facilitates the integration of such systems. Those based on the publish/subscribe paradigm (pub/sub) are the most adopted in this applications domain especially in the industrial world. But they are still far to support real-time needs. This gap is due to the limited or non-existent support of Quality of Service (QoS) parameters. In order to fill it, the OMG proposed a specification of Data Distribution Service for real-time distributed applications called DDS [1] based on existing pub/sub implementations. DDS provides a data exchange model in a distributed context and offers a simple strong mechanism to manage quality of service policies. The middleware facilitates the task for applications developers, but the task became very difficult for middleware developers. For this reason the choice of the programming language for an implementation is very important. Programming language with higher level of abstraction like Erlang [2] and scala [3] provides a simple coding and improves the speed of development. In other hand, programming language have long impacted the development of distributed application. Numerous DDS implementations have made using general language like Java or C++ and developed upon a CORBA

   architecture like OpenDDS1. But the choice of a programming language usually depends on application needs to be developed. Indeed, we must choose the best language suited to such applications, based on its libraries and its properties, to improve system performance.

   In this paper, we present our choice of Erlang to implement the data distribution service. It is a programming language that was designed for telecommunications problems. But by detailing its features (concurrency, performance, messaging, robustness), we notice that is suitable with real- time distributed applications. Based on Erlang Message Passing Mechanism to distribute data between the communicated DDS entities demonstrates that it makes highly reliable and scalable real

   time systems.

   The rest of the paper is structured as follows. In the next two sections we briefly review the basis of our work in order to put it into context. In the section 4, we present the adopted architecture used to support the data distribution service in Erlang (eDDS) and next we present two different communication models that can be made to distribute data in eDDS. Some experiences shown in section 5 demonstrate that communication based on Message passing model is more reliable then CORBA based communication. Section 6 is devoted to present some related work. Our concluding remarks

   are presented in section 7.

2. DDS OVERVIEW

   Data Distribution Service [1] is a standard invented by the OMG. It provides a data exchange infrastructure based on the pub/sub paradigm for supporting real-time distributed systems (aerospace and defense, distributed simulation, industrial automation). The unit of transmission in DDS is the data that will be circulate through a global data space (GDS).

   Participants in this GDS specify only the data that will be published or received neither how nor when it should be done. DDS specification offers two levels of interfaces:

   • DCPS (Data-Centric Publish-Subscribe), a low-level layer that handles the efficient delivery of data to the proper recipients.

   • DLRL (Dated Premises Reconstruction Layer) an optional high-level API that allows for a simple integration of the service into the application layer.

     Fig.1 shows the overall DCPS model, which consists of the following entities: DomainParticipant, Publisher, DataWriter, Subscriber, DataReader and Topic.

     Fig. 1. DDS Entities

   • DomainParticipant acts as a container for all other entities. It is responsible for creating Publishers, Subscribers and Topics. It represents the local membership of the application in a domain that isolates application running on the same physical computer from each other.

   • Domain is a construct which binds all the applications capable to communicate with each others. Applications that belong in different domains cannot communicate.

   • Publisher is the object responsible for the dissemination of the data. It can contain several different DataWriters. It includes several DataWrietrs.

   • DataWriter is the object by with an application can write data on the global data space. It is associated with a specific data type.

   • A Subscriber is the object responsible for receiving the published data from the global data space according to its subscriptions. It contains several different Datareaders.

   • Each DataReader is associated with a specific data type. It is responsible for making the data available to the application.

   • Topic is the entity that represents the basic connection point between publishers and subscribers. Each Topic corresponds to a specific data type that defines the data that can be communicated when publishing or subscribing to this Topic.

   • Each DCPS entity can apply a subset of available QoS policies (debit, latency, lasted validity) which allow the designer to control some aspect of the behavior of the DDS Service [6].

   • DDS supports two notification mechanisms:

     • Listener-based: is a generic asynchronous notification mechanism that notify the application of

       relevant events, such as QoS violation. Each entity has its own set of listeners;

     • Wait-based: is a synchronous notification mechanism that allows the application to wait for a condition to occur.

3. THE ERLANG PROGRAMMING LANGAGE

   1. Characteristics of Erlang

      Erlang was originally designed at Ericsson Computer Science Laboratory to solve telecommunication problems. Erlang is both a language and runtime environment. It has many libraries grouped under the name of OTP (Open Telecom Platform) to develop highly and reliable systems.

      It is a declarative dynamically typed language that was developed as an extension of Prolog and influenced by functional languages like ML and Miranda and concurrent languages like Ada and Modula. It possesses qualities tha simplify program design ensures comfort development of applications. There are no assignments or mutable data structures. It uses pattern matching for variable binding and function selection.

      Concurrence in Erlang is the key factor of its success. Unlike other languages like C or Java that use threads of the operating system, Erlang manages processes created by the virtual machine (VM). Each Erlang process has its proper memory space and runs independently of other processes. Processes communicate with each other via asynchronous message passing mechanism where response times in the order of milliseconds. Memory management in Erlang is based on the garbage collector concept, Memory is allocated automatically when required, and deallocated when no longer used. Erlang provides a mechanism for handling errors and exceptions that can be used to program fault tolerant robust applications based on the concept of supervision [5]. All these characteristics ensure that Erlang was successfully used in major industrial projects such as SimpleDB and the http server yaws, as well as Facebook.

   2. Native Erlang communication features

   Erlang processes are running in isolation of each other. The only possible interaction between them is by sending messages. A new process can be created by a built in function (BIF) named "spawn(Mod,Fun,ArgList)" that will return an identifier of the created process. This process has as entry point a function F defined in module M by using the arguments Args. The process identifier (Pid) can be used to send a message to the process using the following syntax : "Pid ! Message". To receive messages, the process uses the primitive "receive" which is a pattern matching operation. The process evaluating this primitive is suspended until a message which matches one of the patterns (pattern1, pattern2,…) is received or until a timeout expiration. Processes can be named using the BIF "register(Name,Pid)". This name must be an atom and may be used as an alias for Pid when sending a message using the syntax: "Name ! Message". Next, we present a simple example that demonstrates how to create processes and send messages between them:

   -module (echo).

   -export([loop/0]). loop()->

   receive

   {Form,Message}->

   io:format("~p from ~p ~n",[Message,Form]), loop()

   end.

   To create a process that will execute the function loop in module echo we use this syntax:

   > Pid= spawn(echo,loop,[]).

   <0.35.0>

   The message "{self(),hello}" will be sent to the process (Pid). "self()" is a BIF which returns the process identifier of the current process.

   >Pid!{self(),"hello"}. "hello" from <0.31.0>

4. EDDS: AN ERLANG SUPPORT OF DDS MIDDLEWARE

   1. The eDDS API

      In this section, we explain the use and show the anatomy

      of

      our Erlang based API for DDS. This application programming interface named eDDS is an Erlang module (edds.erl) that contains all the basic functions necessary for developing and deploying an Erlang distributed application based on the DDS middleware. Data distribution with eDDS is made in 3 stages:

      .

      1. Initialisation of the GDS: It is the process which serves to initialize the system and to create the DomainParticipantFactory. This entity is responsible for creating DomainParticipants and configuring its initial quality of services parameters. Each Global Data Space (GDS) has a single DomainParticipantFactory. An identifier of the factory can be stored in a file shared by the different nodes of the distributed applications. To access to the GDS, a node should retrieve the identifier of the factory from this file and create its local DomainParticipant. edds:create_domainparticipantfactory(Path).

      2. Creation of DDS entities:

         Dp = edds:create_domainparticipant

         (Dpf,Dom,ParQos,Listener,Mask) Top=edds:create_topic (Dp,Topic_name,Type,TopQos,Listener,Mask)

         Pub = edds:create_Publisher

         (Dp,PubQos,Listener,Mask).

         Sub = edds:create_Subscriber

         (Dp,SubQos,Listener,Mask).

         Writer = edds:create_datawriter

         (Pub,Dp,Top,WriterQos,Listener,Mask).

         Reader = edds:create_datareader

         (Pub,Dp,Top,ReaderQos,Listener,Mask).

         DomainParticipant is the entity responsible of creating

         Topics, Publishers and Subscribers. A DomainParticipant can be created using the function "create_domainparticipant".This method has as parameters the DomainParticipantFactory (Dpf), Domain (Dom), the quality of service parameters, Listener and Mask. This function will check first if a DomainParticipant is already created for this Domain. If it is the case, it returns its identifier; otherwise it must create a new

         DomainParticipant instance.

         To create a Topic, we use the function "create_topic". The parameters are the DomainParticipant (DP), the name of the Topic (Topic_name), the name of the used data type (Type), the quality of service parameters, Listener and Mask. Its return value is an identifier of the created Topic. Theres a common feature among the three entities previously presented: The DomainParticipantFactory should be unique to the global data space. The DomainParticipant and the Topic should be uniques within the Domain. If an application wants to create a topic, the system must check if this Topic is not yet created. If this is the case it must return to the application an instance of this Topic. The required behavior is close to the singleton design pattern that restricts the instantiation of a class to one object.

         Publisher is the entity responsible of creating DataWriters and configuring its initials quality of service parameters. To create a Publisher we use the function "create_publisher". The used parameters are the DomainParticipant (DP), the quality of service parameters, Listener and Mask. Its return value is the identifier of the created Publisher. This identifier will be sent to the DomainParticipant to save it as one of its publishers by using the "add_newPublisher" function. This is the same strategy used to create a Subscriber and save it on the DomainParticipant using the function "add _newSubscriber".

         DataWriter is the entity used by the application to submit data for publication. To create a DataWriter we use this method "create_datawriter". Once the DataWriter is created, its identifier will be saved at the DomainParticipant by invoking the add publication function. In turn, the DomainParticipant sends the list of DataReaders already enrolled at the same Topic to the created DataWriter by calling the set datareader function. The DataReader is the entity is used to read the samples related to the Topic to which it is associated. To create a DataReader we use this function "create_datareader". When the DataReader is created, its identifier will be saved at the DomainParticipant by invoking the "add_subscription" function.

      3. Data Dissemination: Once the participants are created, the service is ready to publish data. To send data, the application calls thewrite operation of the DataWriter. Data will be propagated in form of sequences of bits to the DataReaders interested by the Topic if their QoS specification match. Data will arrive on the DataReader as an Erlang tuple, as a result of read or take calls. Next is an example of a Sensor data type presented like IDL (Interface Definition Language) struct.

         % Data type: IDL struct struct Sensor{

         l ong id; float temp;

         float press; };

         % Data dissemination edds:write

         (Writer,#Sensor{id=1,temp=10,press=10}).

   2. eDDS Architecture Details

      The API proposed in the previous section is independent of the effective services used to manage the life cycle of the DDS entities and transmit data. Like the most other DDS implementations and in order to made our approach more gneric, we make the choice to use CORBA to manage DDS entities. The official Erlang distribution includes a CORBA implementation called Orber2 as part of the platform OTP. The server shown in Fig.2 acts as an intermediary that matches compatible publishers and subscribers (wrt Topic and QoS). When a client requests a subscription for a given Topic, the server locates this Topic and informs all existing DataWriter for the location of the new DataReader. eDDS entities are created as CORBA objects. These entities are referenced by their unique Interpretable Object Reference (IOR). Each DDS entity which acts as factory of other entities, must contain in its state all information about the entities that it was responsible for their creation. For example, the DomainParticipant state contains lists of the publishers, subscribers and topics which was created by it, as well as their default Qos parameters, etc.

      Fig. 2. The Architecture of eDDS

      DDS entities are deployed regardless of their locations. As a possible deployment scheme (Fig.3), we can define a DomainParticipantFactory, DomainParticipant and Topic in the same node (the server). The other entities used to send and receive data can be deployed on different other nodes according to the requirements of the distributed application. To improve the dissemination of data is done directly from DataWriters to DataReaders. The DDS standard defines DataReaders and DataWriters for each IDL type which is necessary with procedural and object oriented languages like C, C++, java and Ada. The data dissemination model that we adopted allows us to reuse the same entities independently of the type of data being transmitted.

      Fig. 3. The Deployment of eDDS entities

   3. Data dissemination with eDDS

      On eDDS, we have used two different communication models. The first one is a classic model based on the CORBA calls and the second is based on the Erlang messahe passing mechanism.

      1. EDDS_C: Communication model based on CORBA call: In this eDDS communication model, processing data is made by using CORBA calls (Fig.4). Indeed, when creating a DataReader, its reference (IOR) will be registered on the DomainParticipant. Subsequently, the DomainParticipant is responsible for notifying all concerned DataWriters. Each one has a list that contains references of all interested DataReaders (which have the same Topic of the concerned DataWriter). When calling the "write" function of the DataWriter, a call of the "read" function of each DataReader registered in its list is established: data is transferred as a parameter of the "read" method.

         Fig. 4. EDDS_C communication

      2. EDDS_E: Communication model based on Erlang Message passing mechanism: As depicted in Fig.5, the communication model of eDDS is based on the Erlang message passing mechanism. Datawriter directly sends new data to the dataReader using the native communication infrastructure provided by the Erlang virtual machine. We associate to each DataReader a new process which waits for new Erlang messages (by executing the BIF utils:listen). The identifier of this process is registered using a remote call on the add subscription method available in the DomainParticipant interface. The DomainParticipant is responsible of giving each DataWriter a list that contains the identifiers of each DataReader interested by the given Topic and provided a compatible QoS specification. This is done using the elegant list comprehension mechanism provided by the Erlang programming language.

   Fig. 5. EDDS_E communication

5. RESULTS AND DISCUSSION

   Our Erlang-based support of the DDS standard has two major advantages. First, it provides an API that allows to easily develop and deploy DDS-based applications using the Erlang language. Consequently, it is possible to combine the functional approach and the DDS distribution technology. Moreover, it will be shown that the use of Erlang to develop eDDS internals provides excellent performance and scalability

   results. We made two benchmarks to evaluate the latency and the throughput of an eDDS based application, knowing that we used only one machine to execute all the different process.

   1. Experiment 1: Performance

      This experience investigates the performances by measuring the variation of latency and throughput of the "write" operation by increasing the volume of the data to be sent. Latency presented in Fig.6 is measured as the time elapsed from invocation of the "write" operation of the DataWriter to the arrival of data to the DataReader. This experience shows that the difference between the two communication model is very remarkable: eDDS_E is almost seven times faster than eDDS_C. Fig.7 shows the average of the throughput.

      Fig. 6. eDDS_E vs eDDS_C: Latency

      Fig. 7. eDDS_E vs eDDS_C: Throughput

   2. Experiment 2: Scalability

      Experience2.a: One publisher to many subscribers: In this experience, we examined the impact of the increase number of subscribers. We studied the variation of latency of the "write" operations by increasing the volume of data to be sent from one publishers to many subscribers. Fig.8 shows the average latency of the "write" operation for 10 successive calls towards 1, 50 and 100 subscribers. eDDS_E for 100 subscribers is still faster than eDDS_C for 50 subscribers.

      Fig. 8. The impact of varying subscribers number on the average latency

      Experience2.b: Many publishers to one subscriber: In this experience, we examined the impact of increasing the number of subscribers on the communication latency. We studied the variation of latency of the "write" operations by increasing the volume of data to be sent from many publishers to one subscriber. Fig.9 shows the average latency of the "write" operation for 10 successive calls from 1, 50 and 100 publishers to one subscriber. The results show that the difference between the two types of communication is highly significant: the latency time of eDDS_E for the case of 100 publishers is still lower than the latency time of eDDS_C in the case of one publisher.

      Fig. 9. The impact of varying publishers number on the average latency

      Experience2.c: Many publishers to many subscribers: We studied in this experience the variation of latency of the "write" operations by increasing the volume of data to be sent from many publishers to many subscribers. Fig.10 shows the average latency of the operation "write" for 10 successive calls form 100 Publishers towards 100 Subscribers. eDDS_E is faster than eDDS_C.

      Fig. 10. The average latency in the case of 100 publishers and 100 subscribers

      We can conclude that eDDS_E has very interesting scalability properties: the curves are almost linear. Although this is due to the natural scalability properties of the pub/sub model; it shows that our design choices and our implementation strategy do not impede these properties. In other hand, eDDS_E may be seven times more powerful than eDDS_C. This high difference is due to the encoding of CDR data of CORBA which consumes more bandwidth than a simple Erlang message.

6. RELATED WORK

   There are several commercial and open source implementations compliant with the DDS standard. The majority of them are made using non-functional paradigm. Here, we present only two approach that related to our work. The first one [15] is developed by the OCI in 2011 that present a C Node, an Erlang node implemented in C which acts as a gateway between Erlang and DDS. This allows any Erlang node to participate in the system based on OpenDDS middleware (a C implementation of DDS) as a subscriber by receiving data or as a publisher by sending data to the system. But this approach is far to be considered as an Erlang DDS implementation.

   However, the first implementation using a functional language is appeared in 2012: Escalier [16] presents a Scala API for DDS. In ontrast with our eDDS implementation that completely based on Erlang, this Scala API is based on an open-source implementation of OpenSPLICE.

   Using functional programming languages are more suitable for distributed real-time systems based on concepts and techniques such as message passing mechanism of Erlang to support concurrence, distribution, real time and large scale requirements. Many new research works on distributed computing domain show a great interest to choose using the Message Passing Model on the communication infrastructure of the distributed systems. Next, there are some of these works.

   • Reference [8] talks about the influence of choosing programming language on the development of distributed systems. Instead of general language like C++ or Java. The development of future distributed applications will be made using alternative languages like Erlang or scala according on its characteristics on concurrency, scalability, messaging and robustness. Reference [4] also worked on the scalable

     performance of the scala Message-Passing Concurrency.

   • Another work improved this idea is presented in [9]: A cloud Haskell language that provides a Message Passing communication Model (MPI) [13], inspired by Erlang for developing distributed computing applications. Like our work on eDDS_E, any data that need to be communicated from one process to another are sent explicitly in messages.

   • A Distributed Rule Engine Based on Message- Passing Model is presented in [10]: a new approach to address the issue of dealing with many rules and facts efficiently.

7. CONCLUSION

In this paper we make an Erlang support of the DDS middleware (eDDS). In a first step , we have adopted the CORBA call communication to distribute data between nodes (eDDS_C). But stopping here, we do not profit completely from famous properties of Erlang in distribution context. For this reason, we have created another communication model of eDDS based on Message Passing Mechanism of Erlang (eDDS_E). Performance and scalability evaluation of these two models shows that the difference between them is highly significant. eDDS_E may be seven times more powerful than eDDS_C. This high difference is due to the encoding of CDR data which consumes more bandwidth than a simple Erlang message.

DDS and Erlang are very complementary to each other. By profiting on performance and scalability from the Erlang message passing mechanism like communication model in DDS, adding a DDS support of QoS to the Erlang OTP is an important advantage to this language. eDDS presents an implementation of the minimum profile of DDS. As a future work, we can improve this implementation by adding the RTPS (Real Time Publish Subscribe) protocol of DDS to support the interoperability with other DDS implementations.

REFERENCES

1. Object Management Group, Data distribution service for real-time systems specification,version 1.2, 2007.

2. Ericsson Computer Science Laboratory, Erlang programming language, http://www.erlang.org, 1980.

3. Odersky, M., Altherr, P., Cremet, V., Emir, B., Micheloud, S., Mihaylov, N., … & Zenger, M. (2004). The Scala language specification.

4. BATE, A, Scalable performance for scala message-passing concurrency, Communicating Process Architectures,pages 113131, 2013.

5. Cesarini, F. and Thompson, S, Erlang programming, OReilly, 2009.

6. Corsaro, A., Querzoni, L., Scipioni, S., Piergiovanni, S. T.,and Virgillito, Quality of service in publish/subscribe middleware,Chapter in Global Data Management, 2006.

7. Cosmin Arad, J. D. and Haridi, S, Message-passing concurrency for scalable, stateful, reconfigurable middleware, 2012.

8. Debasish Ghosh, Justin Sheehy, K. K. T. and Vinoski, S, Programmin language impact on the development of distributed systems, J Internet Serv Appl, 2012.

9. Jeff Epstein, A. P. B. and Peyton-Jones, S, Towards Haskell in the cloud, Haskell 11 Proceedings of the 4th ACM symposium on Haskell, pages 18129, 2011.

10. Jeff Epstein, A. P. B. and Peyton-Jones, S, A Distributed rule engine based on message-passing model to deal with big data, 2014.

11. Object Management Group, Common object request broker architecture (CORBA),http://www.omg.org/spec/CORBA/3.3, version 3.3, 2012.

12. RATHEE, S, An efficient run-time system for concurrent programming Language,Master of Computer Science,Concordia University Montreal, Quebec,Canada, 2009.

13. W. Gropp, E. L. and Skjellum., A, Using MPI: portable parallel programming with the message passing interface, MIT Press, Cambridge, MA, 1999 .

14. Joe Armstrong, Erlang – A survey of the language and its industrial applications, Proceedings of the symposium on industrial applications of Prolog (INAP96), 1996.

15. Charles Calkins, Using Erlang with CORBA and DDS, Object Computing, Inc.(OCI), Janvier 2011.

16. A. Corsaro, High performance distributed computing with DDS and Scala, In PrismTech Corp, Janvier Jul 2012.