An easy way to build microRESTservices with Jolie

This post is about a couple of tools [https://github.com/jolie/jester] we developed for facilitating the programming of REST microservices with Jolie. Personally I am not a big supporter of REST services, but I think that a technology which aims at being a reference in the area of microservices like Jolie must have some tools for supporting REST services programming. Why? Because REST services are widely adopted and we cannot ignore such a big evidence.

Ideally, Jolie as a language is already well equipped for supporting API programming also using http, but REST approach is so deep coupled with the usage of the HTTP protocol that it introduces some strong limitations in the service programming paradigm. Which ones? The most evident one is that a REST service only exploits four basic operations: GET, POST, PUT and DELETE. The consequence of such a strong limitation on the possible actions is that the resulting programming style must provide expressiveness on data. This is why the idea of resources has been introduced in REST! Since we cannot programming actions we can only program resources.

Ok,  let's go with REST services!

...But,
...here we have a language, Jolie, that is more expressive than REST because the programmer is free to develop all the operations she wants. From a theoretical point of view, in Jolie she can program infinite actions instead of only four!

…

- Houston we have a problem! We need to put infinite operations inside just four!!!

- Why only four when we can have infinite?

- This does not matter now Houston, we have to do it!

…

Ok no problem! Follow our instructions!
First of all, take note that here we want to preserve all the benefits of using Jolie, thus the possibility to develop all the operations we want, but finally publish the microservice as a REST service. We achieve such a result by exploiting a specific microservice architetcure which is a composition of a router and the target microservice we want to publish as a REST one. The router, as described in Fabrizio's paper [https://arxiv.org/abs/1410.3712], is in charge to transform REST calls into standard Jolie calls.



Ok, in order to exaplain how to proceed, let us consider as a target microservice the demo one reported in the jester project [https://github.com/jolie/jester/tree/master/src/jolie/tools/demo]. It is a very simple service which emulates a manager of orders which supplies four operations: getOrders, getOrdersByItem, putOrder, deleteOrders.

Ok, there are four operations but it is just an example, we could have more than four operations :-). The main question now is:

What we have to do for transforming these operations in REST operations?
We need to use the jolie2rest.ol tool you can find the jester project. Such a tool analyzes the interface of the demo service and it extracts a descriptor for the router which enables it to publish the demo service as a REST service.

Very simple! But before running the tool we need to know something more. We need to define how to convert the single operations of the demo microservice will be tranformed into the REST one. In particular, fro each target operations we need to specify if we want a GET operation, a POST operation, a PUT operation or a DELETE operation. It is possible to provide these instructions, directly into the interface of the demo service by exploting the annotation @Rest into the documentation comments. As an example let us consider the operation getOrders:

/**! @Rest: method=get, template=/orders/{userId}?maxItems={maxItems}; */
getOrders( GetOrdersRequest )( GetOrdersResponse )

The annotation defines that the operation getOrders it must be transformed into a GET http method and the URL template that must be adopted is /orders/{userId}?maxItems={maxItems}. What is the template URL?
Since the REST services deal with resources, the URL is used as a means for expressing the resource we want to access. Here we use the template as the means for transforming a call to an operation of the target service into a resource. In particular, the parameters between curly brackets will be directly filled by using the corresponding values of the request message node which is defined in the GetOrdersRequest type:

type GetOrdersRequest: void {
    .userId: string
    .maxItems: int
}

Now, we can proceed by running the following command:

jolie jolie2rest.ol localhost:8080 swagger_enable=true

where localhost:8080 is the location where the router is deployed and the parameter swagger_enable specifies if we want to enable the creation of the swagger json descriptor file.
Note: the file service_list.txt contains the list of the target microservices to be transformed and the related inputPorts to be converted (more instructions can be found here: ref).

As a result the tool will generate two files:

• router_import.ol
• swagger_DEMO.json

The former file must be copied in the router folder and it contains the script which enables the router to transform the REST calls into the operations call of the target service. The latter is just the swagger descriptor to be provided in input into a SwaggerUI. You don't have a local SwaggerUI available? Follow these instructions to have it locally, otherwise go to point 6:

1. Prepare the web server of the SwaggerUI application by downloading Leonardo from here [https://github.com/jolie/leonardo]
2. Go to this SwaggerUI URL [https://github.com/swagger-api/swagger-ui/archive/v2.2.3.zip] and download the related web application project. 
3. Copy the content of the folder dist of the SwaggerUI project inside the folder www of Leonardo.
4. Open a shell and run jolie leonardo.ol
5. Open a browser at the url http://localhost:8000, the SwaggerUI web application should appear.
6. Copy the swagger_DEMO.json file into the folder where it is reachable from the SwaggerUI, in the Leonardo scenario put it inside the www folder.
7. Write in the explorer bar of the SwaggerUI the URL for reaching the swagger descriptor, in the Leonardo case write: http://localhost:8000/swagger_DEMO.json

After this step, you should see the Swagger definition of your target demo service transformed into a REST one. In particular, you should see something like this:







As you can see the four operations of the target service have been transformed in the four different types of REST operations. Have a look to the interface annotations of the demo service for finding the matches with the Swagger interface!

Nice, but what happen if I want to transform more than one Jolie microservices instead of a single one?
This question is dealing with an architecture like the following one where the router is connected with several microservices:



There are no particular problems in achieving such a result. It is sufficient to list all the target microservices with the related input ports in the file service_list.txt and re-launch the jolie2rest tool!

Ok Houston! Here we are! But is it possible to avoid the usage of the router?
Yes, it is possible. But only if you accept to not completely adhere to the REST approach. A jolie microservice can be directly published by using a http inputPort with message format set to json [http://docs.jolie-lang.org/#!documentation/protocols/http.html#http-parameters]. In this case the microservice will be able to serve the http requests without requiring any router or proxy in the middle. If you want this, change the inputPort protocol of the demo service to a http one and then use the jolie2rest tool with the parameter easy_interface set to true.

jolie jolie2rest.ol localhost:8080 swagger_enable=true easy_interface=true

In such a case the router_import.ol file is not generated but only the swagger_DEMO.json one. The operations are all converted in POST methods and the URLs do not follow the templates but the request json messages must be entirely defined in the body of the message. Try to replace the file swagger_DEMO.json in the SwaggerUI and perform some calls.

Generating client stubs from an existing Swagger definition:
A last tool which can be very useful when integrating existing REST services in a Jolie architecture, is jolie_stubs_from_swagger.ol. Such a tool takes in input an existing Swagger definition and generates a Jolie client for each published api.

As an example you could try it by generating the clients for the petstore example supplied by the Swagger project [http://petstore.swagger.io/v2/swagger.json]. In order to do it, create a target folder where you want to store all the generated clients (for example petstoreFolder) and then run the following command:

jolie jolie_stubs_from_swagger.ol http://petstore.swagger.io/v2/swagger.json petstoreService petstoreFolder

where petstoreService is the token that will be used for generating the Jolie outputPort name of the petstore service inside the clients. As a result, in folder petstoreFolder you will find a list of Jolie clients. In particular, you will have a client for each api defined in the petstore swagger definition.

If you want to try to send a request just open one of them and create the request message. For example open the getOrderById.ol and prepare the request message by adding the following jolie code:

with( request ) {
.orderId = 8
};
then run the client as a usual jolie script:

jolie getOrderById.ol

the result should be printed out on the console!

You can exploit these clients inside your existing jolie microservices. Just note that the generated file outputPort.iol defines all the information necessary to declare the outputPort to be used. Thus just include it in your microservice project and then make the calls when is more useful for you!

Conclusion
Houston, everything is clear now! :-) With the REST tools described in this article we want to improve the Jolie language providing the possibility to publish Jolie microservices as REST services and giving an easy way for generating clients from existing REST services. We hope this could be useful for your  projects and, please, do not forget to send us your feedbacks and improvements!

Integrating Wordpress with a Jolie service

Wordpress developers could have the problem to integrate the website they are developing with some extra data which come from external resources. Data synchronization could be not easy, it must be scheduled, data are visible on the web application with a period equal to the synchronization timeout and it usually requires a level of maintenance that is high with respect to the amount of synchronized data.

In these cases it could be useful to synchronously call an external server in order to retrieve the data we need. Jolie could be very helpful for developing a fast API server to be integrated within your web site.

This short post describes a way for easily integrate Wordpress with a Jolie service.

First of all, let me consider the simple case of getting a list of addresses from a Jolie service. This service could be written as it follows:

type GetAddressRequest: void {
    .name: string
    .surname: string
}

type GetAddressResponse: void {
    .address: string
}

interface ServerInterface {
    RequestResponse:
        getAddress( GetAddressRequest )( GetAddressResponse )
}

execution{ concurrent }

inputPort AddressService {
    Location: "socket://localhost:9001"
    Protocol: http
    Interfaces: ServerInterface
}

init {
    with( person[ 0 ] ) {
        .name = "Walter";
        .surname = "White";
        .address = "Piermont Dr NE, Albuquerque (USA)"
    };
    with( person[ 1 ] ) {
        .name = "Homer";
        .surname = "Simpsons";
        .address = "Street 69, Springfield USA"
    };
    with( person[ 2 ] ) {
        .name = "Sherlock";
        .surname = "Holmes";
        .address = "221B Baker Street"
    }


}

main {
    getAddress( request )( response ) {      
        index = 0;
        while( index < #person
                && person[ index ].name != request.name
                && person[ index ].surname != request.domain ) {
            index ++
        };
        if ( index == #person ) {
            throw( PersonNotFound )
        } else {
            response.address = person[ index ].address
        }
    }

}

In this case I simulated a database with a simple vector of persons. The service is located in socket://localhost:9001 and you can easily invoke him through a browser by using this url: http://localhost:9001/getAddress?name=Sherlock&surname=Holmes

Clearly, this is an example which runs on your local machine, it is simple to export it in a distributed scenario by simply running Jolie on the remote machine, let me suppose it is located at IP X.X.X.X.

When the Jolie service is executed, we can invoke it within a wordpress page. In order to do this, simply add the following PHP code on the page where you want to show the results:

$data = array(
    'method' => 'POST',
    'timeout' => 45,
    'httpversion' => '1.0',
    'blocking' => true,
    'headers' => array(),
    'body' => array( 'name' => 'Sherlock', 'surname' => 'Holmes' )
    );

$url = 'http://X.X.X.X:9001/getAddress';
$result = wp_remote_post( $url, $data );

print_r( $result );

if( is_wp_error( $result ) ) {
echo $result->get_error_message();
}

Where $result contains the response message.

You can also exploit a JSON format message by adding the following parameter to the http protocol of the Jolie service and re-arranging the PHP code for managing JSON messages.

 inputPort AddressService {
    Location: "socket://localhost:9001"
    Protocol: http { .format="json" }
    Interfaces: ServerInterface
}

Moreover you could connect the Jolie service to a database for retrieving the data you need, check the Jolie tutorial page for getting how to use databases in Jolie.

An abstract means for dealing with service and microservice componentization

Some months ago I read this article about microservices and I found it very interesting because of the experience we have had so far in the development of Jolie language. Jolie was born as a language for crystalizing SOA principles in a specific domain language but, day by day, we were becoming more and more confident that the boundaries of SOA were not enough for dealing with some issues that we had encountered during programming with Jolie. Thus, I was very surprised and excited when I read about microservices because I think they enlarge the domain  of service oriented programming in a way which is coherent with the results we have had so far. This is why I would like to share our experience with the microservices community. On the one hand my aim is to promote the usage of Jolie as one of the technologies which could help in the design and the development of microservices, and on the other hand I hope to give my contribution in the definition of concepts and practices for microservices. In particular, in this post, I would like to introduce a simple concept called Service Threshold which could be of help when reasoning about service and microservice componentization. The main idea behind the service threshold is to provide an abstract boundary which allows for the separation among a world of services and a world of components, where a component is a general piece of software.

The Service Threshold

The Service Threshold is the architectural line which separates services from other components (that are not services) as shown In Fig 1. where I simply represent services as hexagons and other components as squares.

Fig 1. The Service Threshold abstractly separates services (hexagons) from 

components that are not services (squares).

It is a very simple concept but some questions need to be addressed:

Why do I need to introduce it?

A line always represents a division, in this case a logical division between two kinds of things that we consider different. I introduce it because it helps me to change my perspective about distributed software architectures. Indeed, I need to think of them in a way which allows me to design, deploy and manage them quickly and easily. Summarizing, I need that line because I am looking for a simplification of distributed and heterogeneous systems engineering. 

Are there different properties above and under the service threshold?

Clearly, there are differences between the top plane and the bottom one. If I operate "above" the line I would like to be in a world where a service is the basic element which can be composed and managed by following some specific rules related to services, no other kind of software components exist. Otherwise, if I am "under" the line, the service is just something I have to create, it is the target of my work.

Which are the differences between a service and component?
A service is an autonomous running piece of software able to provide its functions when invoked. Its execution does not depend on components out of those strictly required.  The service functions should be always available for invocations, if not the service should reply with a specific fault message. Invocations are always achieved by means of message exchanges which could be synchronous or asynchronous. It does not matter the transport protocol used for the message passing. A service can be stateless or stateful.
A component is a piece of software which cannot be considered as a service.

Which is the right place to put the threshold line?

This is the core question every distributed software architect should deal with. Depending on the answer, I would design a different system with different properties. If I put this line very high I will have few services and a lot of components, if I put this line at a low level I will have a lot of services and few components. So, the question could be changed in: how many services do I need? Again, which are the basic properties that I need to take into account in order to identify all the required services?

The service threshold is not a line but a boundary

A distributed system is a system where software artifacts are separated, thus even though I have discussed the service threshold as a horizontal line, it fits better to think about it as a boundary which allows me to isolate and identify the different services involved into a system.

Fig 2. The Service Threshold is a boundary which allows me to isolate and identify the services.

Once defined, the service  threshold allows me to focus on the single service without being worried about its connections too much. I can just limit my effort to the development of the service as an autonomous specific provider of functions. The final target is to provide a basic set of services which represent the core functions of the application I am interested in. Some of them could be bought from a third party supplier, the others could be developed from scratch. The most important think is that each service function must be well separated from the others and, at the same time, it must be strictly necessary to the overall system. As an example, let me consider the human body where each organ supplies a well defined function which is fundamental for life. They are precisely separated by means of different tissues and they are characterized by different biological properties. 

All of them are needed, because the functions that they supply are needed, nothing is too much, nothing is too less. They are connected in a perfect and inexpensive way, because they all participate to form the human body, a more complex and completely different organism. 

Service connections and dependencies

Once all the services have been identified, I can deal with service communication and connections. A communication is performed when a message is sent by a service and received by another one, where a message is a limited set of data transmitted in a limited interval of time. A connection represents the possibility to perform a communication between two services: a sender and a receiver. Usually a service receiver is not aware of the sender (this is not true in the case of a stateful long running transaction). A connection becomes a dependency when the sender needs to invoke another service in order to accomplish its own tasks.

Fig 3. Service A has a dependency on service B.

Service dependencies allow me to divide services in two main categories: 

• Autonomous services
• Depending services

Autonomous services usually control resources such as databases, electronic devices, computational resources, legacy systems, etc. These kind of services are the software artifacts which enable the integration of  the underlying resource with a system of services and the APIs that they provide represent the only way for interacting with it. They are usually designed to be independent from the context they will be inserted in and they are strongly loosely coupled.

Depending services are built upon other services because they require to call other services to accomplish their tasks. They could be:

• Mixed services
• Pure coordination services

Mixed services control resources as the autonomous ones (e.g. a database) but they also need other information from other services for providing all their functions.
Pure coordination services do not control any data resource but they just call other services for achieving their tasks (orchestrators are usually pure coordination services). 

Fig.4 Autonomous services, Mixed services and Pure coordination services

By componing these kind of services it is possible to obtain a complex distributed system which provides a new set of functions different from those of the constituent ones. I call service complex system a distributed system whose components are all connected services.

Fig. 5  A service complex system is a distributed system whose components are all connected services.

The service threshold can be applied over a set of services

A service complex system could be formed by tens or hundreds of services. Clearly, when the number of services is very high it could be difficult to manage and maintain the system because of its complexity. In this case, in order to simplify the management, I exploit another interesting property of the service threshold: I apply it over a set of existing services by grouping them. The services grouped by a service threshold can be seen and treated as a single service. In this case all the inner connections are hidden for an external observer and the only ones that are relevant for us are those that can be observed outside the service threshold boundary. 

Fig 5. The service threshold can be applied over a set of services in order to group them
and identify them as a single service.

There are no limits to the application of the service threshold over a set of services, thus I could apply it again and again in order to reduce the system to a limited set of services. Since a service is usually stateless and always ready to serve different clients concurrently, it is possible that the same service could be enclosed by two (or more) different service thresholds as in the following picture:

Fig 6. A service could be enclosed by two different service thresholds. 

In this case the resulting services both contain the same service inside. Is this possible? Yes, in the following section I will answer to this question.

Primary thresholds and abstract thresholds

As I said before a service is always a running piece of code. Thus, a set of files in our file system is not a service, it is just a set of file which could contain executable code but they are not a service. They start to become a service when they run. This is a fundamental point because a service exists only if it is able to respond to our requests. This is why it is possible to double a service by wrapping it into two different service thresholds as represented in Fig 6. It is possible because I am grouping the service functions, not the piece of code. I call abstract service threshold all the service thresholds which group existing services. When I apply the abstract service threshold to a set of services I loose the deployment details of a service (where it is, how it is composed, which technology it uses, etc) because I am focusing only on its functions. On the contrary, the primary service threshold is applied to a set of components that, when executed, they become a service. By definition. the primary service threshold has a completely different nature with respect to the abstract one because it will be applied to components for tting services. Nevertheless, I both call them service threshold because they both allow me to identify a new service even if they are applied to different domains.

Fig 7. The Abstract Service Threshold (on the left) groups existing services, the primary service threshold groups a set of components that, when executed, they become a service.

The primary threshold is a technological threshold

The Primary Threshold is a technological threshold because it clearly defines the boundary between service composition mechanisms and other technologies mechanisms. It does not matter which kind of technology I select, but it is important that I will be able to satisfy some minimum requirements that allow me to create a service (I suppose you would ask me which are the minimum requirements for creating a service, wait wait wait I will discuss them in another post). Within the primary threshold I could exploit any kind of technology and any kind of communication approach (such as in memory data exchanges, file exchange, etc.).  I can also use specific applications such as web servers, application servers, etc. The final target is to prepare the basic software layer which allow me to jump into the service world. Everything happens inside this threshold is a matter strictly related to the technologies I chose for building the service. This point is important because allow me to state that there exists a trade off between the benefits obtained by the introduction of a service and the technological overhead I have to pay. Is it simple to maintain a service developed with a specific technology? Which skills I require for achieving the development? Does the technology easily scale with the service load? Moreover, do I need to adopt the same technology for all the services of our system or do I need to change it? When? Will I be able to maintain the required skills for managing all the existing services I have? These are not trivial issues because more is the number of technologies I adopt, more is the human resource knowledge of my team I have to manage. Indeed, if I adopt a technology I need someone who is able to manage it. 

Conclusions

In this post I discussed service and microservices componentization by introducing the concept of service threshold which is a useful tool for approaching distributed systems based on services and microservices. I hope this post could be useful for those people that are trying to approaching service system design in a simple and intuitive way. Service thresholds could be a useful means for structuring the design and defining the development phases of a service system.