Skip to main content

· 8 min read
Aaron Nwabuoku

Discoverer

Having a discoverable API makes it easier for your users to understand and build mental models of your application. New data and relationships can be found without the user knowing of their existence. It also makes it easy to inline documentation with links to external documentation.


When designing Web APIs, many constraints and factors affect the decisions we make as software architects and developers. Some of these constraints are internal (like what tools and libraries are available within our tech stacks), and others external (like choosing a representational format easy for clients to consume, such as JSON). Regardless of the specific constraints facing each API designer, we almost universally want our Web APIs to be easy to use, flexible, and consistent. This article by Jordan Ambra does a great job of explaining these "golden rules" we all hope to follow as API designers.

The best Web APIs allow API users to use a service with minimum effort and knowledge of the system. Presenting the API user with only the information required to perform a task at hand, with links to related information that they may be interested in, allows for the user to focus on using your API to do what they need to do at the moment while having enough context to learn more about your API should the need arise. This scheme leads to APIs that are discoverable: API users can interact with your API from a single entry point in a way that requires little to no prior knowledge, and "discover" the rest of your API as needed.

Having a discoverable API makes it easier for your API users to understand, build mental models of your application and its data. New data and relationships can be found, even without the user knowing of their existence. It also makes it easy to inline documentation with links to external documentation. Client applications built on discoverable APIs have simplified logic since the responsibility of creating link URLs remains with your API, meaning as your API changes and evolves, it's easier to change your URL-structure without breaking client applications.

An architectural style for discoverable APIs

Okay, we can't discuss discoverable APIs without talking about REST. The Representational State Transfer architectural style introduced by Roy Fielding, specifically its Uniform interface constraint, describes a powerful scheme to design discoverable APIs that has influenced virtually every discoverable API. The Uniform interface constraint introduced key concepts of a resource, resource identifier, resource representation and hypermedia controls to describe discoverable distributed systems. I'll be using these concepts, especially through the Richardson Maturity Model to describe an architectural style for asynchronous event-driven APIs. Let's go through these concepts in the context of a synchronous RESTful HTTP system first to familiarize ourselves with the concepts before we go asynchronous.

Richardson Maturity Model Overview The Richardson Maturity Model breaks down the core elements of a REST architecture into three steps.

Resources

The most basic concept of a REST style API is the concept of a resource. We think about our system as a collection of resources, each resource being a logical entity within the system with a unique resource identifier. Each resource has a resource representation exposing its data in an agreed representation format. A resource can be manipulated using verbs acting on its identifier.

In an HTTP context, our resource identifiers are URLs referencing our resources. We return resource representations as HTTP responses with a chosen content-type like JSON or XML.

Verbs

Resources within our system can be manipulated using verbs. Verbs define the actions we allow API users to perform on our resources. The REST styles a finite set of verbs to manipulate resources uniformly.

In an HTTP context, we can use the four main HTTP methods, GET, POST, PUT, and DELETE as verbs to fetch, create, update and delete resources respectively.

Hypermedia controls

Hypermedia controls are the final and most consequential concept for RESTful APIs. Hypermedia controls are what make REST APIs discoverable; by including references to a resource and other related resources in its representation and specifying what verbs can act on the resource, we control what actions an API user can take on within our system and can selectively expose information related to the user's context.

In an HTTP context, our resource representation can include the URL identifiers of related resources, the HTTP methods the user can use to send HTTP requests to those URLs, as well as additional information like external links to documentation.

What about asynchronous APIs?

The REST architectural style isn't specific to HTTP or any concrete application-level protocol. At ChatKitty, we applied RESTful principles to develop StompX - a convention for asynchronous event-driven APIs that extends the Simple Text Oriented Messaging Protocol (STOMP) with RESTful concepts and discoverability. REST has been applied extensively to synchronous HTTP APIs with formats and media types like HAL providing concrete conventions and structure for RESTful APIs. As real-time applications become more popular, we believe it's important to have an open standard and conventions to represent resources, their relationships and actions with a REST style to build flexible, open and discoverable APIs.

StompX is a simple set of conventions that gives a consistent and convenient way to hyperlink between resources in asynchronous event-driven APIs and discover actions that can be taken on those resources. StompX provides an opinionated set of conventions, rules, and structures for creating event-driven APIs that are compatible with the Richardson Maturity Model using the STOMP protocol as its underlying messaging protocol.

Resources

StompX is based on the concept of resources; the logical entities within your application and their representations. A StompX messaging API returns resource representations inside the text body of a STOMP message frame, inside of an event or relay event. A StompX resource representation is a normal JSON object with your resource state as you would have it any JSON response, as well as metadata providing information about how to fetch more data, take actions, and listen to events related to the resource and other relevant resources.

Actions

API users manipulate StompX resources using actions. Unlike in an HTTP RESTful where there are a few verbs, a StompX API consists of multiple actions, each corresponding to a real-world action a user can take to interact with resources within the system. Actions performed changes the state of your application and can trigger API events. API clients perform actions by sending STOMP send frames to an action destination with a JSON text body containing parameters required to perform the action.

Events

Since you can't be event-driven without events, events are first class citizens in StompX APIs. API events inform your users what happens in your system and allows them to react in real-time. A StompX event has a type tied to a specific type of action, is always triggered by an action previously performed and embeds an API resource that was a consequence of causing the action. API users can subscribe to API topics, listen to events of a specific type and asynchronously perform their own actions as a result. When an action occurs and triggers an API event, the event is sent via a MESSAGE frame to API users subscribed to a topic related to the event.

Topics

A StompX resource that can be acted on may expose one or more topics. A StompX topic is a STOMP destination that API users can subscribe to receive events related to the resources. Events sent to a topic may have multiples types but can only embed resources of the same type. API clients can subscribe to a topic while they're interested in events related to the topic and unsubscribe once they are no longer in receiving new events.

Relays

Relays let you fetch resources and retrieve data asynchronously using a request-reply pattern. Relay acts like a special action/topic/event, where you subscribe to a relay destination, it triggers a relay event with the data you requested, and automatically unsubscribes you from the relay destination.

Hypermedia controls

Like RESTful APIs, StompX resource representations include hypermedia controls to be discoverable. Resources representations embed the actions, topics, relays relevant to themselves, allowing the API user to traverse through the API.

The StompX specification at this point is still very informal and incomplete. We've been using StompX both internally and externally with our Open Chat API, and we plan on formalizing the specification. As web APIs move towards asynchronous and event-driven architectures, we hope StompX can provide an open and easy way for messaging services to communicate.


This article features the image "AMF Discover" by Wha'ppen licensed under CC BY 2.0

· 9 min read
Aaron Nwabuoku

Keep Things Simple

STOMP is a straightforward and easy to use open protocol that provides an interoperable text based format for message brokers and applications to communicate. STOMP allows real-time messaging components to share messages and communicate effectively.


To build real-time web applications you'll need full-duplex communication between a client application and a web server - having a message broker is also essential if you want reliability, and if you want to be able to handle messaging at scale. These different components have very different requirements and often share very little in terms of their tech stack, the programming languages they are written in, and the libraries they use. Despite these differences, the components involved in real-time messaging systems need to be able to send and receive messages agnostic of each component's tech stack in both directions. The WebSocket Protocol was introduced in 2011 to enable two-way communication between clients, and a remote host over a single TCP connection.

Since then, the WebSocket Protocol has become widespread, and it is supported across the web, both by browsers and mobile devices. The WebSocket Protocol although ubiquitous is a low level protocol functioning as a thin layer over TCP. The WebSocket Protocol upgrades a HTTP connection into a duplex connection and transforms a stream of bytes into a stream of messages without defining what those messages look like or how they're structured. With the WebSocket Protocol alone there isn't enough structure for real-time messaging components to route or process messages. This lack of structure makes the WebSocket Protocol by itself too low level for any serious application. Real-time messaging systems can take advantage of the wide support for WebSocket and provide structured messaging by using a WebSocket sub-protocol. The WebSocket Protocol allows itself to be extended by application-level messaging protocols that structure messages and provide additional information to messaging components.

TLDR - we need a simple messaging protocol we can use on top of the WebSocket protocol to allow our real-time messaging components to share messages and communicate effectively.

A simple application level messaging protocol

The Simple Text Oriented Message Protocol (STOMP for short) makes a great WebSocket sub-protocol. Inspired by HTTP, STOMP defines text frames consisting of a command, an optional set of headers and an optional body. Real-time messaging components send and receive these frames containing messages, and are able to forward the messages based off their destination. STOMP is also widely supported with libraries and tools across the web ecosystem.

Setting up a STOMP client

At ChatKitty, we've created StompX - an open protocol based off and compatible with STOMP. Our Real-time Messaging (RTM) API uses the StompX protocol, and you can connect to it with using any STOMP compliant client library to begin building your chat app powered by ChatKitty.

Here are a few STOMP client libraries we recommend:

  • STOMP JS A JavaScript library for Web browsers using STOMP Over WebSockets

  • StompClientLib An iOS library written in Swift using Facebook's SocketRocket.

  • stomp.py A Python STOMP client library

Connecting to a STOMP service

We'll be using the STOMP JS client library to connect to the STOMP compliant ChatKitty RTM API.

First, we have to add STOMP JS to our JavaScript project:

Using npm we install stompjs to our project dependencies

npm install @stomp/stompjs websocket --save

Now that we've added STOMP JS, we can now connect to ChatKitty using the STOMP protocol:

import {Stomp} from '@stomp/stompjs';

var url = `wss://api.chatkitty.com/stompx/websocket
?api_key=c81cd8ae-2d42-41d3-9a75-433f39c63782
&stompx_user=my_test_user`;
var client = Stomp.client(url);

client.connect({ 'host': 'api.chatkitty.com' }, function () {
console.log('Connected to ChatKitty');
}, function () {
console.log('Error connecting to ChatKitty');
});

Awesome! Just like that we were able to connect to ChatKitty. 🙌

What's happening here?

If we check the STOMP JS debug logs, we'll see a few STOMP frames sent back and forth while we connect to ChatKitty:

Web Socket Opened...

>>> CONNECT
host:api.chatkitty.com
accept-version:1.0,1.1,1.2
heart-beat:10000,10000


Received data

<<< CONNECTED
user-name:202:user:my_test_user
version:1.2
session:ID:b-b43a10b0-57c6-47bd-8bef-21b73be6fa9b-1.mq.us-east-1.amazonaws.com-37871-1596011033764-3:201
heart-beat:10000,10000
server:ActiveMQ/5.15.12
content-length:0


connected to server ActiveMQ/5.15.12

The CONNECT STOMP frame

After an underlying TCP connection has been made, a STOMP client can start a STOMP session by sending a CONNECT frame:

>>> CONNECT
host:api.chatkitty.com
accept-version:1.0,1.1,1.2
heart-beat:10000,10000

If the service accepts the connection attempt, it will respond with a CONNECTED frame:

<<< CONNECTED
user-name:202:user:my_test_user
version:1.2
session:ID:b-b43a10b0-57c6-47bd-8bef-21b73be6fa9b-1.mq.us-east-1.amazonaws.com-37871-1596011033764-3:201
heart-beat:10000,10000
server:ActiveMQ/5.15.12
content-length:0

To connect to a STOMP service, our client must send a couple of headers in our CONNECT frame.

The host header specifies the host we're connecting to. In our case, it's the hostname of the ChatKitty service api.chatkitty.com.

The accept-version header tells the service what versions of the STOMP protocol our client is able to support. The service can then negotiate which version to use based on the highest version supported by both the client and the service. Currently, there are three versions of the STOMP protocol and STOMP JS is able to handle all three, so we specify 1.0,1.1,1.2.

Optionally, we've included a heart-beat header to test the healthiness of our underlying TCP connection. We've told the service that we can send a heart-beat every 10000 milliseconds, and we can handle a heart-beat from the service every 10000 milliseconds. A heart-beat is simply an empty message sent through the connection.

The CONNECTED STOMP frame

If the service accepts the connection attempt, a STOMP service responds to a CONNECT frame with a CONNECTED frame:

<<< CONNECTED
user-name:202:user:my_test_user
version:1.2
session:ID:b-b43a10b0-57c6-47bd-8bef-21b73be6fa9b-1.mq.us-east-1.amazonaws.com-37871-1596011033764-3:201
heart-beat:10000,10000
server:ActiveMQ/5.15.12
content-length:0

The user-name header tells us what user owns this STOMP session. It should match the stompx-user value we specified the connection URL.

The version header returns the version of the STOMP protocol the service negotiated for our connection based on the accept-version we sent in our CONNECT frame.

The session header value uniquely identifies our STOMP session.

The heart-beat header has the same value as the heart-beat we sent in our CONNECT frame if the server accepts the heart-beat frequencies we requested.

The server header contains information about the server the STOMP service uses. ActiveMQ/5.15.12 means ChatKitty uses a version of Apache ActiveMQ as a message broker. 🤔

The content-length header specifies the size of a frame's body in bytes, since the CONNECTED frame doesn't have a body, it has a value of 0.

Subscribing to a STOMP destination

We subscribe to a STOMP destination to listen to messages sent to that destination.

Using the client from earlier we subscribe to two destinations

client.subscribe('/topic/v1/channels/1', function (message) {
console.log('Received message: ' + message.body);
}, {'receipt': 'receipt-0'});

client.subscribe('/topic/v1/threads/1.messages', function (message) {
console.log('Received message: ' + message.body);
}, {'receipt': 'receipt-1'});

We subscribe to the channel destination to "enter" a channel, and to the channel's main thread's messages destination to listen to message events.

If we check the STOMP JS logs again, we see:

>>> SUBSCRIBE
receipt:receipt-0
id:sub-0
destination:/topic/v1/channels/1


>>> SUBSCRIBE
receipt:receipt-1
id:sub-1
destination:/topic/v1/threads/1.messages

<<< RECEIPT
receipt-id:receipt-0
content-length:0

<<< RECEIPT
receipt-id:receipt-1
content-length:0

Interesting, the service responds with a couple of RECEIPT frames with receipt-id headers matching the receipt headers we sent. We'll get into why in a bit.

We're sending a few headers here:

We specified receipt headers. Any client STOMP frame after the initial CONNECT frame can include a receipt header with an arbitrary value. Specifying a receipt causes the service to acknowledge the processing of a client frame with a RECEIPT frame. The received RECEIPT frames have receipt-id headers with the same value as the receipt header we sent.

We also include a id header to uniquely identify this subscription within our STOMP session. The id header allows us and the STOMP service to associate MESSAGE and UNSUBSCRIBE frames with our subscription.

Lastly, we have a destination header - this tells the STOMP service where we're subscribing to. Once we subscribe to a destination, we're able to receive messages sent to the destination.

Sending a STOMP message

We subscribe to a STOMP destination to listen to messages sent to that destination.

Using our client instance we send a message

client.send('/application/v1/threads/1.message', {
'content-type': 'application/json',
'receipt': 'receipt-2'
}, JSON.stringify({'type': 'TEXT', 'body': 'Hello world!'}));

We send a message to the channel's main thread's messages destination (which we subscribed to earlier).

If we configure our client to log STOMP frame bodies and check the logs, we see:

>>> SEND
destination:/application/v1/threads/1.message
content-type:application/json
receipt:receipt-2
content-length:37

{"type":"TEXT","body":"Hello world!"}

<<< RECEIPT
receipt-id:receipt-2
content-length:0

<<< MESSAGE
content-length:341
timestamp:1597028776273
content-type:application/json
message-id:ID\cb-b43a10b0-57c6-47bd-8bef-21b73be6fa9b-1.mq.us-east-1.amazonaws.com-37871-1596011033764-3\c122\c-1\c1\c41
priority:4
subscription:sub-0
destination:/topic/v1/threads/1.messages
expires:0
content-length:341

{"type":"message.created","version":"v1","resource":{"type":"TEXT","user":{"type":"PERSON","name":"my_test_user"},"receipt":"receipt-2","body":"Hello world!","createdTime":"2020-08-10T00:36:37.655Z","_relays":{"user":"/application/v1/users/110.relay","thread":"/application/v1/threads/1.relay","channel":"/application/v1/channels/1.relay"}}}

We send a JSON string payload to the service, the service acknowledges our message and forwards us the message creation response, since we're also subscribed to the message's destination.

The content-type header helps a STOMP service understand what type of message our client is sending and the destination header tells the service where to deliver the message.

We see a few new headers in the response sent by the service:

The timestamp header isn't specifically defined by the STOMP protocol, but the protocol allows for custom headers. Custom headers allow you to extend the protocol to better suit the needs of your application. Here the timestamp represents the time the service sent the message in Unix Epoch Time.

The service also includes a message-id header to uniquely identify the returned message.

The subscription header matches the id we sent when creating our subscription to let us know which subscription should receive the message.

The expire header is another custom header. A value of 0 means our message is always valid and does not expire.

And just like that, we're able to send and receive messages from a real-time messaging service - without having to know anything about its internal implementation. By using the STOMP protocol we were able to structure our messages and enrich each message with information needed by both the service and our client. Because the STOMP protocol is text-based and involves a few commands, the process was easy to debug and understand. Wasn't that simple? 😉

The STOMP protocol also defines more advanced features like message transactions and client message acknowledgment. You can learn more about the STOMP protocol here: https://stomp.github.io/stomp-specification-1.2.html/


This article features an image by gdsteam.