MessageBus – How and What We Defined as Content for Our Messages

_{by Martin Führlinger, Software Engineer Backend}

In my last blog post I wrote about decoupling services using a message bus.

This post gives deeper insight into the actual content of our messages, explaining what we send within a message, what other possibilities we considered, and why we chose our content pattern.

Possibilities

Basically you can send whatever you want as message content, as long as it is serializable. As I described in my previous blog post, we are sending CRUD messages for our entities. Because we are using JSON API on our interfaces, we thought of using it for our messages as well. Here we can reuse our existing serializers. We invested in two different content styles:

1. small messages, containing some basic information on the entity only
2. full messages, containing the whole entity we want to send

Small messages

For this option we just sent the entity type and the entity i.d. For a run_session this looks like this:

{
  "id": "564d9756762542468b000001",
  “type": "run_session",
}

That’s enough information to find the entity again, but it leads to an additional HTTP request to the service responsible for that entity. Imagine 10 consumers are listening to that type of message. This would result in 10 additional HTTP requests to fetch the full entity.

Advantages

• less disk space on rabbitmq
• less traffic on the network regarding messages

Disadvantages

• nearly no information available in the message
• consumers have to fetch the full entity via HTTP, resulting in additional traffic
• consumers cannot decide if the message is relevant
• does not scale as more consumers lead to much more traffic

Full messages

Besides type and id, full messages include all attributes and maybe some important relationships of that entity. A fully serialized run_session looks like this (shortened, as it has much more attributes):

{
  "id": "564d9756762542468b000001",
  "type": "run_session",
  "attributes": {
    "user_id": "594",
    "start_time": 1447925590000,
    “distance”: 10000,
    < many more attributes >
  },
  "relationships": {
    < many relationships >
  }
}

Depending on the entity, this could become pretty huge. Mandatory relationships have to be defined upfront on an entity level. For some use-cases it makes sense to add relationships, for others it does not. It highly depends on the use-case and what the consumer will do. Try to find a trade-off between all consumers.

Advantages

• consumers have the full entity already, no need to refetch (less network traffic)
• consumers can decide if the message is relevant based on the entities attributes
• consumers can decide if the message is relevant based on the relationships (if applicable)
• scales
  

Disadvantages

• more disk space necessary in rabbitmq
• more network traffic regarding messages
• the entity could already be outdated when processing the message

Small vs Full

Comparing both possibilities, the amount of network traffic is less for the full message, compared to small messages, where you need an http request for every consumer. After doing some load-tests with both message contents, we decided to go for the full message option. It provides more benefits than the small message option, and we did not see any performance or disk issues. Basically the queues should be empty all the time anyway, as the messages should be consumed instantly.

Enveloping

In addition to sending entities in their JSON API serialized representation, we envelope it. So we put the relevant data into an envelope, which conforms with JSON API too. It follows this pattern:

{
  jsonapi: {
    version: "1.0"
  }
  data: {
    <serialized message>
  }
  included: [
    <included data e.g. serialized run_session>
  ]
}

We do this to have some additional meta-data for the message itself. As you can see in the example below, we are sending which server published the message, the time it was created, and also which type of event it is. Although we are currently using RabbitMQ, which allows us to filter according to the message topic via routing keys, we might not be able to  do this with another messaging system. No matter what message broker we are using, we could use the envelope for filtering, without checking the data of the message beforehand. Below you can see a serialized run_session.created message example.

{
  id:   "ab4d9338764642468b000006",
  type: "sample_message",
  attributes: {
    event:      "created",
    created_at: 1445511959000,
    created_by: "server-001"
  },
  relationships: {
    sample: {
        data: { id: "1234", type: "run_session" }
    }
  }
}

In the end, a full message would look like the example below. It has the envelope including the relevant data as a relationship.

{
  "data": {
    "id": "564d9758762542468b000003",
    "type": "sample_message",
    "attributes": {
      "event": "created",
      "created_at": 1447925592,
      "created_by": "server-001"
    },
    "relationships": {
      "sample": {
        "meta": {
          "version": 2
        },
        "data": {
          "type": "run_session",
          "id": "564d9756762542468b000001"
        }
      }
    }
  },
  "included": [
    {
      "id": "564d9756762542468b000001",
      "type": "run_session",
      "attributes": {
        "user_id": "594",
        "start_time": 1447925590000,
        < many more attributes >
      },
      "relationships": {
        < many relationships >
      }
    }
  ],
  "jsonapi": {
    "version": "1.0"
  }
}

To sum it up, enveloping and adding relationships helps us in various ways. We can add more information to the message of the entity. Some more details and what we achieved with this structure will be part of the next blog post about our MessageBus.

Want to continue reading our series about message bus? In our next post, we talk about how we keep our receivers fast and resilient.

***