MessageBus – Automated Documentation of our Messages

Written by Martin Führlinger, Software Engineer Backend

ALL ABOUT MESSAGE BUS

Welcome to the 6th post in this message bus related blog post series. I highly recommend reading the others too, to get the full context. These are: 

• Using RabbitMQ for decoupling our services
• How we defined our message content. 
• How we keep our receivers fast and resilient. 
• How dead letter handling with a dead letter exchange works
• How we are handling our dead letters

Now, I want to provide more insights into our documentation around messages and how we automated this.

Message Documentation

Documentation is hard. No matter who you ask, documentation is always hard to create and most important keep up to date. As always, we started with documenting our messages in our company wiki’s knowledge space. Whenever new messages were added or existing ones changed, someone had to add or update them. The message topics were listed in a matrix, containing the producer, the topics, the consumers, and some additional notes. It looked like this:

This process was slow and error-prone. So we wanted to improve this. 

How to Automate Documentation 

During one of our Day of New Ideas (DONIs), some colleagues and myself started to work on a small service to automatically document the existing messages in our system. We came up with a pretty simple service, named MessageDoc, just listening to all possible messages, so to all routing keys (find details about routing here), by listening to `#.#`.

Whenever we receive a message, we parse it and store its information. This means we store a document in a small MongoDB containing the messages meta data, like the producer, the message itself and some more attributes. This is done per topic. The flow of receiving and storing can be seen in the following diagram.

To see how this looks like in our database, here is an example: 

{
  "_id" : "sample.updated.altered",
  "producer" : "samples",
  "description" : "",
  "consumers" : [
    <list of consumers>
  ],
  "entities" : [ {
    "type" : "run_session",
    "updated_at" : ISODate("2020-11-11T14:56:02.304Z"),
    "example" : <complete message payload>,
  } ],
  "version" : 4124,
  "created_at" : ISODate("2019-11-28T11:15:26.105Z"),
  "updated_at" : ISODate("2020-11-12T06:00:12.950Z")
}

As we don’t want to store every single message, we only update the message’s content once a day. We sometimes have polymorphic messages. This means we are sending different types of entities inside the same message type. For instance, a “sample_message” can be of type “run_session”, “sleep_session” or others. We store only one example per entity type.

We know who the producer of a message is, as it is part of the message payload (see how we defined our message content). Whenever a document is written to mongodb, we also update the consumers asynchronously. We achieved this by querying the Rabbitmq API configuration via a HTTP request (using the all-configuration path), parsing it and updating all consumers of all our stored messages. Details about “consumers” can be found in the first blog post in this series.

Building a simple Web-Interface

To be able to access the stored information without querying the database directly, we built a simple web interface on top of this. Basically, the overview looks similar to the previous matrix in the wiki. It shows all topics grouped together by the producer.

When accessing a topic, the topic details are shown, with all the consumers again for this specific topic. This also means the consumers of a topic can be different in the details view, as e.g.”samples.created.completed” messages are consumed by other services than “samples.updated.altered”. This topic detail view also allows to add some description to the topic.

As mentioned above, we store an example message for every topic and every entity type once a day. These can be shown too by expanding the corresponding entity link below.

Making Manual Message Documentation Obsolete 

With this service in place, we were able to deprecate the message documentation in the wiki. Even if we delete data stored in the message doc service, it is automatically recreating all the information. The only thing which would be lost are the additional descriptions per topic which were added manually. This simple service, with less than 1000 lines of code overall, made manual documentation of messages and its content obsolete, and our life easier.

***