Real-time Monitoring and Analytics
Quick is based on Apache Kafka. It organizes and stores event streams in topics. In our case, we have a
vehicle topic containing the vehicle name and range as well as a
status topic for the emitted status events. We can process such event streams with the help of Kafka Streams. For example, we can accumulate status events with the same trip id into a trip. We simply group the incoming status events by their trip id and append them to a list. The result is written into the
You can find the full code in our repository. The Kafka Streams application is written with our streams-bootstrap library, which, among others, offers sensible defaults and reduces the required boilerplate code. Of course, you are not limited to simple aggregation. With Kafka Streams you can also build more advanced applications. One could for example build a predictive maintenance service with it.
Having our topics defined, we start by modeling the data required in the dashboard. Quick’s querying logic is built upon the data query language GraphQL. It allows us to create a global schema of our data and the supported operations. Subscriptions are one type of such operations, which allow us to consume real-time updates of the data through websocket connections. The first snippet shows an exemplary GraphQL schema for live updates of the emitted status events. With that, we have a subscription operation called
statusUpdates that we can use to get live updates of
Status events. The second snippet shows how a
Status looks like.
Besides the live updates, we also require access to a single trip. A trip is the accumulation of all statuses with the same trip id. As we want to query this information on demand, subscriptions do not work in this case. GraphQL offers the
Query operation instead. The third snippet shows a query called
trip that allows us to pass an id as an argument and returns the corresponding
Quick introduces a custom GraphQL directive called
@topic. It allows you to annotate fields and connect them to a topic. With that, we can define the relationship between our GraphQL Schema and Kafka. We first connect the
statusUpdates subscription to the status topic. It ensures that each event written to the Kafka topic is pushed into the GraphQL websocket connection.
Second, we want to display information about a vehicle when querying a trip. Instead of creating a separate operation, we can add this information to
Trip itself: Trip has a new field
vehicle. It populated with the data from the
vehicle topic based on the value of the trip’s vehicle id field. One major advantage of GraphQL is its flexibility. When querying a trip, the user can decide if they indeed require the vehicle information. If this is not the case, the corresponding data is never loaded and thus no overhead occurs.
We are ready to process and query our data. We start by setting up our Quick instance. First, we initialize the Quick CLI, which requires a base URL and an API-Key. Second, we create a new gateway and apply our GraphQL schema.
First, we create all required topics. The command expects the topic name as well as the type or schema of key and value. Since we have complex values, we define a GraphQL schema. Then, we can start our Kafka Streams application. Quick supports running dockerized applications.
When all resources are up, we can start to ingest data into our system. Quick supports the ingest through a REST-API. For example, the first snippet shows a command ingesting new vehicles into the
vehicle topic. When cars are ingesting their status events into the system, we can start to use our query and subscribe operations. For example, we can run a subscription with these results: