(Techs: Log One)
Abstract: Creation of a system that gathers timeseries data from decentralized IoT devices and visualize it on dashboard. Bidirectional communication maybe needed in order to optimize IoT parameters on the fly.
Data Aggregation: In that project we used TimescaleDB https://github.com/timescale/timescaledb which works as PostgreSQL extension and solve many problems related to time-series. Thanks to TimescaleDB data partitioning, timeseries data aggregation is easy, data maybe partitioned by timestamp or day.
TimescaleDB: Let’s check what our Python Engineer Hamed wrote about that, I will quote few paragraphs.
“A key feature of TimescaleDB is the support for automatic partition of the data array. The incoming data stream is automatically distributed among the partitioned tables.”
“Sections are created based on time (each section stores data for a certain period of time) or relative to an arbitrary key (eg device identifier, location, etc.). Partitioned tables can be spread across different drives to optimize performance.”
“For queries, a partitioned database looks like a large table, called a hypertable. A hypertable is a virtual representation of many separate tables in which incoming data is accumulated.”
If You want to know more or need help, I recommend to check their slack https://timescaledb.slack.com, very cool, professional help.
Microservices: We decided to start with one service based on FastAPI (https://fastapi.tiangolo.com/) and task queue Celery(https://docs.celeryproject.org/en/stable/index.html). About celery, recently we were postponing addition of the task queue because we didn’t want to increase the complexity of the system by adding next microservice. But this was caused by wrong reasoning about Celery and treating it as standalone microservice which is not true.
Amount of microservices must be decided based on use cases and production requirements. So, if our system is integrated with third party service through a webhook which at highest pick may send large amount of data, it is wise to have dedicated service that handle only that as we can easily scale it up on demand.
Celery maybe deployed as standalone service but in above mentioned case we used it as a tool/extension for an existing microservice to handle repetitive, heavy task in the same use case domain.
Recently somebody said that my reasoning is atypical and that’s true, because of that I always give priority to business requirements and use case, never other way around.
Autonomous System: The IoT device we work with is a Smart Meter (https://en.wikipedia.org/wiki/Smart_meter), it records consumption of electric energy. We need to solve here 2 cases, get data from the smart meters and update them on when software is changed (continuous integration).
The update part is nothing new for us, we will use ansible(https://www.ansible.com/), docker registry and optimized docker image(https://www.docker.com/) that will run in raspberry Pi next to smart meter (raspberry Pi is reading historical data from smart meter through the serial port connection).
Registering of the service that will be deployed on raspberry Pi is not a complex process, but it is very curious for me because of its autonomous nature, the service only task is to get data from smart meter and send it to central system. Although before it does it, it must register itself.
Developer nature is full of imagination and creating over complicated solutions that will never find it’s way in business requirements, I tend to follow this path time to time, fortunately only in mind.
Smart meter has its protocol, protocol will be changed and updated over the time, it would be nice if our service could identify the smart meter product and gets it’s specification from main system, so we know how to serialize the data.
End: Autonomous design follow many other principles present in microservices, like loosely coupled and single responsibility principle, but among all of them the most important is to solve problem and be in problem domain.
And finally, protocols and communication, generic way to exchange information, be reactive on events that are in service problem domain, that’s the real key to autonomous system.