We are still only scraping the surface of what we have to learn and gain from the internet of things (IoT). Conversely, the IoT also has much to learn from other industries to make its vast task simpler. Collecting, processing, compressing, storing and retrieving data is an intrinsic part of the IoT, yet some industries, such as the financial technology (fintech) industry, have been in the business of carrying out those functions for years, even decades. I believe it is time for the IoT people to talk to and learn from the experience of those other industries.
Data Deluge For The IoT
Making the IoT work is not without its difficulties. After speaking with other IoT professionals, the general consensus is that the IoT is generating a lot more data than people expected it to back when they were putting the standards for it together. Base stations do not always cope with the load as well as they should, thanks to the deluge of information being thrown at them from a multitude of devices, whether they be within a smart home, a smart store or a smart city.
Signal collision and physical obstacles as seemingly simple as walls are leading to a need for more hardware. To fix this, more base stations are being installed to move signals cleanly through the network, driving both infrastructure and data processing costs. Meanwhile, each of those base stations is interconnected with every other one, all exchanging information back and forth, so the volume of data is multiplying far higher than the developers of the standards had bargained for — resulting in bandwidth jammed with surging packets of data.
All of these issues result in a never-ending need for more hardware, better software, higher licensing costs and, overall, a lot more money being spent to control the flood of data. Yet still IoT data processing solutions are not coping with the ever-increasing amount of data. So what can be done to help?
Fintech-Proven Approach
The data created by the IoT works in the same way that data in fintech has operated for years. Both industries’ types of data carry simple messages: a time stamp, an identifier and payload information. Therefore, for us in fintech, the nature of the IoT data package in terms of the size and type of the information being transmitted is very similar to what we are used to.
The processing of trading data happens in the same way that the IoT is attempting to do. From exchanges to market data providers, to brokers and then to traders, trading data is stored and managed in the same way. The delivery of financial market data is the basis of trading, investing and analysis. Today in fintech, we are processing about 12 million events per second, with millions of symbols and up to 15 data types (quotes, orders, trades, etc.), and yet this technology is working efficiently, in real time.
The data processing solutions used in fintech are field-proven and rigorously tested. These technologies represent a full technology stack — a data ecosystem that is used to collect, manage, normalize, store and distribute different types of data, and gives us an ability to work with historical data in a very efficient way. Here are the key data technologies that can be applied to the IoT:
• Data normalization: This is a translation of all the data received from different exchanges into a unified format (as every source uses its own format). It allows the user to apply unified methods of data management and not bother about the origin of data. It’s like translating the data into a metalanguage. The result of this normalization is that all events are delivered in a specific data scheme.
• Messaging system: This is a scalable data scheme allowing the distribution of any type of events and to ensure low latency and high throughput even in volatile market conditions. For example, during a mini-crash in the stock markets on February 5th, 2018, our clients didn’t have any downtime during the peak loads generated by harsh market movements. They continued their operation as usual, though the number of orders on their trading platform on that day was about 250,000 per hour (between 1 p.m. and 2 p.m.), with a total of about 1.4 million per day. The number of transactions in a single very large retail store generated during Black Friday is not even close to this figure.
• Conflation: This is the feature of the messaging and data distribution system that guarantees true real-time delivery of data. Even if the data feed has information microbursts, the data processing platform cleans out this feed to guarantee real-time streaming of key messages. At the same time, other “unnecessary” messages are not erased; all the events are saved in historical storage and can be retrieved when necessary. For example, a EUR/USD quote can change its price thousand times per second. While an average human eye cannot perceive more than 25 updates per second, it is enough to only see the key messages and send other messages straight to storage. A similar case in the IoT industry might be a temperature transmitter sending a hundred updates showing the same 0.1 degree, while for monitoring purpose five updates per minute are more than enough.
• Compressed data format (CDF) storage: This is used to effectively retrieve an amount of data in any file format or arrange the streaming of this data. In the IoT, this can be required to perform a thorough audit or back-testing of monitoring and management systems.
Proprietary technology to process billions of events already exists in the fintech industry. It is easy for companies that are building fintech solutions, with years of experience in collecting vital data from and for traders, to adapt the processing, transmission, compression, storage and retrieval ability of this data for the IoT. Even the volumes of data generated by the IoT are not enough to phase a fintech company; tens of gigabits of data per second is nothing new here. As such, I believe the IoT could gain a lot by tapping into Big Data experience from the fintech sector.
