“A black alarm clock on a chair” by Szűcs László on Unsplash
Imagine a world without alarm clocks. You'd go to bed knowing that you cannot be late for a 9 o'clock meeting tomorrow. But how do you make sure that you get good a night's sleep without the risk of oversleeping?
One possibility is to ask or pay someone else to wake you up. That way, they have would have to worry about not oversleeping. So, they might not sleep at all. Wasting human time and energy on such a common and mindless task is not justified.
What people did instead was come up with a mechanism to get notified when to wake up. And this mechanism doesn't include a human in the loop. The solution in this case —the alarm clock — is so obvious that we rarely think about it.
Yet, in everyday life, we often use a different mechanism instead of notifications: polling. That is when we do periodic checks to see if a state has changed. Say that you would like to buy fresh fish. You go to 3 local stores, but they only have frozen fish. You decide to wait until they get some fresh fish, so you come there every morning. That is an example of polling 101!
If you decide to spare your time and energy, you might ask the friendly staff to save your contact number and give you a call when fresh fish arrives. But you'd be an exception, and that is the only reason why the staff would be able to do you that favor. Imagine that other people take advantage of the staff'’s good will — soon their good will wouldn't be able to scale enough to notify all interested people. They'd need to remember who was interested in what, find a way how to notify them quickly…soon they would either give up, or start to build their own version of a pen-and-paper information system.
You might say that this is an unrealistic example. But it illustrates the point. How many times have you found yourself checking up on something that you wished to be notified about? The question then becomes: can we do better? Can we subscribe to changes that interest us? Changes on the web, but also in the physical world. Isn't that what IoT is about? Some companies already use systems that detect patterns in their events, but we need to make this available to consumers also, at the press of a button. Before we see how that can be achieved, let's first examine how we subscribe today.
Subscriptions today
What can we subscribe to nowadays? And when I say subscribe, I’m not talking about a monthly Netflix subscription — I’m talking about a subscription to notifications of certain events of interest happen.
First, we've got time-based subscriptions. Our alarm clock example is in this category. The event it tells us about is that now is the time that we were interested in. A stopwatch is a slightly different, as it tells us that a certain amount of time has passed instead. But it belongs to the same category.
Another type of subscriptions are those that are implicit in systems with closed feedback loops, like your air-conditioner. Not so long ago we had to pause and increase/decrease the power of such systems manually just to achieve and maintain the desired conditions. Today, because these systems monitor events about change of temperature through sensors, you can just find your preferred temperature once, and stop obsessing about it.
There is one more thing to notice in the AC example: we are not simply notified when the event has happened (i.e. room temperature has gone below or above desired temperature), but instead, an action is taken by the system.
Event-condition-action is a common structure in event driven architecture. Event is a change of state, a signal generated by an event source, or an event generator. Condition evaluates the change, and determines whether an action should be taken or not. The combination of an event and a condition can be observed as a new event source itself. Action can be "send an SMS message or notification", or "adjust the vents". For the sake of simplicity, only notifications will be considered as actions in this article. I urge you to consider implications of the system that I am going to describe if it could not only ingest events and push notifications, but also send instructions to devices within the IoT ecosystem and other web services.
Finally, the third form of subscriptions that we have today is a subscription to content. In the physical world that's newspapers and magazines, while in the digital world the range of subscriptions is much wider, but still limited to content. We are notified about the same type of event, which is that new content has been published — new text, new video, new photo. Truth be told, most of this content is selected for us by recommendation systems such as those behind Instagram or Facebook, but that is orthogonal to the question whether we can be notified about changes in the physical and digital world. In case of subscriptions, we choose what to subscribe to, while in case of recommendation systems, it is the algorithm that chooses for us, based on our previous interactions.
Democratization of publishing
It is too hard for a digital business to implement publish-subscribe functionality into their website. That is why we never see it. For local businesses, however, this task is practically impossible today. One of the enabler technologies for publishers was RSS. It was adopted as a sort of standard, and provided common language for publishers and subscribers. For digital businesses, it is relatively easy to publish content through RSS. But that is it. It is limited to content.
There is no processing in RSS. To put it in a simple way: you can not specify conditions that you want to subscribe to. Choice is limited. And it's for already digital businesses only.
There are of course custom implementations of publish-subscribe functionality, that businesses with adequate investment in IT can provide. These lack standard, so it is hard to build systems on top of it. And for consumers, there is no common place to manage subscriptions.
The final problem is that these events are in silos. What subscriptions provide is access to aggregated information. And you cannot detect a variety of patterns across different sources if you only have access to aggregated information.
These problems pose some interesting challenges. To solve them, we must imagine a platform that would support these requirements. But we also have to imagine an ecosystem where all necessary participants have incentive to use the platform. Note that the platform design deeply impacts the ecosystem, and that the ecosystem dictates the platform's functionalities.
Platform
Publishers shouldn't have to set-up an infrastructure for processing or storing events. The cloud should be leveraged for this. Of course, other ideas can be discussed, like blockchain (i.e. Ethereum or IOTA), and that is certainly an interesting direction, and a topic for one of the next articles.
So let's consider cloud platform as an event processor. All events, from various event sources from different publishers would flow to this platform. This inherently solves the “silos” problem from above.
This cloud platform would need to process events to detect multiple patterns at the same time. To achieve this, it needs to perform distributed computation that scales horizontally, and dynamically, according to the current load.
From the client side, a small library for publishers with simple API is what is required. It needs to be possible to integrate it easily within web applications, i.e. Javascript, as well as within embedded systems, i.e. Raspberry Pi. The latter would be used to provide local businesses publishing capabilities and presence in the digital world.
If local businesses are able to buy cheap devices that transmit data from sensors to the platform, then we would know about things from our environment, through the _internet —_hence, internet of things. Well, at least part of it and its promises. The other part, autonomous interaction between things, is not the focus of this article. We need to sense the world first, and not just in a smart city, or smart home fashion. The internet is not jut a city or home. It is the world.
Ecosystem
You might wonder: how do we make businesses release their data to the public. The answer is: we create an ecosystem.
It would consist of 3 roles:
- Publishers
- Developers
- Subscribers
We've described publishers enough so far. They are usually businesses who own the data. If they wish, they can use the cloud platform for their own needs, without making data public, and act as all 3 roles at the same time. Easy functionality without much effort. Alternatively, they can sell their data or give away data that they don't need for competitive advantage for free.
Developers implement pattern detection algorithms that the cloud platform runs to process the events. Cloud platform provides a framework in which algorithms can be implemented. The platform acts as a multi-tenant host, where each tenant is one pattern-detection algorithm. Developers are the owners of an algorithm, and can decide whether they would like to sell it or not. It is very similar to app stores on Android, Windows, or iOS.
Subscribers register their subscriptions to one or more pattern-detection algorithms. Detection of a pattern in events is what triggers a notification to subscribers.
The cloud platform provides more than one interface for subscribers.
For tech-savvy users and other software companies, it provides a library to integrate subscriptions into their services. In that way, 3rd party software companies can perform various actions when events occur — for example, they can use this library to provide a service to book a flight when a low-cost offer is detected.
For other users, which are the vast majority of users, there is a web portal. It provides a friendly user interface that displays available subscriptions and their parameters, and lets users configure and manage them. Comparing to the example above, instead of a flight being booked for you, you would just be notified when a low-cost offer for a flight appears. Some of us prefer to be more in control of what happens.
Event schema and ontology
In order for the algorithms to work, they must match the events' schema. But with different publishers, schemas can be different. One way in which they can differ is by naming the same property differently. Another way in which they can differ is by content, some publishers publish more details, while others publish less.
The first kind of difference can only be solved through standardization. Ontologies are a perfect solution here, as they provide a common vocabulary. However, it is not realistic to expect everyone to use the same ontology. Hence, a more probable outcome would be that this is driven by popularity. If a source is popular, developers are going to accommodate their algorithms to that source. And vice-versa. Schemas and algorithms evolve, and it is to be acknowledged that backward compatibility of the system is very important.
The second kind of difference is absorbed by dynamic schema detection. What is there is read, what is not is not taken into consideration.
Conclusion
This platform is not about performance. It is not meant to be used for algorithmic trading. It is about integration. The platform should provide a holistic picture of our environments, and not the one partitioned by silos. And it should allow us to build applications that connect us with things that matter to us, both in the digital and physical world.
There is so much more that I researched in detail than what is the scope of this article. Some of that will be topic for next articles. The level of detail and technical vocabulary is probably going to stay the same. If you'd like to dive a bit deeper into the subject, I would be happy to discuss it with you.