Building IOT platform, Technical challenges
Posted by Jai on January 2, 2017
As discussed in the last post regarding the typical characteristics of IOT platforms, there are different categories of available IOT platform these days and typical feature and functional support provided by each. Device connectivity, device agnostic, domain agnostic and extensibility etc. drive to come up with few required technical aspects of such a platform. In this post, we will discuss different technical aspects to be taken into account to build such IOT platform.
Typical characteristics of any IOT platform are as covered in last post.
In general, we can add below typical technical aspects in the architecture of such IOT platform
- Device Connectivity
- Device repository and management
- Data Processing of Events/Rules Management
- Alerting, Monitoring & Notification support
- Complex Event Processing support
- Scalable & Extensible Data storage layer
- Real time Data access support
- Data Analytics
- Data Visualisation support
- External Ecosystem system Integration support
- IOT Security, Across all layers
- Development/Testing Support
- Testing/POC/Time to Market support
- Device Bi directional support
- Device Firmware management support
- Data reporting/export/publish/share support
1. Device Connectivity
Majority of big platform have been setup with device connectivity for IOT devices. The things that can connect with platform, irrespective of which domain these are used with.
Either IOT devices itself have GSM/GPRS connectivity or through device/field Gateway to connect to the IOT cloud platforms. Based on the data requirements of the IOT devices domains, cellular modules (2G,3G, 4G etc.) are used. One of the big challenge in feature selection for these devices is to be able to bundle all features within the device within the cost boundaries.
There are variety of embedded operating system available for IOT devices, some are proprietary and some are open source OS solutions are available. We will be covering the security aspects around OS and firmware in details in separate post. From device memory and size perspective, the connectivity selection is indeed quite important for the devices. Which protocol to use, what data transfer size strategy apply for the devices etc. are quite important.
From IOT platform perspective, to be able to support a range of devices and their communication protocols is necessary. For a domain, device and feature specific communication, you can have different protocols but to be able to support devices across range you need to implement and support most of the commonly used protocols on the platform side.
RAW TCP/IP or UDP:
Some of existing device vendors still use raw tcp/ip or udp connection for data transfer. Many of the iot platforms provide platform gateways solution to convert raw connection data to platform standard supported protocols data transformation. Security support is also one of key aspect to look into while using such gateways.
Device management and data consumption end points exposed through REST end points is required for devices directly using HTTP protocol. The data size because of additional header information is overhead but with the reduce cost of data transfer, it is easy option as communication layer.
The publisher-subscriber model of the protocol gives additional advantage for the protocol. The reduced data transfer size also gives additional benefits for data intensive devices.
There are additional protocols available for different devices, the additional support is required to make the solution more generic and cover larger set of devices.
Read Understanding The Protocols Behind The Internet Of Things for further details.
Each of the supported protocol has its own pros and cons to be used with the devices. Some of the key factors to decide on are indeed,
- Transport Protocol to use, TCP or UDP, acceptable data loss or not
- Data transfer size , additional header info size
- Bi directional communication support implementation
- Transport layer security integration, latest TLS/DTLS/SSL support
- Quality of service, packet acknowledgment process.
- Messaging model: Request/Response or Publish/Subscribe
To implement data ingestion services on the platform side, should take into account all available data input points. The data is ingested in either form but meeting all below requirements,
- No data loss of events or allowed
- Scalable to meet devices number scalability
- No latency in data ingestion
Devices Network Layer:
One of the typical use cases is the industrial devices which needs field gateways to connect to platform. One of the important aspect is to choose well the devices network communication technologies. There are varieties of communication technologies available for same,
Read through protocols for better understanding.
Telecom Network Layer:
Based on data utilisation requirements, different devices choose different ways to connect to platform. Either directly using Cellular communication from device or using device gateways having cellular access is used to connect to the platform.
Different security aspects needs to be taken into account around telecom network, which we will cover in details in next post.
2. Device repository and management
Device repository is key feature for any iot platform. To be able to upload, test, manage, remove etc. are basic key features support required from the platform.
The process to be able to enrol a device into the system should be supported by the platform. Different manufacturing and testing procedure adopt different ways to enrol devices into the system. Platform additional support to map provisioning requirements along with security aspects is very much required.
Platform needs to provide authentication support for the devices, to guarantee that only genuine devices are connected .
Per device authentication support is also key features to uniquely authenticate a device. We will be covering authentication and security aspects in details in separate post.
At times devices needs some factory settings and also some default settings. Platform should provide provisions to manage device configurations on provisioning or at later stage while in use.
For any solution, you may have different types of devices and would like to group those based on your business domains aspects or some runtime attributes of the devices. Feature to be able to group and filter devices based on such properties really add value to regular operations and business process.
For regular operations and debugging perspective monitoring of devices is very much required. Platform support for regular monitoring, debugging and diagnostics capabilities adds added value to same.
Read through Fundamentals of IoT device management.
Some of features like bi directional communication, firmware over the air upgrade and domain specific attributes for device management etc. are additional features.
3. Data Processing of Events/Rules Management
The fast moving pace of billions of IOT devices and huge amount of data transferred, requires relevant data ingestion and processing capabilities at the platform side. Selecting the right ingestion and streaming solution for devices is indeed quite important.
Typical for iot solution, the platform provides and supports the Lambda Architecture. Its ability to handle streaming data, store it, servicing data support are required.
Data Collectors/Message brokers and message queueing:
There are lot of data collectors available to be able to collect data from devices and share further with any of the steaming solutions. Based on your architecture requirements, you can break and design the way it suits the domain better. From platform perspective, data collectors should be available to devices for data ingestions and then streaming solution to process same further.
Providing MQTT implementations to digest data. Both are typical and well proven message queuing system.
Kafka is streaming solution based on Publish-subscribe model for streaming data.
General data collector integration points available.
Based on the requirements, many platform also directly use data collectors to support streaming data. Other streams options Spark/Storm etc. also used as streaming solutions. Platform should provide streaming capability to handle data and events in streams way.
Many cloud platform also provide different streaming solution, which are worth exploring further.
Data servicing layer should be able to query and provide relevant information for query. Based on the requirements, NoSQL layer is also used for query.
For query huge amount of data, typical Map/Reduce and Spark are used to query on large dataset.
The challenge for any platform side, is to figure out which one to use, and how to go for selection. For each data collection and streaming solution what is the Quality of service for each technology component, what is the data processing speed implications and the overall reliability.
4. Alerting, Monitoring & Notification support
For any iot solution, to be able to process data and generate alert in cases, to be able to monitor its status and notify users is also one of common features.
To be able to both filter devices based on attributes and data properties and alert system is required. In built alerting system and ability to integrate with external alerting system adds added value to the platform capability.
Ability to monitor devices state, devices data and end to end data processing components in the system is very much required from maintenance and operations perspective. Platform ability to provide such monitoring system itself and integrate with external monitoring systems also is required.
The more the notifications options available, the more flexible the platform capability are. Email, SMS, push notifications etc. capability are indeed required.
5. Complex Event Processing support
CEP a technique to process real time events and based on business intelligence extract relevant information, take relevant decision and process same further. One of the direct use cases of CEP is artificial intelligence (AI).
Platform ability to provide CEP enables lot more features for the IOT industry. Its ability to filter relevant information, fill in the gaps for missing information, and use predictive analysis to take right decisions.
6. Scalable & Extensible Data storage layer
Most of the big systems getting developed today required multi data storage systems approach. Platform should also provide different storage layer possibility to be used with the solution.
Relational database management system to be able to manage relational data.
In memory storage:
To reduce latency and be able to serve data under servicing later the in memory storage layer availability is very important.
To store devices streaming data, NoSQL solutions should be available.
For analytics, both streaming and batch processing the platform should provide.
Data ware house availability for Business Intelligence is definitely added value.
Each data storage layer is added and defined based on various requirements,
- Handlong relational data for IOT devices etc.
- Huge amount of unstructured data
- Fast data storage, with low latency to serve
- Long term data storage
- Query on huge data
7. Real time Data access support
To have real time information be available from device and minimise the server side heavy processing and disk storage overhead, the platform should provide features to make data available to view/process as soon as possible. Platform ability to process real time streaming information and make data available out of in memory caching systems is required to make this feature available.
Some of the industry used In memory caching system for key value pair data, are well known. Support for such system is very much required.
8. Data Analytics
For any IOT device the current and real time value addition is by its feature. But the real value addition for any IOT solution lies with the data. Making sense and turning data into currency is going to add more value in long term.
The platform should be having handling big data solutions for analytics purpose. Typical Hadoop stack support for data analysis is also used.
Real time analytics:
To be able to process stream data and generate real time analytics information out of it. For example, Spark solution to handle device data and generate or update analytics report on same. Platform should provide customer real time feedback of events.
For predictive analysis huge amount of training data is required. And all iot devices submitting and storing huge amount of data every second. Platform should provide capability to store and manage this huge amount of data. It should have ability to process this data in batch way for predictive modelling and other reporting purpose.
Based on devices real time data and batch processing, platform should provide system which is able to analyse both and take decisions accordingly to meet business features. Support for machine learning library and algorithm is very much required from analytics platform.
For each domain specific IOT solution, some of the features requirements are,
- Predictive Analysis, based on available information and using predictive modelling be able to suggest analysis of the issue/feature
- Diagnostics, analyse data and find real problem context
- Prognostics, analyse data and indicate future events.
From practical purposes, to evaluate which data makes sense and what value that drives is challenge as industry still evolving and analysing same.
9. Data Visualisation support
Any iot platform will be storing and managing huge amount of data. It’s capability to manage this data is very much required. Data visualisation is also one of core feature of such platform, which enables customers to read real value out of data.
Ability to view data and generate reports on same.
Ability to dynamically use data filters and generate relevant data.
Ability to easily integrate with BI tools to visualise current data and also be able to generate new reports on same.
Publish and Share:
Ability to publish and share the reports and manage its business process.
10. External Ecosystem system Integration support
Not many iot solutions will work in isolation, there is always external system integrations required for a complete solution. Ability to extend the platform in line with external systems of an iot solution ecosystem is very much required.
Integration with external parties to provide complete ecosystem for same. Based on domain integrating with external systems is very much the requirement from a complete iot solution.
Domain specific Features:
Ability to have default system to match domain requirements. For example, system handling geo-spatial data and services around locations data.
CRM System integration:
Platform ability to integrate external crm system.
Real time and detailed and historical information available to customer support.
For example, for Smart cars and self driving cars lot of opportunity around to build the ecosystem,
- Navigational, routing
- Vehicle maintenance
- Road side assistance
- Fleet management
- Incident support
- Vehicle to Vehicle (V2V) communication
- Vehicle to Infrastructure (V2I) integration
- Government rules/policy system integration
- Predictive destinations
- Travel preferences, eating and entertainment places
- Social media integration with friends and contacts
- etc. etc.
11. IOT Security, Across all layers
IOT Platform security model is one of the most critical part of the platform. Also one of the biggest challenges and concern around industry related to devices is the security around it. We will touch base high level security aspects from platform perspective here, and will cover in details in next post.
Following at hardware firmware level needs to be considered,
- Device hardware physical security considerations
- OS and Firmware security consideration
- Device gateway security considerations
- Device gateway communication layer considerations
- Device data credentials/keys storage
- Device data encryption
Transport Layer Security:
The data transfer channel over telecom or internet needs to be encrypted. The cloud platform should support TLS/DTLS/SSL features to enable the communication channel to implement same.
Cloud Physical Security:
Physical security is also important for the platform services. Relevant ISO certifications for physical security should be aligned to IOT platform accordingly.
IOT platform Application security:
IOT platform backend service should support both authentication and authorisation approaches. Fine grained security model for user credentials, roles and IAM support should be there.
We will be discussion different authentication and authorisation approaches from security aspects in next post.
Cloud Data Security:
Data access to read/write and manage policy from applications should be implemented.
Cloud Data privacy:
12. Development/Testing Support
The platform should provide development and testing support for the IOT solutions.
During the development of the solution, platform should provide,
Programming languages support:
C/C++, Java, Python etc.
Platform integration SDK’s in different languages for easy integration with platform.
In case, not all protocols supported intermediate transformation gateway to integrate with platform. eg. raw tcp/ip port integrations.
Development tooling system:
Easy integration of development tooling system support is also helpful.
SDK/API support for clients/apps:
Many platforms also provide SDK’s for mobile applications to consume data and easy integration with mobile applications.
API support to retrieve device data and display/visualise same.
Testing support for dummy devices and testing scenarios should be supported by the platform. Dedicated testing environment or api is also provided as development phase support.
13. POC/Time to Market support
To make it happen for the consumers, the platform should provide easily scalable environment meeting different business approaches,
POC/Proof of Concept:
To be able to easily show case the iot solution using the platform is minimum requirement from the platform side. Proof of concept supports the solution to boom from idea to solution very easily.
Time to market:
Any iot solution will take time to mature and scale accordingly. Be able to take the concept to market very quickly and easily should be supported by the platform. Ability to scale the solution, enable business to focus on business only and platform should enable it.
14. Device Bi directional communication support
Every IOT platform needs to provide two way communication between the device and platform.
Device to Cloud:
As discussed earlier, the data ingestion is data transferred from device to cloud. Different capability as per data protocol is followed. Dedicated rest end points for REST solution, separate topic for MQTT etc. are required.
Cloud to Device:
In many instances you need to send commands to devices also. Platform should provide the capability to do so and be able to send information from cloud to device also. For example,
In case of HTTP dedicate rest end point on which device and listen and pull information regularly is any new information is available. This is pull model, device to pull command information from cloud.
device subscribing to dedicated topic to receive information fromm cloud. This is publisher/subscriber model where device remain connected to the broker.
Ability to provide a mechanism for cloud-to-device command status. Once the device accepts the command, it does not always mean that the command was executing successfully. Platform needs to provide a way where device the submit back the status of the command received and its execution status, so that in case of failure it can be reschduled or ignored based on business scenarios.
Ability to trigger commands from cloud to device scheduler based. For example, the firmware updates once devices not in use etc. which provide client better management of maintenance window.
In certain cases, if there is no way to communicate with device or a scheduled window passes or number of tries passes for the cloud-to-device command, the retrial limit and alerting options should be provided by the platform.
Many of the devices status are also triggered based and those remain the silent mode. For centain case, if the devices having GSM capability, Platform ability to be able to send SMS message to device to wake up and do certain operations should be there. This is push model from cloud to device, once device receives information then it triggers to pull information from cloud.
15. Device Firmware management support
Device maintenance and upgrade from both functional and security is very important aspect of IOT solution. The platform needs to provide Over the air upgrade solution to the devices.
Regular support for firmware repository management. To be able to host different firmaware images and manage same.
Ability to manage devices and configure immediate and scheduled firmware updates.
Firmware updates status:
Ability to monitor, control and notify the status of firmware updates.
Delta OTA Updates:
Many devices are capable of handling the delta updates/patches for the firmware. The platform and device both needs to support same. Platform ability to manage and control delta updates should be there.
Being one of the critical part of the IOT solution, to be able to patch devices, the platform should provide secure means of firmware upgrades.
We will covering security aspects around FOTA in later posts.
To build a robust and secure over the air upgrade on platform side is quite challenging.?
16. Data reporting/export/publish/share support
One of the biggest advantage of cloud platforms is to use existing PAAS services to integrate with the IOT solution. The platform needs to have capability to share relevant data with other sources. Some of the practical examples are,
Export data to data warehouse solution to be able to generate reports on same, if required.
Ability to push devices data to either client location or third party services to use data for further purposes.
Ability to export scheduler based data export in different format and share across with other systems.
Hope this provides some details of challenges and aspects we need to take into account. In later posts, we will cover some of these aspects in further deep technical details.