How To Process Real-Time Data Analytics With Amazon Kinesis?

Traditionally data was dumped into some data warehouse to run analytics queries in batch mode. However, processing of big data analytics in real-time has become possible with the advent of latest streaming technologies.

What is Amazon Kinesis?

Amazon Kinesis is a cloud service that ingests data such as audio, video, server logs, IOT data and processes them in real-time or near real-time before putting it back in some backend database or storage layer. It is a cost-effective way to collect and analyze data in real-time to get some business insights.

Real-time

Amazon Kinesis enables ingesting data in real-time, so you can run analytics queries instantly without waiting for your data to accumulate to analyze in future.

Fully managed

With Amazon Kinesis, you do not have to maintain servers or run any software upgrades since it is a fully managed service provided by Amazon Cloud.

Scalable

Amazon Kinesis automatically scales up and down and can handle any amount of incoming data.

What are the Capabilities of Amazon Kinesis?

Kinesis Video Streams

Kinesis Video Streams is a service that can ingest videos from various sources in real-time and is integrated with various video analytics services to process them further.

Kinesis Data Streams

Amazon Kinesis Data Streams can ingest big data in real-time from multiple data sources.

Kinesis Data Firehose

Amazon Kinesis Data Firehose can ingest data in near real-time and can transform and store data in other AWS data stores. Moreover, it is well integrated with many business intelligence tools such as redshift for further analysis.

Kinesis Data Analytics

Amazon Kinesis Data Analytics is an analytical engine that can run SQL queries to perform analysis on top of real-time data. You do not need to learn any programming language to run analysis on top of data.

How Does Amazon Kinesis Data Analytics Work?

Data is ingested from various sources using Kinesis Data streams, Video Streams and Kinesis firehose and then data can be directed to Kinesis Data Analytics to run SQL queries to perform some analysis and then results could further be sent to other AWS services for further processing and storage.

For example, processed data can be sent to the AWS Lambda function to take some further action, can be moved to dynamoDB for storage or can be moved to redshift for further analysis.

Kinesis Video Streams

Kinesis Video streams can stream data from millions of connected devices and can playback recorded/live videos on demand. It is well integrated with many aws services such as Amazon Rekognition to perform video analytics.

You have to install Kinesis Video streams SDK on the device to start sending videos to Kinesis video streams for further processing. It uses Amazon S3 as the underlying data store and allows you to search your stored videos based on deviceId and timestamp.

It provides data security by automatically encrypting data at rest and data in transit. It is a fully managed service and you do not have to worry about infrastructure.

Use cases

• Kinesis Video streams can be used to build many IOT based applications such as security control, weather control, lighting control and many more.
• You can analyze body language and can perform facial analysis on live videos using Amazon Rekognition Video and Amazon Kinesis Video Streams

Kinesis Data Stream

Amazon Kinesis Data Streams is a real-time data streaming service that can ingest from petabytes to gigabytes of data in one second from multiple data sources such as IOT devices, Twitter, server logs and can be integrated with Amazon Data analytics for further analysis.

Use cases

• Many known companies for their real-time pipelines use Kinesis Data Stream such as Netflix uses Kinesis to enable high availability by detecting application failures & issues.
• Thomson Reuters uses Amazon Kinesis to gain insights from data generated by its offerings and continuously improve the user experience.

Kinesis Data FireHose

Amazon Kinesis Data Firehose can ingest data from various sources and can load that data into Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk. It is a near real-time, fully managed, scalable service that buffer data for almost 60 seconds before moving it further.

In Amazon Kinesis Data Firehose, you have to pay only for what you use and there is no setup cost involved. It prepares your streaming data by converting raw streaming data into formats required by your destination data stores.

Use cases

• IoT Analytics

  You can capture data from various IOT devices such as temperature sensors, your home devices, mobile devices and send that data to Kinesis firehose, which will then perform some transformation before loading to various data stores such as Amazon S3, Redshift to get real time metrics and insights.

• Log Analytics

  Server log data can be directed to Kinesis firehose to detect anomalies and errors and to trigger alerts in real time. It can integrate with ElasticSearch Service to visualize data with Kibana in real time.

Kinesis Data Analytics

With Kinesis Data Analytics, you can run SQL queries on top of streaming data to gain business insights and to build streaming applications with other AWS services. It also provides java SDK to build java application to transform and analyze data on the go.

It is fully managed and pay as you use service that can scale automatically to match your needs. It integrates with other AWS services such as Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Amazon Elasticsearch Service, Amazon S3, and Amazon DynamoDB.

Use cases

• Many companies are using Kinesis data analytics for their usecases such as Yieldmo company uses Amazon Kinesis to ingest and process data in real time.
• It uses Amazon Kinesis Data Analytics to find user interaction with their ads.
• It also uses Amazon Kinesis Data Firehose to load data to Amazon Simple Storage Service (Amazon S3) buckets. Finally, data is loaded to an in-house data warehouse for further analysis and visualization.

Conclusion

There are other streaming frameworks in the market such as Apache Spark, Flink , Storm but Amazon Kinesis holds its special place because of its serverless architecture and pay as you use model. Amazon Kinesis is a powerful tool that can be used for ingesting real time data warehousing and its integration with other AWS services make it even more powerful and useful.