The EKK solution eliminates the undifferentiated heavy lifting of deploying, managing, and scaling your log aggregation and analytics solution.

With the EKK stack (Elasticsearch, Amazon Kinesis and Kibana), you can fully focus on analyzing logs, improving your application, instead of managing and scaling the system to aggregate the logs.

EKK Solution for Log Analytics Platform

In this article, we explain how to use EKK stack to monitoring logs generated by your application, usually a website.

Components of the EKK Solution

• Amazon Elasticsearch Service
  • Distributed search and analytics engine built on Apache Lucene
  • Send data as JSON via REST APIs
  • Data is indexed - all fields search-able, including nested JSON.
• Kinesis Firehose
  • Send data via other services or REST API
  • Data is buffered. Transformed via Lambda. Backed up to S3
  • Data is delivered to Amazon ES
• Kibana

Kinesis Firehose Data Transformation

• Firehose buffers up to 3 MB of ingested data
• When buffer is full, automatically invokes Lambda function, passing array of records to be processed
• Lambda function processes and returns array of transformed records, with status of each record
• Transformed records are saved to configured destination

Writing Data to Amazon Kinesis Data Streams