AWS Solution Architect(Associate) - Topic 8: Applications

Customers are using AWS high level services(e.g. Amazon Kinesis)to collect, process, and analyze real-time data. In this way, they can react quickly to new information from their business, their infrastructure, or their customers.

For example, Epic Games ingests more than 1.5 million game events per second for its popular online game, Fortnite.

Applications

SQS (Amazon Simple Queue Service)

Amazon SQS is a web services that gives you access to a message queue that can be used to store messages while waiting for a computer to process them. So it is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

So, you should notice the word “Decouple” if you ever see it in the exam.

• SQS is pull-based, not pushed-based
• Messages are 256 KB in size.
• Messages can be kept in the queue from 1 minute to 14 days; the default retention period is 4 days.
• SQS guarantees that you messages will be processed at lease once (detail explained below: visibility).
• Amazon SQS long polling is a way to retrieve messages from your Amazon SQS queues. While the regular short polling returns immediately (even if the message queue being polled is empty), long polling doesn’t return a response until a message arrives in he message queue, or the long poll times out. So, please bear in mind that using long pulling to reduce the cost because long polling essentially won;t turn a response until a message arrives.

Using Amazon SQS with other AWS infrastructure web services

Amazon SQS message queuing can be used with other AWS Services such as Redshift, DynamoDB, RDS, EC2, ECS, Lambda, and S3, to make distributed applications more scalable and reliable. Below are some common design patterns:

• Work Queues: Decouple components of a distributed application that may not all process the same amount of work simultaneously.
• Buffer and Batch Operations: Add scalability and reliability to your architecture, and smooth out temporary volume spikes without losing messages or increasing latency.
• Request Offloading: Move slow operations off of interactive request paths by enqueing the request.
• Fanout: Combine SQS with Simple Notification Service (SNS) to send identical copies of a message to multiple queues in parallel.
• Priority: Use separate queues to provide prioritization of work.
• Scalability: Because message queues decouple your processes, it’s easy to scale up the send or receive rate of messages - simply add another process.
• Resiliency: When part of your system fails, it doesn’t need to take the entire system down. Message queues decouple components of your system, so if a process that is reading messages from the queue fails, messages can still be added to the queue to be processed when the system recovers.

Queue types

Amazon SQS offers two queue types for different application requirements:

Standard Queues	FIFO Queues
Unlimited Throughput: Standard queues support a nearly unlimited number of transactions per second (TPS) per API action.	High Throughput: By default, FIFO queues support up to 300 messages per second. When you batch 10 messages per operation (maximum), FIFO queues can support up to 3,000 messages per second.
At-Least-Once Delivery: A message is delivered at least once, but occasionally more than one copy of a message is delivered.	Exactly-Once Processing: A message is delivered once and remains available until a consumer processes and deletes it. Duplicates aren’t introduced into the queue.
Best-Effort Ordering: Occasionally, messages might be delivered in an order different from which they were sent.	First-In-First-Out Delivery: The order in which messages are sent and received is strictly preserved (i.e. First-In-First-Out).

Standard Queues

You can use standard message queues in many scenarios, as long as your application can process messages that arrive more than once and out of order, for example:

• Decouple live user requests from intensive background work: Let users upload media while resizing or encoding it.
• Allocate tasks to multiple worker nodes: Process a high number of credit card validation requests.
• Batch messages for future processing: Schedule multiple entries to be added to a database.

FIFO Queues

FIFO queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated, for example:

FIFO queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated, for example:

• Ensure that user-entered commands are executed in the right order.
• Display the correct product price by sending price modifications in the right order.
• Prevent a student from enrolling in a course before registering for an account.

SQS Visibility

• Visibility timeout is the amount of time that the message is invisible in the SQS queue after a reader picks up that message will then be processed before the visibility timeout expired, the message will then be deleted from the queue. If the job is not processed within that time, the message will become visible again and another reader will process it. This could result in the same message being delivered twice.
• Visibility timeout maximum is 12 hours.
• It’s a very popular exam question that asking about getting the same message being delivered twice and what could be the cause of it.

---

SWF (Simple Workflow Service)

The Amazon Simple Workflow Service (Amazon SWF) makes it easy to build applications that coordinate work across distributed components.

In Amazon SWF, a task represents a logical unit of work that is performed by a component of your application. Coordinating tasks across the application involves managing intertask dependencies, scheduling, and concurrency in accordance with the logical flow of the application.

SWF vs SQS

• SQS has a retention period of up to 14 days; with SWF, workflow executions can last up to 1 year
• Amazon SWF presents a task-oriented API, whereas Amazon SQS offers a message-oriented API.
• Amazon SWF ensures that a task is assigned only once and is never duplicated. With Amazon SQS, you need to handle duplicated messages and may also need to ensure that a message is processed only once.
• Amazon SWF keeps track of all the tasks and events in a application. With Amazon SQS, you need to implement your own application-level tracking, especially if your application uses multiple queues.

SWF Actors

• Workflow Starters — An application that can initiate (start) a workflow. Could be your e-commerce website following the placement of an order, or a mobile app searching for bus times.
• Deciders — Control the flow of activity tasks in a workflow execution. If something has finished (or failed) in a workflow, a Decider decides what to do next.
• Activity Workers — Carry out the activity tasks.

---

SNS (Simple Notification Service)

Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.

SNS allows you to group multiple recipients using topics, A topic is an ‘access point’ for allowing recipients to dynamically subscribe for identical copies of the same notification.

One topic can support deliveries to multiple endpoint types — for example, you can group together iOS, Android and SMS recipients. When you publish once to a topic, SNS delivers appropriately formatted copies of your message to each subscriber.

SNS Availability

To prevent messages from being lost, all messages published to Amazon SNS are stored redundantly across multiple availability zones.

SNS Benefits

• Instantaneous, push-bases delivery (no polling)
• Simple APIs And easy integration with applications
• Flexible message delivery over transport protocols
• Inexpensive , pay-as-you-go model with no up-front costs
• Web-based AWS Management Console offers the simplicity of a point-and-check interface

SNS vs SQS

Amazon really like to quiz you the differences between SWF & SQS and then SNS & SQS.

• Both Messaging Services in AWS
• SNS - Push
• SQS - Polls (Pulls)

---

Elastic Transcoder

Amazon Elastic Transcoder manages all aspects of the media transcoding process for you transparently and automatically. There’s no need to administer software, scale hardware, tune performance, or otherwise manage transcoding infrastructure.

• Media Transcoder in the cloud
• Convert media files from their original source format in to different formats that will play on smart phones, tablets, PCs, etc.
• Provides transcoding presets for popular output formats, which means that you don;t need to guess about which settings work best on particular devices.
• Pay based on the minutes that you transcode and the resolution at which you transcode.

---

API Gateway

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. APIs act as the “front door” for applications to access data, business logic, or functionality from your backend services.

Using API Gateway, you can create RESTful APIs and WebSocket APIs that enable real-time two-way communication applications. API Gateway supports containerized and serverless workloads, as well as web applications.

• Remember what API Gateway is at a high level: it’s a door to your environment
• API Gateway has caching capabilities to increase performance
• API Gateway is low cost and scales automatically
• You can throttle API Gateway to prevent attacks
• You can log results to CloudWatch
• If you are using JavaScript/AJAX that uses multiple domains with API Gateway, ensure that you have enables CORS on API Gateway.
• CORS is enforced by the client, so it is enforced by your browser

What Can API Gateway Do?

• Expose HTTPS endpoints to define a RESTful API
• Serverlessly connect to services like Lambda & DynamoDB
• Send each API endpoint to a different target
• Run efficiently with low cost
• Scale effortlessly
• Track and control usage by API key
• Throttle requests to prevent attacks
• Connect to CloudWatch to log all requests for monitoring
• Maintain multiple versions of your API

How DO I Configure API Gateway?

• Define Resources and nested Resources (URL Paths)
• For each Resources:
  • Select supported HTTP methods (verbs)
  • Set security
  • Choose target (such as EC2, Lambda, DynamoDB, etc.)
  • Set request and response transformation

How Do I Deploy API Gateway?

• Deploy API to a stage

  • Uses API Gateway domain, by default
  • Can use custom domain
  • Support AWS Certificate Manager: free SSL/TLS certs.

API Gateway Caching

You can add caching to API calls by provisioning an API Gateway cache and specifying its size in gigabytes. The cache is provisioned for a specific stage of your APIs. This improves performance and reduces the traffic sent to your back end. Cache settings allow you to control the way the cache key is built and the time-to-live (TTL) of the data stored for each method. API Gateway also exposes management APIs that help you invalidate the cache for each stage. Caching is available for REST APIs in API Gateway.

Same Origin Policy

In computing, the same-origin policy is an important concept in the web application security model. Under the policy, a web browser permits scripts contained in a first web page to access data in a second web page, but only if both web pages have the same origin.

This is done to prevent Cross-Site Scripting (XSS) attacking.

• Enforced by web browers
• Ignored by tools like PostMan and curl.

CORS is one way the server at the other end (not the client code in the browser) can relax the same-origin policy.

Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources (e.g. fonts) on a web page to be requested from another domain outside the domain from which the first resource was served.

• Browser makes an HTTP OPTIONS call for a URL (OPTION is an HTTP method like GET, PUT, and POST)
• Server returns a response that says:
  • “These other domains are approved to GET this URL”
• Error:
  • “Origin policy cannot be read at the remote resource?”
  • You need to enable CORS on API Gateway.
  • Going into the exam, if you see something where it’s talking about origin policy cannot be read at the remote resource. It means the cause is not enabled on your API Gateway and API Gateway is not able to go and request that information from the other side.

---

Kinesis

Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application.

3 Different Types of Kinesis

• Amazon Kinesis Data Streams
• Amazon Kinesis Data Firehose
• Amazon Kinesis Data Analytics

Kinesis Data Streams

If you see shards come up in your exam, think straight away of Kinesis streams because Kinesis is the only form of Kinesis that has shards.

Review the developer guide

Kinesis Streams Consists Of Shards:

• 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second.
• The data capacity of your stream is a function of the number of shards that you specify for the stream. The total capacity of the stream is the sum of the capacities of its shards.

---

Web Identity Federation - Cognito

Amazon Cognito lets you add user sign-up, sign-in, and access control to your web and mobile apps quickly and easily. Amazon Cognito scales to millions of users and supports sign-in with social identity providers, such as Apple, Facebook, Google, and Amazon, and enterprise identity providers via SAML 2.0 and OpenID Connect.

Cognito is an Identity Broker which handles interaction between your applications and the Web ID provider.

Amazon Cognito provides Web Identity Federation with the following features:

• Sign-up and sign-in to your Apps
• Access for guest users
• Acts as an Identity Broker between your application and Web ID providers. so you don’t need to write any additional code.
• Synchronizes user data for multiple devices
• Recommended for all mobile applications AWS services

User Pool vs Identity Pool

Cognito User Pools

User Pools are user directories used to manage sign-up and sign-in functionality for mobile and web applications. Users can sign-in directly to the User Pool, or using Facebook, Amazon, or Google. Cognito acts as an Identity Broker between the identity provider and AWS. Successful authentication generates a JSON Web token (JWTs).

User Pool is user based. It handles things like user registration, authentication, and account recovery.

Cognito Identity Pools

Identity Pools enable provide temporary AWS credentials to access AWS services like S3 or DynamoDB. It’s all about the authorization of access to AWS resources whereas User Pools are all about your actual users.

So, the difference between user pools and identity pools is that user pools are things like your email address to your password, whereas identity pools is the actual granting you access to an AWS resources.

Amazon Cognito in Action

So we’ve got a user who wants to connect into our website.

• She’s going to log in using her Facebook account once Facebook has authenticated her account as being genuine so her user name and password is correct. It’s going to pass back a authentication token to Cognito User Pool.
• Cognito User Pool then convert that to a JWT (JSON Web Token) token.

• She then sends that JWT token to an Identity Pool and that Identity Pool will grant her AWS credentials in a form ofIAM role.

• Then she will be able to go on and access her AWS resources.

Cognito Synchronization

Cognito tracks the association between user identity and the various different devices that sign-in from. In order to provide a seamless user experience for your application, Cognito uses Push Synchronization to push updates and synchronize user data across multiple devices.

Cognito uses SNS to send a notification to all the devices associated with a given user identity whenever data stored in the cloud changes.

---

References

• AWS News Blog

  • New – Amazon Kinesis Data Analytics for Java

• AWS Compute Blog

  • Increasing real-time stream processing performance with Amazon Kinesis Data Streams enhanced fan-out and AWS Lambda
  • Scale Amazon Kinesis Data Streams with AWS Application Auto Scaling
  • https://aws.amazon.com/blogs/big-data/perform-near-real-time-analytics-on-streaming-data-with-amazon-kinesis-and-amazon-elasticsearch-service/

Dive Into Exam

In SWF, what does a “domain” refer to?

• Answer: A collection of related workflows.
• Explanation: Domains in SWF are a mechanism to scope SWF resources such as workflows, activity types, and workflow execution. All the resources are scoped to a domain. Domains isolate one set of types, executions, and task lists from other ones within an AWS account. When you work with SWF, you need to first define a domain. All the other resources are defined within a domain.