AWS Cloud Developer Associate Certification

17.Monitoring,Logging, Auditing
Index

  1. Monitoring Overview in AWS
  2. CloudWatch
    1. CloudWatch Metrics
    2. CloudWatch Custom Metrics
    3. CloudWatch Logs
    4. CloudWatch Logs Hands On
    5. CloudWatch Agent & CloudWatch Logs Agent
    6. CloudWatch Logs Metric Filters
    7. CloudWatch Alarms
    8. CloudWatch Alarms Hands On
    9. CloudWatch Events
  3. EventBridge Overview
    1. EventBridge Hands On
  4. X-Ray Overview
    1. X-Ray Hands On
    2. X-Ray: Instrumentation and Concepts
    3. X-Ray: Sampling Rules
    4. X-Ray APIs
    5. X-Ray with Beanstalk
    6. X-Ray & ECS
  5. CloudTrail
    1. CloudTrail Hands On
  6. CloudTrail vs CloudWatch vs X-Ray

Monitoring Overview in AWS

  1. AWS CloudWatch:
    1. Metrics: Collect and track key metrics
    2. Logs: Collect, monitor, analyze and store log files
    3. Events: Send notifications when certain events happen in your AWS
    4. Alarms: React in real-time to metrics / events
  2. AWS X-Ray:
    1. Troubleshooting application performance and errors
    2. Distributed tracing of microservices
  3. AWS CloudTrail:
    1. Internal monitoring of API calls being made
    2. Audit changes to AWS Resources by your users

AWS CloudWatch Metrics

  1. CloudWatch provides metrics for every services in AWS
  2. Metric is a variable to monitor (CPUUtilization, NetworkIn…)
  3. Metrics belong to namespaces
  4. Dimension is an attribute of a metric (instance id, environment, etc…).
  5. Up to 10 dimensions per metric
  6. Metrics have timestamps
  7. Can create CloudWatch dashboards of metrics

EC2 Detailed monitoring

  1. EC2 instance metrics have metrics “every 5 minutes
  2. With detailed monitoring (for a cost), you get data “every 1 minute”
  3. Use detailed monitoring if you want to scale faster for your ASG!
  4. The AWS Free Tier allows us to have 10 detailed monitoring metrics
  5. Note: EC2 Memory usage is by default not pushed (must be pushed from inside the instance as a custom metric)

CloudWatch Custom Metrics

  1. Possibility to define and send your own custom metrics to CloudWatch
  2. Example: memory (RAM) usage, disk space, number of logged in users …
  3. Use API call PutMetricData
  4. Ability to use dimensions (attributes) to segment metrics
    1. Instance.id
    2. Environment.name
  5. Metric resolution (StorageResolution API parameter – two possible value):
    1. Standard: 1 minute (60 seconds)
    2. High Resolution: 1/5/10/30 second(s) – Higher cost
  6. Important: Accepts metric data points two weeks in the past and two hours in the future (make sure to configure your EC2 instance time correctly)

CloudWatch Logs

  1. Log groups: arbitrary name, usually representing an application
  2. Log stream: instances within application / log files / containers
  3. Can define log expiration policies (never expire, 30 days, etc..)
  4. CloudWatch Logs can send logs to
    1. Amazon S3 (exports)
    2. • Kinesis Data Streams
    3. • Kinesis Data Firehose
    4. • AWS Lambda
    5. • ElasticSearch

CloudWatch Logs – Sources

  1. SDK, CloudWatch Logs Agent, CloudWatch Unified Agent
  2. • Elastic Beanstalk: collection of logs from application
  3. • ECS: collection from containers
  4. • AWS Lambda: collection from function logs
  5. • VPC Flow Logs: VPC specific logs
  6. • API Gateway
  7. • CloudTrail based on filter
  8. • Route53: Log DNS queries

CloudWatch Logs Metric Filter & Insights

  1. CloudWatch Logs can use filter expressions
    1. For example, find a specific IP inside of a log
    2. Or count occurrences of “ERROR” in your logs
  2. Metric filters can be used to trigger CloudWatch alarms
  3. CloudWatch Logs Insights can be used to query logs and add queries to CloudWatch Dashboards

CloudWatch Logs – S3 Export

  1. Log data can take up to 12 hours to become available for export
  2. The API call is CreateExportTask
  3. Not near-real time or real-time… use Logs Subscriptions instead

CloudWatch Logs for EC2

  1. By default, no logs from your EC2 machine will go to CloudWatch
  2. You need to run a CloudWatch agent on EC2 to push the log files
  3. Make sure IAM permissions are coorect
  4. The CloudWatch log agent can be setup on-premises too

CloudWatch Logs Agent & Unified Agent

  1. For virtual servers (EC2 instances, on-premise servers…)
  2. CloudWatch Logs Agent
    1. Old version of the agent
    2. Can only send to CloudWatch Logs
  3. CloudWatch Unified Agent
    1. Collect additional system-level metrics such as RAM, processes, etc…
    2. Collect logs to send to CloudWatch Logs
    3. Centralized configuration using SSM Parameter Store

CloudWatch Unified Agent – Metrics

  1. Collected directly on your Linux server / EC2 instance
    1. CPU (active, guest, idle, system, user, steal)
    2. • Disk metrics (free, used, total), Disk IO (writes, reads, bytes, iops)
    3. • RAM (free, inactive, used, total, cached)
    4. • Netstat (number of TCP and UDP connections, net packets, bytes)
    5. • Processes (total, dead, bloqued, idle, running, sleep)
    6. • Swap Space (free, used, used %)
  2. Reminder: out-of-the box metrics for EC2 – disk, CPU, network (high level)

CloudWatch Logs Metric Filter

  1. CloudWatch Logs can use filter expressions
    1. • For example, find a specific IP inside of a log
    2. • Or count occurrences of “ERROR” in your logs
    3. • Metric filters can be used to trigger alarms
  2. • Filters do not retroactively filter data. Filters only publish the metric datapoints for events that happen after the filter was created.

CloudWatch Alarms

  1. • Alarms are used to trigger notifications for any metric
  2. • Various options (sampling, %, max, min, etc…)
  3. • Alarm States:
    1. • OK
    2. • INSUFFICIENT_DATA
    3. • ALARM
  4. • Period:
  5. • Length of time in seconds to evaluate the metric
  6. • High resolution custom metrics: 10 sec, 30 sec or multiples of 60 sec

CloudWatch Alarm Targets

  1. • Stop, Terminate, Reboot, or Recover an EC2 Instance
  2. • Trigger Auto Scaling Action
  3. • Send notification to SNS (from which you can do pretty much anything)

EC2 Instance Recovery

  1. Status Check:
    1. Instance status = check the EC2 VM
    2. System status = check the underlying hardware
  2. Recovery: Same Private, Public, Elastic IP, metadata, placement group

CloudWatch Alarm: good to know

  1. Alarms can be created based on CloudWatch Logs Metrics Filters
  2. To test alarms and notifications, set the alarm state to Alarm using CLI

aws cloudwatch set-alarm-state –alarm-name “myalarm” –state-value
ALARM –state-reason “testing purposes

CloudWatch Events

  1. Event Pattern: Intercept events from AWS services (Sources
    1. Example sources: EC2 Instance Start, CodeBuild Failure, S3, Trusted Advisor
    2. Can intercept any API call with CloudTrail integration
  2. Schedule or Cron (example: create an event every 4 hours
  3. A JSON payload is created from the event and passed to a target
    1. Compute: Lambda, Batch, ECS task
    2. • Integration: SQS, SNS, Kinesis Data Streams, Kinesis Data Firehose
    3. • Orchestration: Step Functions, CodePipeline, CodeBuild
    4. • Maintenance: SSM, EC2 Actions

EventBridge Overview

  1. EventBridge is the next evolution of CloudWatch Events
  2. • Default event bus: generated by AWS services (CloudWatch Events)
  3. • Partner event bus: receive events from SaaS service or applications
  4. (Zendesk, DataDog, Segment, Auth0…)
  5. • Custom Event buses: for your own applications
  6. • Event buses can be accessed by other AWS accounts
  7. • Rules: how to process the events (similar to CloudWatch Events)

Amazon EventBridge Schema Registry

  1. EventBridge can analyze the events in your bus and infer the schema
  2. The Schema Registry allows you to generate code for your application, that will know in advance how data is structured in the event bus
  3. Schema can be versioned

Amazon EventBridge vs CloudWatch Events

  1. • Amazon EventBridge builds upon and extends CloudWatch Events.
  2. • It uses the same service API and endpoint, and the same underlying service infrastructure.
  3. • EventBridge allows extension to add event buses for your custom applications and your third-party SaaS apps.
  4. • Event Bridge has the Schema Registry capability
  5. • EventBridge has a different name to mark the new capabilities
  6. • Over time, the CloudWatch Events name will be replaced with EventBridge

AWS X-Ray Overview

  1. Debugging in Production, the good old way:
    1. • Test locally
    2. • Add log statements everywhere
    3. • Re-deploy in production
  2. • Log formats differ across applications using CloudWatch and analytics is hard.
  3. • Debugging: monolith “easy”, distributed services “hard”
  4. • No common views of your entire architecture!
  5. • Enter… AWS X-Ray!

AWS X-Ray advantages

  1. Troubleshooting performance (bottlenecks)
  2. • Understand dependencies in a microservice architecture
  3. • Pinpoint service issues
  4. • Review request behavior
  5. • Find errors and exceptions
  6. • Are we meeting time SLA?
  7. • Where I am throttled?
  8. • Identify users that are impacted

X-Ray: In

AWS X-Ray Leverages Tracing

  1. • Tracing is an end to end way to following a “request”
  2. • Each component dealing with the request adds its own “trace”
  3. • Tracing is made of segments (+ sub segments)
  4. • Annotations can be added to traces to provide extra-information
  5. • Ability to trace:
    1. • Every request
    2. • Sample request (as a % for example or a rate per minute)
  6. • X-Ray Security:
    1. • IAM for authorization
    2. • KMS for encryption at rest

AWS X-Ray How to enable it?

  1. Your code (Java, Python, Go, Node.js, .NET) must import the AWS X-Ray SDK
    1. • Very little code modification needed
    2. • The application SDK will then capture:
    3. • Calls to AWS services
    4. • HTTP / HTTPS requests
    5. • Database Calls (MySQL, PostgreSQL, DynamoDB)
    6. • Queue calls (SQS)
  2. Install the X-Ray daemon or enable X-Ray AWS Integration
    1. X-Ray daemon works as a low level UDP packet interceptor (Linux / Windows / Mac…)
    2. AWS Lambda / other AWS services already run the X-Ray daemon for you
    3. Each application must have the IAM rights to write data to X-Ray

X-Ray Instrumentation in your code

  1. Instrumentation means the measure of product’s performance, diagnose errors, and to write trace information.
  2. • To instrument your application code, you use the X-Ray SDK
  3. • Many SDK require only configuration changes
  4. • You can modify your application code to customize and annotation the data that the SDK sends to XRay, using interceptors, filters, handlers, middleware

X-Ray Concepts

  1. • Segments: each application / service will send them
  2. • Subsegments: if you need more details in your segment
  3. • Trace: segments collected together to form an end-to-end trace
  4. • Sampling: decrease the amount of requests sent to X-Ray, reduce cost
  5. • Annotations: Key Value pairs used to index traces and use with filters
  6. • Metadata: Key Value pairs, not indexed, not used for searching

The X-Ray daemon / agent has a config to send traces cross account:
• make sure the IAM permissions are correct – the agent will assume the role
• This allows to have a central account for all your application tracing

X-Ray: Sampling Rules

  1. With sampling rules, you control the amount of data that you record
  2. You can modify sampling rules without changing your code
  3. By default, the X-Ray SDK records the first request each second, and five percent of any additional requests.
  4. One request per second is the reservoir, which ensures that at least one trace is recorded each second as long the service is serving requests
  5. Five percent is the rate at which additional requests beyond the reservoir size are sampled

X-Ray Write APIs (used by the X-Ray daemon)

  1. PutTraceSegments: Uploads segment documents to AWS X-Ray
  2. PutTelemetryRecords: Used by the AWS X-Ray daemon to upload telemetry.
    1. SegmentsReceivedCount,
    2. SegmentsRejectedCounts,
    3. BackendConnectionErrors
  3. GetSamplingRules: Retrieve all sampling rules (to know what/when to send)
  4. GetSamplingTargets & GetSamplingStatisticSummaries: advanced
  5. The X-Ray daemon needs to have an IAM policy authorizing the correct API calls

X-Ray Read APIs –

  1. GetServiceGraph: main graph
  2. BatchGetTraces: Retrieves a list of traces specified by ID. Each trace is a collection of segment documents that originates from a single request.
  3. GetTraceSummaries: Retrieves IDs and annotations for traces available for a specified time frame using an optional filter. To get the full traces, pass the trace IDs to BatchGetTraces
  4. GetTraceGraph: Retrieves a service graph for one or more specific traceIDs

X-Ray with Beanstalk

  1. You can run the daemon by setting an option in the Elastic Beanstalk console or with a configuration file (in .ebextensions/xray-daemon.config)
  2. Note: The X-Ray daemon is not provided for Multicontainer Docker

X-Ray & ECS

AWS CloudTrail

  1. Provides governance, compliance and audit for your AWS Account
  2. • CloudTrail is enabled by default!
  3. • Get an history of events / API calls made within your AWS Account by:
    1. • Console
    2. • SDK
    3. • CLI
    4. • AWS Services
  4. • Can put logs from CloudTrail into CloudWatch Logs or S3
  5. • A trail can be applied to All Regions (default) or a single Region.
  6. • If a resource is deleted in AWS, investigate CloudTrail first!

CloudTrail Events

  1. • Management Events:
    1. • Operations that are performed on resources in your AWS account
    2. • Examples:
      1. • Configuring security (IAM AttachRolePolicy)
      2. • Configuring rules for routing data (Amazon EC2 CreateSubnet)
      3. • Setting up logging (AWS CloudTrail CreateTrail)
    3. • By default, trails are configured to log management events.
    4. • Can separate Read Events (that don’t modify resources) from Write Events (that may modify resources)
  2. • Data Events:
    1. • By default, data events are not logged (because high volume operations)
    2. • Amazon S3 object-level activity (ex: GetObject, DeleteObject, PutObject): can separate Read and Write Events
    3. • AWS Lambda function execution activity (the Invoke API)
  3. • CloudTrail Insights Events:
    1. Enable CloudTrail Insights to detect unusual activity in your account:
      1. • inaccurate resource provisioning
      2. • hitting service limits
      3. • Bursts of AWS IAM actions
      4. • Gaps in periodic maintenance activity
      5. • CloudTrail Insights analyzes normal management events to create a baseline
    2. • And then continuously analyzes write events to detect unusual patterns
      1. • Anomalies appear in the CloudTrail console
      2. • Event is sent to Amazon S3
      3. • An EventBridge event is generated (for automation needs)

CloudTrail Events Retention

  1. Events are stored for 90 days in CloudTrail
  2. To keep events beyond this period, log them to S3 and use Athena

CloudTrail vs CloudWatch vs X-Ray

  1. CloudTrail:
    1. • Audit API calls made by users / services / AWS console
    2. • Useful to detect unauthorized calls or root cause of changes
  2. • CloudWatch:
    1. • CloudWatch Metrics over time for monitoring
    2. • CloudWatch Logs for storing application log
    3. • CloudWatch Alarms to send notifications in case of unexpected metrics
  3. • X-Ray:
    1. • Automated Trace Analysis & Central Service Map Visualization
    2. • Latency, Errors and Fault analysis
    3. • Request tracking across distributed systems

Pages: Page 1, Page 2, Page 3, Page 4, Page 5, Page 6, Page 7, Page 8, Page 9, Page 10, Page 11, Page 12, Page 13, Page 14, Page 15, Page 16, Page 17, Page 18, Page 19, Page 20, Page 21, Page 22, Page 23

Published by

Unknown's avatar

sevanand yadav

software engineer working as web developer having specialization in spring MVC with mysql,hibernate

Leave a comment