Securely Managing Application Secrets Using AWS Secrets Manager and IAM Roles (No Access Keys)

Learn how real companies securely fetch secrets from AWS Secrets Manager without ever storing AWS access keys on the instance.

In this hands-on guide you will:

• Launch an EC2 instance with an IAM role (no credentials)

• Create a secret in AWS Secrets Manager

• Fetch it using AWS CLI, Bash script, and Node.js

• Perform a live secret update (zero code change, zero redeploy)

This is exactly how production applications handle secrets in 2026.

Why This Matters

Hardcoding secrets or storing AWS access keys on EC2 is one of the biggest security risks in the cloud.
The correct way: Use AWS Secrets Manager + IAM Roles for EC2.

• No access keys in code or environment variables

• Automatic temporary credentials via IAM

• Secrets encrypted at rest with AWS KMS

• Live updates without redeploying your app

Prerequisites

• An AWS account

• One Ubuntu EC2 instance (t3.medium or larger recommended)

• IAM permissions to create roles and secrets

Step 0: EC2 Setup (User Data)

When launching your Ubuntu EC2 instance, add this User Data script so AWS CLI is ready:

#!/bin/bash

sudo apt update -y
sudo snap install aws-cli --classic

1. Create IAM Role for EC2 (Console)

1. Go to IAM → Roles → Create role

2. Trusted entity type: AWS service → EC2

3. Attach permission policies: SecretsManagerReadWrite (managed policy)

4. Role name: SecretsManagerEC2Role

5. Create the role

2. Attach the IAM Role to Your EC2 Instance

1. Go to EC2 → Instances

2. Select your instance → Actions → Security → Modify IAM role

3. Choose SecretsManagerEC2Role

4. Save

3. Verify IAM Role (No Credentials Needed!)

SSH into your EC2 instance and run:

aws sts get-caller-identity

You should see your Account ID and the Role ARN (arn:aws:iam::...:role/SecretsManagerEC2Role).

Important point to remember:
You never ran aws configure. The IAM role automatically provides temporary credentials.

4. Create a Secret in AWS Secrets Manager

aws secretsmanager create-secret \
  --name db-secret-1 \
  --secret-string '{"username":"admin","password":"admin123"}' \
  --region us-east-1

5. Fetch the Secret (CLI Demo)

aws secretsmanager get-secret-value \
  --secret-id db-secret-1 \
  --query SecretString \
  --output text \
  --region us-east-1

You will see:

{"username":"admin","password":"admin123"}

6. Bash Script Demo

Create the script:

vim app.sh

Paste:

#!/bin/bash

SECRET=$(aws secretsmanager get-secret-value \
  --secret-id db-secret-1 \
  --query SecretString \
  --output text \
  --region us-east-1)

echo "Fetched Secret:"
echo $SECRET

Make it executable and run:

chmod +x app.sh
./app.sh

7. Node.js Application Demo (Real App Use Case)

Install Node.js and the AWS SDK:

sudo apt install -y nodejs npm
npm init -y
npm install aws-sdk

Create the app:

vim app.js

Paste:

const AWS = require("aws-sdk");

const client = new AWS.SecretsManager({
  region: "us-east-1"   // ← your region
});

async function getSecret() {
  const data = await client
    .getSecretValue({ SecretId: "db-secret-1" })
    .promise();

  const secret = JSON.parse(data.SecretString);

  console.log("Username:", secret.username);
  console.log("Password:", secret.password);
}

getSecret();

Run it:

node app.js

8. Live Update Demo (The Best Part)

Update the secret in AWS:

aws secretsmanager update-secret \
  --secret-id db-secret-1 \
  --secret-string '{"username":"admin","password":"secure123"}' \
  --region us-east-1

Run the Node.js app again (no code change!):

node app.js

You will now see the new password: secure123

This is production magic — secrets are dynamic.

9. Security Best Practices (What Real Companies Do)

• Never use SecretsManagerReadWrite in production (too broad)

• Use least-privilege policy instead:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:YOUR_ACCOUNT_ID:secret:db-secret-1*"
    }
  ]
}

• Secrets are automatically encrypted with AWS KMS

• Temporary credentials are rotated automatically by IAM

Cleanup (Optional)

• Delete the secret in AWS Secrets Manager console

• Terminate the EC2 instance

Final Takeaways

• No AWS access keys stored anywhere

• IAM roles provide secure, temporary credentials

• Secrets are fully dynamic and encrypted

• This is exactly how production workloads on EC2, ECS, EKS, and Lambda handle secrets

Watch the full video to see every step live: YouTube Video

Woke up to an exciting email from AWS - I’ve been selected as an AWS Community Builder in the Containers category.

This moment feels incredibly special for me, especially because my application was rejected last year. Getting selected this year, even as a college student, makes the achievement even more meaningful.

In this blog, I want to share:

• What the AWS Community Builders program is

• My journey and experience applying

• Tips for anyone who wants to apply in the future

What is the AWS Community Builders Program?

The AWS Community Builders program is an initiative by Amazon Web Services (AWS) that recognizes individuals who actively share knowledge about AWS with the community.

Community Builders contribute by creating technical content such as:

• Blog posts

• Tutorials

• Videos

• Talks

• Workshops

The program includes multiple categories such as:

• Containers

• Serverless

• DevTools

• Machine Learning

• Security

• Data

I was selected in the Containers category, which focuses on cloud-native technologies and container-based architectures.

My Journey

My journey into cloud and DevOps started during my college years when I began exploring technologies like:

• Linux

• Docker

• Kubernetes

• CI/CD

• Cloud platforms

Currently, I’m also working as a DevOps Engineer Intern, where I get to work with tools like Kubernetes, CI/CD pipelines, and cloud infrastructure.

Last year, I applied to the AWS Community Builders program but my application was rejected. Instead of getting discouraged, I focused on improving my skills and continuing to share my learning with the community.

This year, I applied again - and waking up to the acceptance email was an unforgettable moment.

Why This Program Matters

Being part of the AWS Community Builders program provides several opportunities, including:

• Connecting with AWS product teams

• Early access to upcoming AWS features

• Joining a global network of cloud professionals

• Learning and collaborating with other builders

It also encourages members to continue sharing knowledge and helping others learn cloud technologies.

Tips for Future Applicants

If you are interested in becoming an AWS Community Builder, here are a few tips:

1. Share Your Knowledge

Create technical content such as blog posts, tutorials, or videos related to AWS.

2. Build Real Projects

Hands-on projects using AWS services help demonstrate practical experience.

3. Stay Consistent

Consistency in learning and sharing your knowledge with the community is important.

4. Don’t Give Up

If your application is rejected, treat it as motivation to improve and try again.

Final Thoughts

Becoming an AWS Community Builder is an exciting milestone in my cloud journey. I’m looking forward to contributing to the community by sharing knowledge around:

• Cloud-native technologies

• Containers

• Kubernetes

• DevOps practices on AWS

A big thank you to Amazon Web Services, Paxton L. Hall, and Ridhima Kapoor for this opportunity.

I’m excited for the journey ahead.

Understanding the difference between these three is critical if you’re preparing for AWS certifications, working in DevOps, or building production infrastructure.

Private IP is used for internal communication inside a VPC, Public IP allows internet access, and Elastic IP is a static public IP you control and can remap.

Let’s break everything down simply and practically.

1️⃣ What is a Private IP in AWS?

A Private IP address is assigned to an EC2 instance within a VPC (Virtual Private Cloud). It is used for internal communication between resources inside the AWS network.

Private IPs:

• Are assigned from your VPC CIDR block (e.g., 10.0.0.0/16)

• Cannot be accessed directly from the internet

• Remain with the instance for its lifetime

• Are used for backend communication (e.g., app server → database)

Example

If you launch two EC2 instances in the same VPC:

• Instance A: 10.0.1.10

• Instance B: 10.0.1.20

They can communicate using these private IPs without going over the internet.

When to Use Private IP

• Connecting application servers to databases

• Internal microservices communication

• Backend-only systems

• Secure internal networking

In real-world production setups, databases like RDS are accessed only via private IPs for security.

2️⃣ What is a Public IP in AWS?

A Public IP address allows your EC2 instance to communicate with the internet.

Public IPs:

• Are assigned automatically (if enabled)

• Change when you stop and start the instance

• Allow inbound/outbound internet traffic

• Are mapped to the instance’s private IP

If your EC2 instance is in a public subnet and has an Internet Gateway attached to the VPC, it can receive a public IP.

Example

You launch a web server:

• Private IP: 10.0.1.15

• Public IP: 3.110.45.123

Users access your website via the public IP.

Important Limitation

If you:

• Stop the instance

• Start it again

The public IP changes.

This is a big problem for production systems.

3️⃣ What is an Elastic IP (EIP)?

An Elastic IP is a static public IP address that you allocate manually and attach to your EC2 instance.

Unlike regular public IPs:

• It does NOT change when you stop/start the instance

• It belongs to your AWS account

• You can remap it to another instance

Elastic IP solves the “changing public IP” problem.

Why “Elastic”?

Because you can:

• Detach it from one instance

• Attach it to another instance instantly

This is useful in:

• Disaster recovery

• Failover setups

• Production environments

Quick Comparison Table

Real-World Use Case Example

Let’s say you're deploying a production application:

• Load Balancer → Public access

• EC2 App Servers → Private IP only

• RDS Database → Private IP only

In some cases:

• You attach an Elastic IP to a Bastion Host

• Or attach an Elastic IP to a production EC2 server

This setup improves both security and reliability.

Cost Considerations (Important)

Elastic IPs are free only when attached to a running instance.

AWS charges you if:

• You allocate an Elastic IP, but don’t use it

• You attach more than one Elastic IP per instance (in some cases)

Always release unused Elastic IPs to avoid charges.

Security Perspective

Best practice in AWS architecture:

• ❌ Never expose databases with a public IP

• ❌ Avoid unnecessary public IP assignments

• ✅ Use private subnets for backend services

• ✅ Use Elastic IP only when you truly need static public access

Security Groups and NACLs still control traffic regardless of IP type.

Final Thoughts

Understanding Private, Public, and Elastic IPs is foundational for AWS networking.

If you remember just one thing:

• Private IP → Internal communication

• Public IP → Temporary internet access

• Elastic IP → Permanent public identity

Once you master this, VPC architecture becomes much easier to design and troubleshoot.

Happy Learning 🚀

If you’ve spent any time in AWS, you’ve seen the term STS (Security Token Service). It sounds like a boring background process, but it is actually the secret sauce that keeps professional AWS environments secure.

Many people get confused: "If I already have my IAM Access Keys, why do I need STS?"

The answer lies in the difference between who you are and what you are allowed to do right now.

The Analogy: The ATM Card vs. The Transaction

Think of your IAM Access Keys (Long-term) as your ATM Card. You keep it in your wallet for years. It represents your identity at the bank.

AWS STS is the Transaction Receipt/PIN check. Even if you have the card, the bank doesn’t let you walk into the vault. Instead, they give you a temporary "session" to perform one specific task.

In AWS, your long-term keys are used to ask STS for a Temporary Session. This session is like a "Hall Pass" - it has a countdown timer (usually 1 hour). When the timer hits zero, the pass becomes useless paper.

Why "Temporary" is Better Than "Permanent"

You might think, "If my session expires, I just use my keys to get a new one. What’s the point?" The point is Force-Multiplying Security:

1. The MFA Checkpoint: You can set a rule that says: "To get an STS session, you must provide an MFA code." Now, if a hacker steals your permanent keys, they are stuck. They can’t start a session because they don't have your phone.

2. The "Kill Switch": If an admin suspects your account is compromised, they can Revoke all active sessions in one click. Even if you have the permanent keys, STS will refuse to give you a new session.

3. Cross-Account Access: STS allows you to jump from a "Dev" account to a "Prod" account without needing a separate username and password for both. You "Assume a Role," do the work, and the access automatically vanishes after an hour.

4. Least Privilege: Your permanent user can have zero permissions. You only gain power when you request an STS session for a specific IAM Role. If your laptop is stolen while you're at lunch, the thief only has access to a "powerless" user once the session expires.

The Bottom Line

STS turns your static, dangerous permanent keys into dynamic, short-lived permissions. It ensures that every hour (or however long the admin defines), AWS stops and asks: "Are you still who you say you are? And are you still allowed to do this?"

It’s not an inconvenience; it’s a heartbeat check for your cloud security.

Happy Learning!
Amitabh Soni

When working with AWS IAM, one of the most common questions engineers face is:

“Is this permission enough?”
“Did I accidentally give more access than required?”

Testing IAM permissions directly in a real AWS account can be risky, especially in production environments. Creating users, attaching policies, and performing actions just to verify access can lead to unintended security issues.

This is where AWS IAM Policy Simulator becomes extremely useful.

What Is AWS IAM Policy Simulator?

AWS IAM Policy Simulator is a built-in AWS tool that allows you to test and validate IAM permissions without actually creating or modifying resources.

It simulates how AWS evaluates policies and tells you whether a specific action would be allowed or denied for a given IAM user, role, or policy.

In simple terms, it answers:

• Can this user perform this action?

• Which policy allows or denies it?

All without touching real infrastructure.

Why You Should Use IAM Policy Simulator

1. Avoid Testing in Real AWS Accounts

Testing permissions manually often means:

• Creating resources

• Triggering API calls

• Risking security or unexpected costs

The Policy Simulator removes this risk entirely.

2. Validate Least Privilege Access

IAM best practice recommends granting only the permissions required.

With the simulator, you can:

• Check if permissions are insufficient

• Detect over-permissioned policies

• Fine-tune policies before deployment

3. Debug Permission Issues Faster

Instead of guessing why an action is failing:

• Simulate the action

• Identify the exact policy causing the denial

• Fix the issue quickly

This is especially helpful in complex environments with multiple attached policies.

How IAM Policy Simulator Works

At a high level, the simulator follows these steps:

1. Select an IAM user, role, or policy

2. Choose AWS services and actions (for example: s3:PutObject)

3. Simulate the request

4. Review the result (Allowed or Denied)

5. See which policy affected the decision

AWS evaluates permissions exactly as it would during a real API call - just without executing it.

How to Access IAM Policy Simulator

You can access the IAM Policy Simulator using the link below:

https://policysim.aws.amazon.com/home/index.jsp

Steps:

1. Log in to your AWS account

2. Open the IAM Policy Simulator

3. Select a user, role, or group

4. Choose actions to simulate

5. Review the results

Real-World Use Cases

Use Case 1: Before Assigning Permissions

Before attaching a policy to a user or role:

• Simulate required actions

• Confirm permissions are sufficient

• Avoid granting unnecessary access

Use Case 2: Troubleshooting Access Denied Errors

When an application fails due to permission issues:

• Simulate the failing action

• Identify missing permissions

• Update policies confidently

Use Case 3: Security Reviews and Audits

During audits:

• Validate access paths

• Ensure least privilege

• Demonstrate compliance without modifying infrastructure

Limitations to Keep in Mind

While powerful, the IAM Policy Simulator:

• Does not simulate resource-based policies perfectly in all scenarios

• Does not execute real AWS operations

• Should be used alongside logging tools like AWS CloudTrail

It is best used as a pre-deployment and debugging tool, not a replacement for monitoring.

Best Practices When Using IAM Policy Simulator

• Always simulate permissions before production deployment

• Use it to refine least-privilege policies

• Combine it with IAM Access Analyzer and CloudTrail

• Regularly review policies as services evolve

Conclusion

AWS IAM Policy Simulator is an essential tool for anyone working with AWS security and access management.

It allows you to:

• Test permissions safely

• Reduce security risks

• Debug faster

• Follow IAM best practices

If you’re working with IAM and not using the Policy Simulator yet, you’re missing a powerful safety net.

Happy Learning,
Amitabh Soni

Self-hosted runners provide greater flexibility and customization for your GitHub Actions workflows. This guide will walk you through the process of setting up and managing your own runners on AWS EC2 using Ubuntu.

Why Use Self-hosted Runners on EC2?

Benefits of EC2-based Runners:

• Cost Control: Optimize instance types for your specific workloads

• Network Access: Direct access to your AWS resources and VPCs

• Customization: Install specific software and dependencies

• Scalability: Easily scale up or down based on workflow demands

• Persistence: Maintain state between workflow runs if needed

• Resource Control: Choose instance types with the CPU/RAM/storage you need

Prerequisites

Before you begin, make sure you have:

• An AWS account with permissions to create EC2 instances

• A GitHub repository where you want to use self-hosted runners

• Basic knowledge of AWS EC2 and SSH

• Admin access to the GitHub repository or organization

Step 1: Launch an EC2 Instance

1. Log in to AWS Console and Launch an Instance

1. Go to the AWS Management Console

2. Navigate to EC2 Dashboard

3. Click "Launch Instance"

2. Choose an Ubuntu AMI

1. Select "Ubuntu Server 22.04 LTS (HVM)"

2. This provides a stable, long-term supported environment for your runner

3. Select Instance Type

1. For basic workflows: t3.small (2 vCPU, 2 GB RAM)

2. For moderate workloads: t3.medium (2 vCPU, 4 GB RAM)

3. For resource-intensive tasks: m5.large or higher

4. Configure Instance Details

1. Network: Select your VPC

2. Subnet: Choose a subnet with internet access

3. Auto-assign Public IP: Enable

4. IAM role: Attach a role with necessary permissions if your workflows need AWS services

5. Add Storage

1. Root volume: At least 20 GB (more if you'll be building large projects)

2. Volume type: gp3 (general purpose SSD)

6. Configure Security Group

Create a security group with these rules:

• SSH (port 22): Restrict to your IP address

• HTTPS (port 443): Allow from anywhere (for GitHub communication)

• HTTP: Allow from anywhere

7. Launch Instance and Create/Select Key Pair

1. Launch the instance

2. Create or select an existing key pair

3. Download the key pair if creating new

4. Set proper permissions: chmod 400 your-key.pem

Step 2: Connect to Your EC2 Instance

ssh -i your-key.pem ubuntu@your-ec2-public-dns

Step 3: Prepare the EC2 Instance

1. Update System Packages

sudo apt update
sudo apt upgrade -y

2. Install Required Dependencies

# Install basic tools
sudo apt install -y curl wget git jq build-essential

# Install Docker (optional, if you need Docker for your workflows)
sudo apt install -y docker.io
sudo systemctl enable docker
sudo systemctl start docker
sudo usermod -aG docker ubuntu

3. Create a Dedicated User for the Runner (Optional but Recommended)

# Create a user
sudo adduser github-runner

# Add to necessary groups
sudo usermod -aG docker github-runner

# Switch to the new user
sudo su - github-runner

Step 4: Set Up the GitHub Runner

1. Get Runner Registration Token

For Repository-level Runner:

1. Go to your GitHub repository

2. Navigate to Settings → Actions → Runners

3. Click "New self-hosted runner"

4. Select Linux as a Runner Image

5. Select architecture as x64

6. Open a your EC2 instance terminal and run all command step by step as provide by GitHub

During configuration, you'll be asked to, just press enter, if you do not want to change anything:

1. Enter a runner group name (default)

2. Enter a runner name (default is the hostname)

3. Enter additional labels (e.g., ubuntu-ec2, production)

4. Enter a work folder (default is _work)

Step 5: Verify Runner Registration

1. Go back to your repository settings

2. Navigate to Settings → Actions → Runners

3. You should see your new runner listed as "Idle"

4. The status will change to "Active" when running a job

Step 6: Create a Hello World Workflow to Test Your Self-hosted Runner

Let's create a simple "Hello World" workflow to verify your self-hosted runner is working correctly.

Create a .github/workflows/hello-world-runner.yml file in your repository:

# ===================================================
# HELLO WORLD SELF-HOSTED RUNNER TEST - MODULE 3
# ===================================================
# LEARNING OBJECTIVES:
# - Understand how to target self-hosted runners
# - Learn how to verify runner functionality
# - See basic environment information retrieval
# - Experience the difference between GitHub-hosted and self-hosted runners
# ===================================================

name: Hello World Self-Hosted Runner

# LEARNING POINT: Multiple trigger types for flexibility
on:
  # Manual trigger from GitHub UI
  workflow_dispatch:

  # Automatic trigger on push to main branch
  push:
    branches: [ main ]

jobs:
  # LEARNING POINT: Simple job targeting self-hosted runner
  hello-world:
    # LEARNING POINT: This is how you specify a self-hosted runner
    # Instead of ubuntu-latest, windows-latest, etc.
    runs-on: self-hosted

    steps:
      # LEARNING POINT: Standard checkout action works on self-hosted runners too
      - name: Checkout code
        uses: actions/checkout@v4

      # LEARNING POINT: Basic hello world with runner information
      - name: Hello from self-hosted runner
        run: |
          echo "========================================"
          echo "👋 Hello, World! I'm a self-hosted runner!"
          echo "========================================"
          echo "🏷️ Runner name: ${{ runner.name }}"
          echo "💻 Runner OS: ${{ runner.os }}"
          echo "📂 Working directory: $(pwd)"
          echo "📦 Repository: ${{ github.repository }}"
          echo "🔄 Workflow: ${{ github.workflow }}"

      # LEARNING POINT: Simple system information for verification
      - name: Basic system info
        run: |
          echo "========================================"
          echo "📊 System Information"
          echo "========================================"
          echo "🖥️ Hostname: $(hostname)"
          echo "💾 Disk space:"
          df -h / | grep -v Filesystem

          # Show that we're running on the self-hosted machine
          echo "🌐 IP Address:"
          hostname -I || echo "IP address command not available"

# ===================================================
# LEARNING NOTES:
# ===================================================
# 1. Self-hosted runners are specified with "runs-on: self-hosted"
# 2. You can add labels to runners and target them with:
#    runs-on: [self-hosted, linux, production]
# 3. Self-hosted runners have access to their host environment
# 4. You can install custom software on self-hosted runners
# 5. Self-hosted runners can access internal networks
# ===================================================

How to Use This Test Workflow

1. Commit and push the workflow file to your repository

2. Go to the Actions tab in your GitHub repository

3. Select "Hello World Self-Hosted Runner" from the workflows list

4. Click "Run workflow" and select the branch to run on

5. Watch the workflow run on your self-hosted runner

What This Test Verifies

This simple workflow confirms:

• Your self-hosted runner is properly connected to GitHub

• The runner can check out code from your repository

• Basic system information about your EC2 instance

Learning Points

1. Self-hosted runners are specified with runs-on: self-hosted

2. You can add labels to runners and target them with: runs-on: [self-hosted, linux, production]

3. Self-hosted runners have access to their host environment

4. You can install custom software on self-hosted runners

5. Self-hosted runners can access internal networks

If you see output from this workflow, congratulations! Your self-hosted runner on EC2 is working correctly and ready to run your GitHub Actions workflows.

For Sample Output you can visit: Sample Output

EC2 Instance Maintenance

Regular Updates:

sudo apt update
sudo apt upgrade -y

Troubleshooting

Common issues:

• Incorrect permissions

• Network connectivity problems

• GitHub token expired

Jobs Not Being Assigned

Check:

1. Runner is online in GitHub UI

2. Job's runs-on label matches your runner's labels

3. Repository has access to the runner group

Network Connectivity Issues

Test GitHub connectivity:

curl -v https://github.com

Check outbound access:

# Test HTTPS connectivity
curl -v https://api.github.com

# Test network configuration
ip addr
route -n

Cost Optimization

1. Choose the Right Instance Type

Match the instance type to your workload:

• CPU-intensive: Compute-optimized instances (c5, c6g)

• Memory-intensive: Memory-optimized instances (r5, r6g)

• Balanced: General purpose instances (t3, t2, m5)

2. Use Spot Instances

For non-critical workflows, consider spot instances:

• Up to 90% cheaper than on-demand

• Configure fallback to GitHub-hosted runners if spot instances are terminated

Conclusion

You now have a self-hosted GitHub Actions runner running on an AWS EC2 Ubuntu instance. This setup gives you full control over your CI/CD environment while leveraging the flexibility of AWS infrastructure.

By using EC2 for your self-hosted runners, you can:

• Customize the environment to your exact needs

• Access AWS resources directly with low latency

• Control costs by choosing appropriate instance types

• Scale your CI/CD capacity as your needs grow

Additional Resources

• GitHub Actions Self-hosted Runners Documentation

• AWS EC2 User Guide

• GitHub Actions Runner Repository

• AWS Auto Scaling Documentation

Networking is the backbone of modern communication, enabling billions of devices to connect and exchange information across the globe. Whether you're browsing a website, sending an email, or streaming a video, networking concepts are working behind the scenes to make it possible.

In this blog, I’ve summarized key networking fundamentals including how the Internet works, types of networks (LAN, WAN, MAN, PAN), the OSI and TCP/IP models, IP & MAC addressing, network devices like routers and switches, essential protocols and ports, firewall basics, and the client-server architecture.

If you're starting your journey in tech, preparing for DevOps roles, or aiming to strengthen your fundamentals, this blog will serve as a comprehensive guide to networking essentials.

How does the Internet work?

1. It basically uses the Optical fiber cable that is placed in the oceans at a large depth.

   1. to View that visit: https://www.submarinecablemap.com/

2. There are 3 tiers of company architecture used to provide the Internet to end users

   1. Tier 1: A company that owns the optical fiber cable in the ocean, companies like AT&T, NTT, Verizon

   2. Tier 2: A company like Jio, Airtel, and BSNL that takes the cable on RENT and provides it to their end users

   3. Tier 3: Small companies that also take internet from TIER 2 companies and provide the internet in a small area

      1. like, Jio fiber, Air Fiber, Sway broadband, You Broadband

Types of Network:

1. WAN (Wide Area Network):

   • Scope: Covers a very large geographic area, potentially spanning across cities, countries, or even continents.

   • Characteristics: Often involves complex infrastructure and can be a combination of public and private networks. Generally offers lower speeds and higher latency compared to LANs and MANs.

   • Examples:

     The Internet is a multinational corporation's network connecting offices across different countries.

2. MAN (Metropolitan Area Network):

   • Scope: Covers a larger geographic area than a LAN, typically a city or a large campus

   • Characteristics: Can be owned by a single entity or multiple organizations. Offers moderate speed and is suitable for connecting multiple LANs within a city.

   • Examples:

     A city-wide network connecting different government agencies, or a network connecting multiple university campuses within a metropolitan area.

3. LAN (Local Area Network):

   • Scope: Limited geographic area, such as a home, office building, or school.

   • Characteristics: Typically privately owned and managed. Offers high-speed data transmission and low latency (delay).

   • Examples: Your home Wi-Fi network, a computer lab in a school, or the network within a single office building.

4. PAN (Personal Area Network):

   1. Connects devices within a very close proximity, typically a few meters, for an individual's personal use. Examples include Bluetooth connections between a phone and headphones or a wireless mouse connected to a computer.

OSI Model & TCP/IP Model

OSI(Open Systems Interconnection) Model:

The OSI (Open Systems Interconnection) Model is a set of rules that explains how different computer systems communicate over a network. The OSI Model was developed by the International Organization for Standardization (ISO). The OSI Model consists of 7 layers, and each layer has specific functions and responsibilities.

7 Layers are:

1. Application Layer (Layer 7)

   • What It Does: Provides network services directly to user applications, like web browsers or email clients. It’s the layer you interact with when you use the internet.

   • Key Tasks: Handles protocols like HTTP (for web browsing), SMTP (for email), or FTP (for file transfers). It ensures apps can communicate over the network.

   • Example: When you type a URL into your browser, the Application Layer formats the request to fetch the webpage.

   • Analogy: The front desk of a hotel, where guests (users) make requests for services like room bookings or dining.

2. Presentation Layer (Layer 6)

   • What It Does: Translates data between the application and the network, ensuring it’s in a usable format. It handles encryption, compression, and data formatting.

   • Key Tasks: Converts data (e.g., text, images) into a standard format, encrypts sensitive data (like passwords), and compresses data to save bandwidth.

   • Example: When you send an encrypted email, the Presentation Layer encrypts the message before it’s sent.

   • Analogy: A translator who converts your words into a language the recipient understands.

3. Session Layer (Layer 5)

   • What It Does: Manages communication sessions between devices, ensuring connections are established, maintained, and terminated properly.

   • Key Tasks: Sets up, coordinates, and ends conversations between applications. It handles session recovery if a connection drops.

   • Example: During a video call, the Session Layer keeps the connection active and reconnects if the call briefly drops.

   • Analogy: A meeting coordinator who schedules, starts, and ends a conference call.

4. Transport Layer (Layer 4)

   • What It Does: Ensures reliable data transfer between devices, managing flow control, error checking, and data segmentation.

   • Key Tasks: Uses protocols like TCP (reliable, ordered delivery) or UDP (fast, less reliable). It breaks data into packets and reassembles them at the destination.

   • Example: When downloading a file, TCP ensures all packets arrive correctly and in order.

   • Analogy: A courier service that ensures packages are delivered intact and in the right sequence.

5. Network Layer (Layer 3)

   • What It Does: Routes data packets between different networks, determining the best path to the destination.

   • Key Tasks: Uses IP addresses to route packets. Protocols like IP (IPv4 or IPv6) operate here.

   • Example: When you visit a website, the Network Layer routes your request from your home network to the website’s server across the internet.

   • Analogy: A GPS system that finds the best route for your package to travel across cities.

6. Data Link Layer (Layer 2)

   • What It Does: Handles communication between devices on the same network, ensuring error-free data transfer.

   • Key Tasks: Uses MAC addresses to identify devices. It detects and corrects errors from the Physical Layer. Protocols include Ethernet.

   • Example: In your home Wi-Fi network, the Data Link Layer ensures your laptop and router exchange data correctly.

   • Analogy: A local mailroom sorting letters to ensure they reach the right office within a building.

7. Physical Layer (Layer 1)

   • What It Does: Manages the physical connection between devices, transmitting raw bits over hardware like cables or Wi-Fi.

   • Key Tasks: Defines hardware standards (e.g., cables, connectors, signal voltages). It converts data into electrical or optical signals.

   • Example: When you connect to Wi-Fi, the Physical Layer sends radio signals between your device and the router.

   • Analogy: The physical roads or wires that carry packages from one place to another.

Real-World Example: Sending an email involves all layers. The Application Layer formats the email, the Presentation Layer encrypts it, the Session Layer maintains the connection, the Transport Layer ensures reliable delivery, the Network Layer routes it to the recipient’s server, the Data Link Layer handles local network communication, and the Physical Layer sends the data over cables or Wi-Fi.

TCP/IP Model:

The TCP/IP model (Transmission Control Protocol/Internet Protocol) is a four-layer networking framework that enables reliable communication between devices over interconnected networks. It provides a standardized set of protocols for transmitting data across interconnected networks, ensuring efficient and error-free delivery. Each layer has specific functions that help manage different aspects of network communication, making it essential for understanding and working with modern networks.

TCP/IP was designed and developed by the Department of Defense (DoD) in the 1970s and is based on standard protocols. The TCP/IP model is a concise version of the OSI model. It contains four layers, unlike the seven layers in the OSI model.

4 Layers are:

1. Application Layer

   • What It Does: Combines the OSI’s Application, Presentation, and Session layers. It handles user-facing services, data formatting, and session management.

   • Key Tasks: Supports protocols like HTTP (web), SMTP (email), FTP (file transfer), and DNS (domain name resolution). It ensures apps can communicate over the network.

   • Example: When you visit a website, the Application Layer sends an HTTP request to the server and displays the webpage.

   • Analogy: A one-stop service desk handling customer requests, translations, and session coordination.

2. Transport Layer

   • What It Does: Manages end-to-end data transfer, ensuring reliability and flow control. It’s identical to the OSI Transport Layer.

   • Key Tasks: Uses TCP for reliable, ordered delivery (e.g., for emails) or UDP for faster, less reliable delivery (e.g., for streaming). It segments data into packets.

   • Example: When streaming a video, UDP sends packets quickly, prioritizing speed over perfection.

   • Analogy: A shipping company that decides whether to use guaranteed delivery (TCP) or express shipping with some risk of loss (UDP).

3. Network Layer

   • What It Does: Routes data packets across different networks, finding the best path to the destination. Matches the OSI Network Layer.

   • Key Tasks: Uses IP (IPv4 or IPv6) to assign addresses and route packets. Works with routing protocols to navigate the internet.

   • Example: When you send a message via WhatsApp, the Network Layer routes it from your phone to the recipient’s server.

   • Analogy: A postal service that forwards packages across cities using addresses.

4. Network Access Layer

   • What It Does: Combines the OSI’s Data Link and Physical layers. It handles data transfer within a single network and the physical transmission of bits.

   • Key Tasks: Uses MAC addresses for local communication and manages hardware (e.g., Ethernet, Wi-Fi). Converts data into signals for cables or wireless.

   • Example: In your home network, the Network Access Layer ensures your laptop’s data reaches your router via Wi-Fi.

   • Analogy: A local delivery truck that picks up packages and sends them over physical roads or airwaves.

Real-World Example: When you stream a movie, the Application Layer handles the streaming app’s interface and protocols (e.g., HTTPS), the Transport Layer uses UDP to deliver video packets quickly, the Network Layer routes packets to the streaming server, and the Network Access Layer sends signals over your Wi-Fi or Ethernet cable.

What is IP & MAC Address?

IP:

1. Internet Protocol (IP):

• It's a fundamental protocol that defines how data is transmitted across networks.

• It ensures that data packets (small chunks of data) are correctly routed to their destination.

• Essentially, it's the language that devices use to communicate with each other on the internet.

2. IP Address:

• An IP address is a unique numerical identifier assigned to each device connected to a network that uses the Internet Protocol.

• It acts like a postal address for your device on the internet, enabling other devices to find and communicate with it.

• There are two main versions of IP addresses: IPv4 and IPv6

  • IPv4: addresses are 32-bit numbers, typically written in dotted-decimal notation (e.g., 192.168.1.1).

  • IPv6: addresses are 128-bit numbers, designed to address the shortage of IPv4 addresses.

• IP addresses can be public (visible to the outside world, used for internet communication) or private (used within a local network).

MAC Address:

• Physical Address: MAC addresses are unique, hardware-level identifiers permanently assigned to a network interface card (NIC).

• Local Network Communication: They are used for communication within a local network, like a home or office network, to ensure data packets reach the correct device.

• Unchanging: MAC addresses are typically fixed and do not change.

Routers & Switches

Routers:

• Function: A router connects multiple networks, allowing devices on different networks to communicate, including connecting your local network to the internet. It acts as a traffic director between different networks.

• Layer: Routers operate at the Network Layer (Layer 3) of the OSI model.

• Connectivity: Connects LANs to other LANs or to the Internet.

• Example: A router connects your home network (LAN) to your internet service provider's network, enabling you to access the internet.

Switches:

• Function: A switch connects devices within the same network, allowing them to communicate with each other. Think of it as a traffic controller within a single office building.

• Layer: Switches operate at the Data Link Layer (Layer 2) of the OSI model

• Connectivity: Connects devices like computers, printers, and servers within a LAN.

• Example: In a home network, a switch might connect your computers, gaming consoles, and smart TVs to your router.

Firewall, Ports, Protocols

Protocols:

A protocol is a set of rules that govern how data is transmitted and received between devices on a network.

1. HTTP (Hypertext Transfer Protocol)(Port: 80): The foundation of data communication for the World Wide Web. It's used to transmit hypertext (like web pages) between web browsers and servers.

2. HTTPS (Hypertext Transfer Protocol Secure)(Port: 443): An encrypted version of HTTP. It uses TLS/SSL to provide a secure connection, protecting sensitive data during transmission.

3. FTP (File Transfer Protocol)(Port: 20 (Data), 21 (Control)): Used to transfer files between computers over a network. It allows for uploading and downloading files.

4. SMTP (Simple Mail Transfer Protocol)(Port: 25, 465, 587): The standard protocol for sending emails. It handles the transmission of email messages from a client to a mail server and between mail servers.

5. TCP (Transmission Control Protocol): A connection-oriented protocol that provides reliable, ordered, and error-checked delivery of data. It's used for applications where accuracy is crucial.

6. UDP (User Datagram Protocol): A connectionless protocol that offers a faster but less reliable way to transmit data. It's suitable for applications where speed is prioritized over absolute accuracy.

7. IP (Internet Protocol): The core protocol responsible for routing data packets across networks, ensuring that information reaches its intended destination.

8. ARP (Address Resolution Protocol): A protocol that translates IP addresses into MAC addresses, enabling devices to locate each other on a local network segment.

9. SSH(Port: 22): SSH, or Secure Shell, is a network protocol that provides a secure way to access and manage remote computers. It uses encryption to protect data transmitted between a client and a server, making it suitable for use over unsecured networks. Essentially, SSH allows you to securely log in to another computer, execute commands, and transfer files.

Ports:

• Ports are virtual locations on a device that are used to identify specific services or applications.

• They are identified by numbers, ranging from 0 to 65535.

• Each port is associated with a specific protocol and service. For example, port 80 is typically used for HTTP (web traffic), and port 443 is used for HTTPS (secure web traffic).

• Ports allow a device to handle multiple network connections simultaneously, directing incoming data to the appropriate application or service.

Firewall:

Firewalls act as security checkpoints, inspecting network traffic and enforcing rules to protect networks from unauthorized access and malicious activity. They use protocols and port numbers to identify and filter traffic, allowing or blocking connections based on pre-defined rules. For example, a firewall can be configured to block all traffic on port 21 (FTP) while allowing traffic on port 80 (HTTP) and 443 (HTTPS).

Client-Server Architecture

The client-server model is a network architecture where clients request resources or services from a central server. The server then provides these resources or services to the client. This model is fundamental to how many applications and services operate on the internet and within local networks.

Here's a more detailed breakdown:

• Clients: These are devices or software applications that initiate requests for data or services. Examples include web browsers, email clients, and mobile apps.

• Servers: These are powerful computers or software that provide resources and services to clients. They store data, run applications, and handle requests.

• Communication: Clients and servers communicate over a network using protocols like HTTP, TCP/IP, and others.

• Request-Response: The core of the model is the request-response pattern. A client sends a request, and the server processes it and sends back a response.

• Separation of Concerns: The client-server model promotes separation of concerns. Clients handle user interfaces and some processing, while servers handle data storage and complex operations.

Examples:

• When you visit a website, your web browser (client) sends a request to the web server, which then sends back the website's content.

• When you send an email, your email client (client) connects to an email server, which then relays the message to the recipient's email server.

• Online banking involves a client (web browser or app) making requests to a bank's server to access accounts, transfer funds, etc.

Benefits:

• Centralized Management: Servers can be managed and maintained centrally, making it easier to update and secure resources

• Scalability: The model can easily scale by adding more clients or servers as needed.

• Resource Sharing: Servers can share resources like databases, applications, and storage with multiple clients.

• Improved Performance: By offloading complex tasks to the server, clients can operate more efficiently

Conclusion

Understanding networking is crucial for anyone entering fields like DevOps, cybersecurity, system administration, or cloud computing. These core concepts, from layered communication models to addressing, routing, and protocols, form the foundation upon which modern computing relies.

As I continue my journey in tech, diving deep into these topics has helped me understand not just how the internet works, but why it works the way it does. I hope this blog helps you the same way it helped me consolidate my learning.

Stay curious, keep exploring, and happy learning!

What is Elastic Container Service?

• Amazon Elastic Container Services (Amazon ECS) is a fully managed container orchestration service that helps organizations easily deploy, manage, and scale containerized applications.

• Learn more about ECS

What is Elastic Container Registry?

• Amazon Elastic Container Registry (ECR) is a fully managed Docker container registry service provided by Amazon Web Services (AWS). In simple terms, it's a place where you can store, manage, and deploy Docker container images, making it easier for you to run applications in the cloud using containers.

• Learn more about ECR

What is Route 53?

• Amazon Route 53 is a scalable and highly available Domain Name System (DNS) web service provided by Amazon Web Services (AWS). It is named after the TCP/IP port 53, which is used for DNS services. Route 53 is designed to provide reliable and cost-effective domain registration, DNS routing, and health checking of resources within your AWS infrastructure.

• How Does DNS Route Traffic To Your Web Application? See the diagram below:

  

Tasks:

1) Deploy a two-tier application on Elastic Container Service (ECS) and configure Elastic Container Registry (ECR) to push Docker images.

Note: The Docker image must be fetched from ECR.

ANS:

1. Task 1 done - Application running ss:

   

2. EC2 instance used to build and push an image

   

3. Repository where the image is pushed.

   

4. The Docker image:

   

5. Task Definition:

   

6. Task inside cluster:

   

7. That task configuration:

   

8. Accessed the app at that public IP with port 8000

   

2) Understand the concept of CloudFront and try to perform below sub-tasks:

- What is caching in CloudFront?

- Create an EC2 instance with an Apache web server

- Create a CloudFront distribution and attach it to an EC2 instance to access the Apache webpage.

ANS:

To learn about AWS CloudFront, read this blog

1. What is Caching in CloudFront?

Caching in AWS CloudFront involves storing copies of your content at over 600 edge locations globally, as noted in the AWS CloudFront Documentation. When a user requests content, CloudFront checks the nearest edge location for a cached copy. If available, it delivers the content immediately, reducing latency. If not, it fetches the content from the origin (e.g., an EC2 instance), caches it, and serves it. Key aspects include:

• Cache Key: Determines what’s cached, based on headers, cookies, or query strings.

• TTL Settings: Controls how long content stays cached (default or custom via cache policies).

• Cache Hit Ratio: The proportion of requests served from the cache, which you can optimize using tools like Origin Shield.

This caching mechanism improves performance, reduces origin server load, and enhances reliability by distributing content closer to users.

2. Creating an EC2 Instance with Apache Web Server

Follow these steps to launch an EC2 instance and install Apache:

1. Log into AWS Management Console:

   • Access the AWS Console at AWS Management Console and navigate to the EC2 dashboard.

2. Launch an EC2 Instance:

   • Click “Launch Instance” and select an AMI, such as Amazon Linux 2 or Ubuntu Server 20.04.

   • Choose the t2.micro instance type (free tier eligible).

   • Configure instance details (use defaults unless specific needs arise).

   • Add storage (8 GB default is sufficient).

   • Add optional tags (e.g., Name: Apache-Server).

   • Configure a security group:

     • Create a new security group.

     • Add rules for:

       • SSH (port 22) from your IP for secure access.

       • HTTP (port 80) from 0.0.0.0/0 to allow web traffic.

   • Review and launch, selecting or creating a key pair (e.g., my-key.pem) for SSH access.

3. Connect to the Instance:

   • Once the instance is running, note its public DNS (e.g., ec2-xx-xx-xx-xx.compute-1.amazonaws.com).

   • Use SSH to connect: bashCopyssh -i /path/to/my-key.pem ec2-user@public-dns-name

     • For Amazon Linux, use ec2-user; for Ubuntu, use ubuntu.

4. Install and Configure Apache:

   • For Amazon Linux: bashCopy

       sudo yum update -y 
       udo yum install httpd -y 
       sudo systemctl start httpd 
       sudo systemctl enable httpd
     

   • For Ubuntu: bashCopy

       sudo apt update 
       sudo apt install apache2 -y 
       sudo systemctl start apache2 
       sudo systemctl enable apache2
     

   • Verify Apache is running by accessing the instance’s public DNS in a browser (e.g., http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com). You should see the Apache default page.

5. Optional: Customize Webpage:

   • Edit the default webpage (e.g., /var/www/html/index.html) to add custom content, ensuring it’s accessible via the browser.

     

3. Creating a CloudFront Distribution for EC2

To serve the Apache webpage through CloudFront, create a distribution with the EC2 instance as the origin:

1. Access CloudFront Console:

   • Navigate to CloudFront in the AWS Console at CloudFront Console.

2. Create a Distribution:

   • Click “Create Distribution” and select “Web” delivery method.

3. Configure Origin Settings:

   • Origin Domain Name: Enter the EC2 instance’s public DNS (e.g., ec2-xx-xx-xx-xx.compute-1.amazonaws.com).

   • Origin Path: Leave blank.

   • Origin Protocol Policy: Select “HTTP Only” (unless your EC2 supports HTTPS).

   • Minimum Origin SSL Protocol: Use default settings.

4. Configure Default Cache Behavior:

   • Viewer Protocol Policy: Choose “Redirect HTTP to HTTPS” for security or “HTTP and HTTPS”.

   • Allowed HTTP Methods: Select “GET, HEAD” for static content.

   • Cache Policy: Use “Managed-CachingOptimized” for typical static content caching, or customize TTL settings (e.g., Minimum TTL: 0, Default TTL: 86400 seconds).

   • Origin Request Policy: Select “Managed-AllViewer” to forward all viewer headers.

5. Configure Distribution Settings:

   • Price Class: Select “Use All Edge Locations” or restrict based on your audience’s location.

   • Alternate Domain Names (CNAMEs): Leave blank (no custom domain specified).

   • SSL Certificate: Use the default CloudFront certificate.

   • Leave other settings as default unless specific requirements apply.

6. Create and Deploy:

   • Click “Create Distribution”. Deployment may take 5–20 minutes.

   • Once the status changes to “Deployed”, note the distribution’s domain name (e.g., d1234567890.cloudfront.net).

     

7. Test the Distribution:

   • Open a browser and enter the CloudFront domain name (e.g., http://d1234567890.cloudfront.net).

   • Verify that the Apache webpage displays, matching the EC2 instance’s content.

1. EC2 DNS URL:

   

2. Cloud Front Domain Name:

   

3. You can check the URL in both SS.

3) Learn about AWS's fully managed DNS Service (Route53) and write a detailed blog post and post it on LinkedIn.

ANS:

AWS Route 53 is a highly available and scalable Domain Name System (DNS) web service, as detailed in the AWS Route 53 Documentation. It translates domain names (e.g., example.com) into IP addresses, enabling users to access websites and applications. Key features include:

• Domain Registration: Register or transfer domains directly through Route 53.

• DNS Management: Create hosted zones to manage DNS records (e.g., A, CNAME, MX).

• Traffic Routing: Supports policies like simple, weighted, latency-based, geolocation, and failover routing.

• Health Checking: Monitors resource health and reroutes traffic from unhealthy endpoints.

• Integration: Works seamlessly with AWS services like EC2, S3, and CloudFront.

Route 53’s global network of DNS servers ensures reliability, and its pay-as-you-go pricing makes it cost-effective for businesses of all sizes.

To Learn more about AWS Route 53, read this blog

Here's a more detailed explanation:

Key Features and Benefits:

• Global Reach:

  CloudFront has a network of edge locations (data centers) around the world, ensuring content is delivered quickly to users regardless of their location.

• Low Latency:

  By caching content in edge locations, CloudFront reduces the distance and time it takes for data to reach users, resulting in faster loading times and a better user experience.

• High Data Transfer Speeds:

  CloudFront's optimized network infrastructure enables high-speed data transfer, allowing for efficient delivery of large files like videos or software updates.

• Scalability:

  CloudFront can automatically scale to handle large volumes of traffic, ensuring consistent performance even during peak times.

• Security:

  CloudFront provides security features like HTTPS support and DDoS protection to safeguard your content and users.

• Cost-Effective:

  CloudFront is a pay-as-you-go service, meaning you only pay for the resources you use, and there are no minimum fees or long-term commitments.

• Flexible Integration:

  CloudFront can be integrated with various AWS services and can also be used with custom origins, allowing you to tailor your content delivery strategy.

How it Works:

1. 1. Content Origin:

   Your content (images, videos, web pages, etc.) is stored on a server known as the "origin".

2. 2. Content Delivery Network (CDN):

   CloudFront's edge locations act as a globally distributed network of servers that cache copies of your content.

3. 3. User Request:

   When a user requests your content, CloudFront's DNS routing system directs the request to the nearest edge location.

4. 4. Content Retrieval:

   If the content is already cached in the edge location, it's delivered to the user immediately.

5. 5. Origin Retrieval:

   If the content is not cached in the edge location, CloudFront retrieves it from the origin and stores a copy in the edge location for future requests.

In essence, CloudFront acts as a proxy, caching your content at the edge of the network and delivering it to users with minimal delay, resulting in faster website loading times, quicker file downloads, and improved overall user experience.

The Domain Name System (DNS) is the internet’s phonebook, translating user-friendly domain names like example.com into IP addresses (e.g., 192.0.2.1) that computers use to communicate. Without DNS, accessing websites would require memorizing complex numerical addresses. DNS ensures seamless navigation, making it a critical component of the internet.

What is AWS Route 53?

AWS Route 53 is Amazon Web Services’ highly available and scalable DNS web service, named after the standard DNS port 53. It provides a robust platform for domain registration, DNS management, traffic routing, and health checking. Route 53 connects user requests to AWS resources (like EC2 instances or S3 buckets) or external infrastructure, ensuring reliable and efficient access.

Key Features of Route 53

Route 53 offers a comprehensive set of tools to manage DNS and optimize traffic flow:

• Domain Registration: Register new domains or transfer existing ones through Route 53’s user-friendly console. For example, you can purchase mywebsite.com directly.

• DNS Management: Create hosted zones to store DNS records, such as A (for IP addresses), CNAME (for aliases), or MX (for email servers).

• Traffic Routing: Route 53 supports multiple routing policies:

  • Simple Routing: Directs traffic to a single resource.

  • Weighted Routing: Distributes traffic across resources based on assigned weights.

  • Latency-Based Routing: Routes to the resource with the lowest latency.

  • Geolocation Routing: Directs traffic based on user location.

  • Failover Routing: Reroutes traffic to backup resources during failures.

• Health Checking: Monitors resource health by sending automated requests and reroutes traffic if a resource becomes unavailable.

• Integration with AWS: Seamlessly connects to services like CloudFront, S3, and EC2 for streamlined DNS management.

How to Use Route 53: A Practical Example

Let’s walk through setting up Route 53 to point a domain to a CloudFront distribution, a common use case for hosting a website.

1. Register a Domain:

   • In the Route 53 console, search for an available domain (e.g., mywebsite.com).

   • Complete the registration process, providing contact details and payment information.

2. Create a Hosted Zone:

   • Navigate to “Hosted Zones” and click “Create Hosted Zone”.

   • Enter your domain name (mywebsite.com) and select “Public Hosted Zone”.

   • Note the assigned name servers (e.g., ns-123.awsdns-45.com).

3. Add DNS Records:

   • In the hosted zone, create an A record:

     • Name: mywebsite.com (or www.mywebsite.com for a subdomain).

     • Type: A – IPv4 address.

     • Alias: Yes.

     • Alias Target: Select your CloudFront distribution (e.g., d1234567890.cloudfront.net).

   • Optionally, add a CNAME record for www.mywebsite.com pointing to mywebsite.com.

4. Update Domain Name Servers:

   • If the domain was registered elsewhere, update its name servers to those provided by Route 53.

5. Test the Setup:

   • After DNS propagation (up to 48 hours), access mywebsite.com in a browser to verify it loads the CloudFront-hosted content.

This setup ensures users access your website via a custom domain, leveraging CloudFront’s performance benefits.

Advanced Features

Route 53 offers advanced capabilities for complex use cases:

• Traffic Policies: Use the visual Traffic Flow editor to create sophisticated routing configurations.

• Route 53 Resolver: Enables DNS resolution for hybrid cloud environments, connecting on-premises networks with AWS VPCs.

• DNS Firewall: Filters outbound DNS traffic to protect against malicious domains.

Benefits of Route 53

Route 53 stands out for its:

• High Availability: Leverages AWS’s global DNS server network for reliable query resolution.

• Scalability: Automatically handles large query volumes without performance degradation.

• AWS Integration: Simplifies DNS management for AWS resources like CloudFront and EC2.

• Advanced Routing: Optimizes performance and reliability with policies like latency-based routing.

• Cost-Effectiveness: Offers pay-as-you-go pricing, with no upfront costs.

Conclusion

AWS Route 53 is a powerful DNS service that simplifies domain management, enhances traffic routing, and ensures application reliability. Whether you’re hosting a simple website or managing a global application, Route 53’s features and AWS integration make it an essential tool. Explore Route 53 in the AWS Console to streamline your DNS needs and boost your application’s performance.

Ready to dive into Route 53? Share your experiences or questions in the comments below!

A Virtual Private Cloud (VPC) is a secure, isolated portion of a public cloud infrastructure that allows users to create their own virtual network, similar to a private cloud. It enables organizations to host and manage their resources within a specific, controlled environment, providing security and flexibility on a public cloud platform.

Here's a more detailed explanation:

• Isolation:

  VPCs offer logical isolation, separating resources from other users and tenants on the same public cloud.

• Control:

  Users can configure and manage their VPC, including setting up subnets, assigning IP addresses, and managing security groups.

• Benefits:

  VPCs offer benefits such as increased security, flexibility, and scalability, allowing users to adapt their infrastructure to changing needs.

• Use Cases:

  VPCs are commonly used for hosting web applications, migrating workloads to the cloud, and building secure environments for sensitive data and applications.

• Cloud Providers:

  Major cloud providers like AWS, Google Cloud, and IBM offer VPC services, allowing users to leverage the advantages of public cloud infrastructure while maintaining control over their virtual network.

Subnet:

A subnet, or subnetwork, is a logical division of a larger IP network. It allows for the efficient management of network traffic and resource allocation by breaking down a large network into smaller, more manageable segments. Each subnet has its own unique IP address range.

Key Concepts:

• Logical Partition:

  Subnets are not physical divisions of a network but rather logical groupings of devices based on their IP addresses.

• IP Address Ranges:

  Each subnet is assigned a specific range of IP addresses, and devices within that range can communicate directly without needing to go through a router.

• Subnet Mask:

  A subnet mask is used to identify the network portion and host portion of an IP address, determining which IP addresses belong to a specific subnet.

• Benefits:

  • Improved Network Efficiency: Subnets reduce the amount of traffic on the main network by isolating communication within each subnet.

  • Simplified Management: Subnets make it easier to manage and troubleshoot network issues by isolating problems within specific subnet segments.

  • Enhanced Security: Subnets can be used to create separate security zones, limiting access and protecting sensitive data.

  • Better Scalability: Subnets allow for a more flexible and scalable network design, making it easier to add new devices or expand network segments.

• Types of Subnets

  1. Public Subnets:

  • These subnets have a direct route to an internet gateway, allowing resources within the subnet to access the public internet.

  • Instances in public subnets can be assigned public IP addresses.

  • They are commonly used for web servers, applications, or services that need to be accessed from the internet.

2. Private Subnets:

• These subnets do not have a direct route to an internet gateway.

• Resources in a private subnet typically require a NAT device (Network Address Translation) or VPN connection to access the internet.

• Private subnets are often used for internal applications, databases, or other services that do not need to be publicly accessible.

• Example:

  Imagine a company with multiple departments on different floors. Each department could be assigned a subnet, allowing devices within each department to communicate directly, while routing between departments would occur through the company's main router.

Internet Gateway:

An internet gateway is a virtual component that facilitates communication between a virtual private cloud (VPC) and the internet. It acts as a bridge, enabling resources within public subnets of a VPC, such as EC2 instances, to connect to the internet and vice versa. Essentially, it allows your VPC to interact with the wider internet.

Here's a more detailed explanation:

Functionality:

• Two-way communication:

  Internet gateways enable both outbound (from VPC to internet) and inbound (from internet to VPC) traffic.

• Public IP addresses:

  Resources within public subnets that have public IP addresses can use the internet gateway to connect to the internet.

• Network Address Translation (NAT):

  For IPv4 traffic, internet gateways perform NAT, which translates private IP addresses within the VPC to public IP addresses when communicating with the internet.

• Routing:

  They serve as the target in route tables within the VPC for internet-routable traffic, ensuring that traffic destined for the internet is routed correctly.

• Highly available and redundant:

  Internet gateways are designed to be highly available and redundant, ensuring reliable connectivity.

Key Use Cases:

• Connecting EC2 instances to the internet:

  You can use internet gateways to allow EC2 instances in your VPC to access the internet for tasks like software updates, data transfer, or accessing web services.

• Providing public access to web applications:

  You can use internet gateways to make web applications running in your VPC accessible to users on the internet.

• Receiving inbound connections from the internet:

  Internet gateways enable your VPC resources to accept connections from the internet, which is crucial for services that need to receive data or handle requests from external sources.

In essence, an internet gateway is a vital component for enabling your VPC to interact with the internet, allowing your resources to connect to and be accessed from the wider web.

Routing Table:

A routing table is a database that helps determine the best path for data packets to travel across a network. It's like a map that tells devices (like routers) where to send network traffic based on the destination IP address.

Key aspects of a routing table:

• Routing Decisions:

  Routers use routing tables to make decisions about where to forward data packets, ensuring efficient and reliable network traffic flow.

• Database of Network Paths:

  The table contains information about the network, including the destination IP address, the next hop IP address, and the interface to use for forwarding.

• Stored in RAM:

  Routing tables are typically stored in the random access memory (RAM) of routers or network switches.

• Dynamic Updates:

  Routing tables can be updated dynamically, for example, when a network link goes down or a new network is discovered.

How it works:

When a router receives a data packet, it looks up the destination IP address in its routing table. Based on the table's information, the router determines the best next hop IP address and the interface to use for forwarding the packet towards its destination.

Types of Routing Tables:

• Static Routing Tables: These are manually configured and do not dynamically update.

• Dynamic Routing Tables: These automatically update based on network changes, using routing protocols like Open Shortest Path First (OSPF) or Border Gateway Protocol (BGP).

AWS ECR stands for Amazon Elastic Container Registry. It's a fully managed container registry service provided by AWS that allows you to store, manage, and deploy Docker container images. Essentially, it's a place where you can keep your container images, similar to how you would keep your code in a Git repository.

Here's a more detailed breakdown:

• Amazon Elastic Container Registry (Amazon ECR):

  The official name of the AWS service.

• Fully Managed:

  AWS handles the infrastructure and maintenance of the registry, so you don't have to.

• Container Registry:

  It's a service for storing and retrieving container images, which are like pre-packaged software environments.

• Docker Container Images:

  ECR primarily works with Docker images, but it also supports other container formats like Open Container Initiative (OCI) images.

• Storage and Management:

  You can use ECR to store your images, manage them, and then deploy them to other AWS services like ECS or EKS.

• Security:

  ECR offers security features like access control through IAM (AWS Identity and Access Management).

• Integration:

  ECR integrates well with other AWS services, making it easy to build, store, and deploy containerized applications.

AWS ECS:

AWS ECS stands for Amazon Elastic Container Service. It is a fully managed container orchestration service offered by Amazon Web Services that simplifies the deployment, management, and scaling of containerized applications.

Elaboration:

• Fully Managed:

  Amazon ECS handles the underlying infrastructure, including provisioning and managing servers, clusters, and container instances, relieving users from these tasks.

• Container Orchestration:

  It orchestrates and manages Docker containers, allowing for the deployment, scaling, and coordination of containerized applications.

• Scalability and Availability:

  ECS ensures the application's availability by automatically managing container instances, scaling up or down as needed to meet demand.

• Integration with AWS:

  ECS seamlessly integrates with other AWS services like Elastic Load Balancer, AWS Fargate, and Amazon RDS, simplifying application deployment and management.

• Flexibility:

  ECS supports various deployment options, including running on Amazon EC2 instances, AWS Fargate (a serverless compute engine), or on-premises with Amazon ECS Anywhere.

AWS Fargate:

AWS Fargate is a serverless compute engine for containers that runs within Amazon ECS and EKS, allowing users to run containers without managing servers or clusters of virtual machines. It eliminates the need for users to provision, configure, or scale servers, focusing instead on application development and deployment.

Here's a more detailed explanation:

• Serverless Compute Engine:

  Fargate is a serverless service, meaning AWS manages the underlying infrastructure and resources, including server provisioning, configuration, and scaling.

• Containers:

  Fargate is designed to run Docker containers, making it ideal for cloud-native applications.

• ECS and EKS Integration:

  Fargate seamlessly integrates with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS).

• Simplified Management:

  Fargate removes the need for users to manage the infrastructure, allowing them to focus on their applications and their deployment.

• Resource Management:

  Users specify the required resources (CPU, memory) for their containers, and Fargate automatically allocates and manages those resources.

• Pay-as-you-go:

  Fargate uses a pay-as-you-go pricing model, where users only pay for the resources they consume.

• Benefits:

  Fargate provides benefits such as simplified infrastructure management, resource right-sizing, improved security through application isolation, and potentially lower costs.

1. Learn about the following to get started with VPC and post it on LinkedIn:

   ANS:

   1. Virtual Private Cloud(VPC): A Virtual Private Cloud (VPC) is a secure, isolated portion of a public cloud infrastructure that allows users to create their own virtual network, similar to a private cloud. It enables organizations to host and manage their resources within a specific, controlled environment, providing security and flexibility on a public cloud platform.

   2. Subnet: A subnet, or subnetwork, is a logical division of a larger IP network. It allows for the efficient management of network traffic and resource allocation by breaking down a large network into smaller, more manageable segments. Each subnet has its own unique IP address range.

   3. Internet Gateway: An internet gateway is a virtual component that facilitates communication between a virtual private cloud (VPC) and the internet. It acts as a bridge, enabling resources within public subnets of a VPC, such as EC2 instances, to connect to the internet and vice versa. Essentially, it allows your VPC to interact with the wider internet.

   4. Route table: A routing table is a database that helps determine the best path for data packets to travel across a network. It's like a map that tells devices (like routers) where to send network traffic based on the destination IP address.

   5. Peering connections: A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs or with a VPC in another AWS account. The VPCs can be in different regions (also known as inter-region VPC peering connections).

2. Imagine you’re the cloud architect for a tech company, ByteConnect Inc. They’ve expanded rapidly, and each department operates in its own isolated cloud space(VPC). Now, the challenge is to establish a communication channel for their instances to communicate seamlessly using AWS Transit Gateway.

   1. Reference: AWS Transit Gateways

ANS:

Read this blog for the Task/Project guide: AWS Transit Gateway

3. You are an AWS intern at XYZ company, and you have to implement the concept of CloudWatch for your AWS resource to monitor its logs.

   1. What needs to be done:

      1. Create an instance and deploy an nginx web server on that instance.

      2. Create a CloudWatch and connect it with your nginx server to monitor

ANS:

Read this blog for the Task/Project guide: Nginx CloudWatch Monitoring

Step 1: Launch an EC2 Instance

• Go to AWS Console → EC2 → Launch Instance.

• Configure the following:

  • Name: nginx-server

  • AMI: Ubuntu 22.04 LTS

  • Instance Type: t2.micro (Free Tier eligible)

  • Key pair: Create or select an existing one.

  • Security Group: Allow inbound rules for:

    • Port 22 (SSH)

    • Port 80 (HTTP)

• Launch the instance.

Step 2: Install and Start Nginx

Connect to the EC2 instance using SSH:

ssh -i your-key.pem ubuntu@your-ec2-public-ip

Update and install nginx:

sudo apt update -y
sudo apt install nginx -y

Start and enable nginx service:

sudo systemctl start nginx
sudo systemctl enable nginx

Verify by accessing the public IP address of the EC2 instance in a web browser. You should see the Nginx welcome page.

Step 3: Create an IAM Role for CloudWatch

• Navigate to IAM → Roles → Create Role.

• Trusted Entity: EC2.

• Attach the following policies:

  • CloudWatchAgentServerPolicy

  • AmazonSSMManagedInstanceCore (optional but recommended for easier management)

• Name the role appropriately, for example: EC2CloudWatchAgentRole.

• Attach this role to the running EC2 instance (EC2 → Actions → Security → Modify IAM Role).

Step 4: Install the CloudWatch Agent

Since amazon-cloudwatch-agent is not available via apt on Ubuntu, install it manually:

cd /tmp
wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Verify the installation:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

Step 5: Create a CloudWatch Agent Configuration File

Create a configuration file to specify which logs to collect:

sudo vim /opt/aws/amazon-cloudwatch-agent/bin/config.json

Paste the following configuration:

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/nginx/access.log",
            "log_group_name": "nginx-access-logs",
            "log_stream_name": "{instance_id}-access",
            "timezone": "UTC"
          },
          {
            "file_path": "/var/log/nginx/error.log",
            "log_group_name": "nginx-error-logs",
            "log_stream_name": "{instance_id}-error",
            "timezone": "UTC"
          }
        ]
      }
    }
  }
}

Save and exit the file.

Step 6: Start the CloudWatch Agent

Start the CloudWatch Agent with the created configuration:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m ec2 \
  -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json \
  -s

This will start collecting the nginx logs and push them to CloudWatch.

Step 7: Verify in AWS Console

• Go to AWS Console → CloudWatch → Log Groups.

• You should see two log groups:

  • nginx-access-logs

    

    

    

• You can now monitor the nginx access and error logs directly from the CloudWatch console.

Final Outcome

• A running EC2 instance with an nginx web server installed and active.

• CloudWatch Agent installed and configured to collect nginx logs.

• Log streams visible and accessible in the AWS CloudWatch service for monitoring.

This guide provides detailed steps to establish a network architecture for ByteConnect Inc., where multiple departments operate in isolated Virtual Private Clouds (VPCs) within AWS. The goal is to enable seamless communication between instances in these VPCs using AWS Transit Gateway, as depicted in an architecture with three VPCs, each containing a public subnet and an EC2 instance, interconnected via a Transit Gateway.

Prerequisites

Before starting, ensure you have:

• An AWS account with permissions to create and manage VPCs, subnets, Internet Gateways, route tables, EC2 instances, and Transit Gateways (AWS Identity and Access Management).

• Basic knowledge of navigating the AWS Management Console.

• A key pair created in the EC2 console for SSH access to instances (Create a key pair).

• All actions performed in the same AWS Region (e.g., us-east-1) for consistency.

Architecture Overview

The architecture consists of:

• Three VPCs:

  • test-vpc-1: CIDR 12.0.0.0/16, with public subnet 12.0.0.0/24.

  • test-vpc-2: CIDR 13.0.0.0/16, with public subnet 13.0.1.0/24.

  • test-vpc-3: CIDR 14.0.0.0/16, with public subnet 14.0.1.0/24.

• Public Subnets: Each hosts an EC2 instance with a public IP for accessibility.

• Internet Gateways: Enable internet access for public subnets.

• EC2 Instances: One per VPC for testing communication.

• AWS Transit Gateway: Central hub connecting all VPCs, facilitating inter-VPC communication.

• Route Tables: Configured to route traffic between VPCs via the Transit Gateway.

• Security Groups: Allow necessary traffic (e.g., ICMP, SSH) between VPCs.

The Transit Gateway simplifies network management by acting as a hub, eliminating the need for complex VPC peering. Each VPC’s route table will include routes to the other VPCs’ CIDR blocks, pointing to the Transit Gateway.

Image credit to Rahul Wagh at YT

Step-by-Step Instructions

Step 1: Create Three VPCs

1. Log in to the AWS Management Console.

2. Navigate to the VPC dashboard under “Networking & Content Delivery.”

3. Click Create VPC and configure:

   • VPC 1:

     • Name tag: test-vpc-1

     • IPv4 CIDR block: 12.0.0.0/16

     • Tenancy: Default

   • VPC 2:

     • Name tag: test-vpc-2

     • IPv4 CIDR block: 13.0.0.0/16

     • Tenancy: Default

   • VPC 3:

     • Name tag: test-vpc-3

     • IPv4 CIDR block: 14.0.0.0/16

     • Tenancy: Default

4. Click Create VPC for each and verify creation in the VPC dashboard.

Step 2: Create Public Subnets

1. In the VPC dashboard, select Subnets and click Create subnet.

2. Configure one subnet per VPC:

   • VPC 1:

     • VPC: test-vpc-1

     • Subnet name: public-subnet-1

     • Availability Zone: e.g., us-east-1a

     • IPv4 CIDR block: 12.0.0.0/24

   • VPC 2:

     • VPC: test-vpc-2

     • Subnet name: public-subnet-2

     • Availability Zone: us-east-1a

     • IPv4 CIDR block: 13.0.1.0/24

   • VPC 3:

     • VPC: test-vpc-3

     • Subnet name: public-subnet-3

     • Availability Zone: us-east-1a

     • IPv4 CIDR block: 14.0.1.0/24

3. Click Create subnet for each and confirm in the Subnets list.

Step 3: Create and Attach Internet Gateways

1. In the VPC dashboard, select Internet Gateways and click Create internet gateway.

2. Create and attach one per VPC:

   • VPC 1:

     • Name tag: igw-test-vpc-1

     • After creation, select it, click Actions > Attach to VPC, and choose test-vpc-1.

   • VPC 2:

     • Name tag: igw-test-vpc-2

     • Attach to test-vpc-2.

   • VPC 3:

     • Name tag: igw-test-vpc-3

     • Attach to test-vpc-3.

3. Verify attachments in the Internet Gateways section.

Step 4: Create and Configure Route Tables

1. In the VPC dashboard, select Route Tables and click Create route table.

2. Create one per VPC:

   • VPC 1: Name: rt-test-vpc-1, VPC: test-vpc-1

   • VPC 2: Name: rt-test-vpc-2, VPC: test-vpc-2

   • VPC 3: Name: rt-test-vpc-3, VPC: test-vpc-3

3. For each route table:

   • Select the route table, go to Routes tab, and click Edit routes.

   • Add a route:

     • Destination: 0.0.0.0/0

     • Target: Internet Gateway (e.g., igw-test-vpc-1 for rt-test-vpc-1)

   • Click Save routes.

4. Associate each route table with its public subnet:

   • Select the route table, go to Subnet associations, click Edit subnet associations.

   • Select the corresponding subnet (e.g., public-subnet-1 for rt-test-vpc-1) and save.

Step 5: Launch EC2 Instances

1. Navigate to the EC2 dashboard and click Launch instances.

2. Configure one instance per VPC:

   • AMI: Choose Amazon Linux 2 or similar.

   • Instance type: t2.micro (free tier eligible).

   • Network settings:

     • VPC 1: VPC: test-vpc-1, Subnet: public-subnet-1, Auto-assign Public IP: Enable

     • VPC 2: VPC: test-vpc-2, Subnet: public-subnet-2, Auto-assign Public IP: Enable

     • VPC 3: VPC: test-vpc-3, Subnet: public-subnet-3, Auto-assign Public IP: Enable

   • Storage: Default settings.

   • Tags: Add Name tag (e.g., instance-vpc-1).

   • Security group: Create a new security group with:

     • Inbound rule: SSH (port 22) from your IP (e.g., 203.0.113.0/32).

     • Inbound rule: All ICMP - IPv4 from anywhere (0.0.0.0/0) for testing.

   • Key pair: Select an existing key pair or create a new one.

3. Launch each instance and note their public and private IPs.

Step 6: Create AWS Transit Gateway

1. In the VPC dashboard, select Transit Gateways and click Create Transit Gateway.

2. Configure:

   • Name tag: test-tgw

   • Description: Optional (e.g., “Transit Gateway for ByteConnect”)

   • Amazon side ASN: Default (64512-65534)

   • DNS support: Enable

   • Default route table association/propagation: Enable

3. Click Create Transit Gateway and wait for the status to become “Available” (may take a few minutes).

Step 7: Attach VPCs to Transit Gateway

1. In the VPC dashboard, select Transit Gateway Attachments and click Create Transit Gateway Attachment.

2. For each VPC:

   • Transit Gateway ID: test-tgw

   • Attachment type: VPC

   • VPC ID: Select test-vpc-1, test-vpc-2, or test-vpc-3

   • Subnet IDs: Select the public subnet (e.g., public-subnet-1 for test-vpc-1)

3. Click Create attachment for each VPC and verify attachments in the list.

Step 8: Configure Route Tables for Inter-VPC Communication

1. In the VPC dashboard, select Route Tables.

2. Update each route table to include routes to other VPCs’ CIDR blocks:

   • rt-test-vpc-1:

     • Destination: 13.0.0.0/16, Target: Transit Gateway (test-tgw)

     • Destination: 14.0.0.0/16, Target: Transit Gateway (test-tgw)

   • rt-test-vpc-2:

     • Destination: 12.0.0.0/16, Target: Transit Gateway (test-tgw)

     • Destination: 14.0.0.0/16, Target: Transit Gateway (test-tgw)

   • rt-test-vpc-3:

     • Destination: 12.0.0.0/16, Target: Transit Gateway (test-tgw)

     • Destination: 13.0.0.0/16, Target: Transit Gateway (test-tgw)

3. For each route table, click Edit routes, add the routes, and click Save routes.

Step 9: Configure Security Groups for Inter-VPC Traffic

1. In the EC2 dashboard, select Security Groups.

2. For each instance’s security group, add inbound rules to allow traffic from other VPCs:

   • Instance in test-vpc-1:

     • Type: All ICMP - IPv4, Source: 13.0.0.0/16

     • Type: All ICMP - IPv4, Source: 14.0.0.0/16

   • Instance in test-vpc-2:

     • Type: All ICMP - IPv4, Source: 12.0.0.0/16

     • Type: All ICMP - IPv4, Source: 14.0.0.0/16

   • Instance in test-vpc-3:

     • Type: All ICMP - IPv4, Source: 12.0.0.0/16

     • Type: All ICMP - IPv4, Source: 13.0.0.0/16

3. Optionally, add rules for other protocols (e.g., TCP port 80 for HTTP) based on application needs.

4. Save changes for each security group.

Step 10: Test Connectivity

1. SSH into an EC2 instance (e.g., instance-vpc-1) using its public IP:

    ssh -i your-key.pem ec2-user@<public-ip>
   

2. Ping the private IP of an instance in another VPC (e.g., instance-vpc-2):

    ping <private-ip-of-instance-vpc-2>
   

3. Curl the private IP of an instance in another VPC instance (e.g., instance-vpc-2):

    curl <private-ip-of-instance-vpc-2>
   

4. Repeat from other instances to confirm bidirectional communication.

5. If pings fail, verify:

   • Route table entries are correct.

   • Transit Gateway attachments are in “Available” state.

   • Security groups allow ICMP traffic.

   • Network ACLs (if customized) permit the traffic.

Additional Notes

• CIDR Blocks: The CIDR blocks (12.0.0.0/16, 13.0.0.0/16, 14.0.0.0/16) are chosen to avoid overlap, which is critical for proper routing. Adjust as needed for your environment.

• Public Subnets: These subnets are configured for internet access, suitable for testing. For production, consider private subnets with NAT Gateways for enhanced security.

• Transit Gateway Route Table: By default, enabling route table propagation associates all attachments with the Transit Gateway’s default route table, allowing communication between VPCs. For isolation, create separate route tables (Transit Gateway Route Tables).

• Security Considerations: Restrict security group rules to specific IPs or CIDRs in production. Use network ACLs for additional control if needed.

• Scalability: Additional VPCs can be attached to the Transit Gateway without modifying existing configurations, making this architecture scalable.

• Cost: Transit Gateway incurs charges based on attachments and data transfer. Review AWS Transit Gateway Pricing for cost estimates.

Troubleshooting

• Ping Fails: Check security group rules, route tables, and Transit Gateway attachment status. Ensure instances are running and reachable.

• Attachment Issues: Verify subnets are correctly associated with Transit Gateway attachments. Attachments should be in “Available” state.

• Routing Errors: Confirm no overlapping CIDR blocks and that routes point to the correct Transit Gateway ID.

Example Configuration Summary

This setup ensures that instances in test-vpc-1, test-vpc-2, and test-vpc-3 can communicate seamlessly, fulfilling ByteConnect Inc.’s requirement for inter-departmental VPC connectivity.

Key Citations

• Security group rules for different use cases

• Troubleshoot VPC-to-VPC connectivity through a transit gateway

• through transit gateway, I can't send ping request to instance in the other vpc

• Get started with using Amazon VPC Transit Gateways

• Security group rules

• Unrestricted ICMP Access

• I used All ICMP policy in Security Group to make ping work

• Cannot ping AWS EC2 instance

Output Images/Images

1. VPCs:

   

2. Subnets:

   

3. Internet Gateways:

   

4. Route Tables:

   

5. EC2 Instances:

   

6. EC2 Security Group Inbound Rule:

   

7. Transit Gateway:

   

8. Transit Gateway Attachments:

   

9. Dept-A-Server:

   

10. Dept-B-Server:

    

11. Dept-C-Server:

    

Introduction

In a cost-conscious cloud environment, optimizing resource usage is critical. One effective way to manage EC2 instance costs is to automate the process of starting and stopping instances during business hours. In this blog, we’ll walk through a clean and efficient approach using AWS Lambda and EventBridge, with Python (Boto3) to control instances based on tags.

Objective

To automatically start or stop specific EC2 instances during defined hours using a Lambda function. Instances are identified using a custom tag, enabling flexibility and control.

Scenario

You're an AWS expert managing a budget-friendly project with EC2 instances. To save money, you're using AWS Lambda to automatically start and stop instances when they're not needed during non-business hours. 👇

What needs to be done:

• Create an AWS Lambda function that will start/stop instances based on their instance tag.

Prerequisites

Before implementing the solution, ensure the following:

1. Tagged EC2 Instances

   • Tag Key: AutoSchedule

   • Tag Value: true

     

2. IAM Role for Lambda
   Create and assign an IAM role to your Lambda function with the following permissions, or EC2 Full Access:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "ec2:DescribeInstances",
            "ec2:StartInstances",
            "ec2:StopInstances"
          ],
          "Resource": "*"
        }
      ]
    }
   

Lambda Function Code

Below is the Python code using Boto3 to start or stop EC2 instances based on the AutoSchedule tag.

import boto3

ec2 = boto3.client('ec2')

def lambda_handler(event, context):
    action = event.get('action')  # 'start' or 'stop'
    if action not in ['start', 'stop']:
        return {'statusCode': 400, 'body': "Invalid action."}

    filters = [
        {'Name': 'tag:AutoSchedule', 'Values': ['true']},
        {'Name': 'instance-state-name', 'Values': ['stopped'] if action == 'start' else ['running']}
    ]

    instances = ec2.describe_instances(Filters=filters)
    instance_ids = [i['InstanceId'] for r in instances['Reservations'] for i in r['Instances']]

    if not instance_ids:
        return {'statusCode': 200, 'body': f"No instances to {action}."}

    if action == 'start':
        ec2.start_instances(InstanceIds=instance_ids)
    else:
        ec2.stop_instances(InstanceIds=instance_ids)

    return {'statusCode': 200, 'body': f"{action.capitalize()}ed: {instance_ids}"}

Manual Testing

Before scheduling, it's important to verify the Lambda function manually.

Steps:

1. Navigate to your Lambda function in the AWS console.

2. Click Test > Create new test event.

3. Use the following test input:

   • For starting:

       {
         "action": "start"
       }
     

   • For stopping:

       {
         "action": "stop"
       }
     

4. Execute the test and check the results in the output logs.

Automating with EventBridge

To run this function at specific times daily, use EventBridge (CloudWatch Events).

Schedule to Start Instances

• Go to EventBridge → Rules → Create Rule

• Name: StartEC2Instances

• Schedule pattern: cron(0 9 * * ? *) (Every day at 9 AM UTC)

  

• Target: Your Lambda function

• Input JSON:

    {
      "action": "start"
    }
  

  

Schedule to Stop Instances

• Name: StopEC2Instances

• Schedule pattern: cron(0 18 * * ? *) (Every day at 6 PM UTC)

  

• Input JSON:

    {
      "action": "stop"
    }
  

  

Guide to set up SNS notifications for EC2 start/stop events:

Step 1: Create an SNS Topic

1. Open the SNS Console.

2. Click Create topic.

3. Select Standard as the topic type.

4. Enter the following:

   • Name: ec2-notifications

5. Leave the default settings for other options and click Create topic.

Step 2: Subscribe to the Topic (Email)

1. After the topic is created, click on the topic name.

2. Click Create subscription.

3. Configure the subscription:

   • Protocol: Email

   • Endpoint: Enter your email address (e.g., you@example.com).

4. Click Create subscription.

5. Check your email inbox and confirm the subscription by clicking the confirmation link in the email.

   

   

Step 3: Create CloudWatch/EventBridge Rule for EC2 Start & Stop Notifications

You need to create two rules: one for EC2 instance start notifications and another for EC2 instance stop notifications.

1. Create Rule for EC2 Instance Start Notification

1. Go to the Amazon CloudWatch Console.

2. In the left sidebar, select Rules under Events/EventBridge (depending on your UI version). For older UIs, navigate to Events > Rules.

3. Click Create Rule.

4. In the Rule details section:

   • Name: EC2InstanceStartNotify

   • Description (optional): Notify when EC2 instance starts.

5. Under Event Pattern, select Event Pattern and paste the following:

{
  "source": ["aws.ec2"],
  "detail-type": ["EC2 Instance State-change Notification"],
  "detail": {
    "state": ["running"]
  }
}

6. Add the target:

   • Click Add target.

   • Select SNS topic.

   • Choose the ec2-notifications topic you created earlier.

     

7. Click Create to finalize the rule.

   

2. Create Rule for EC2 Instance Stop Notification

1. Repeat steps 1–4 to create another rule, but with the following changes:

   • Name: EC2InstanceStopNotify

   • Description (optional): Notify when EC2 instance stops.

2. Under Event Pattern, paste the following for EC2 stop notifications:

{
  "source": ["aws.ec2"],
  "detail-type": ["EC2 Instance State-change Notification"],
  "detail": {
    "state": ["stopped"]
  }
}

3. Add the target:

   • Click Add target.

   • Select SNS topic.

   • Choose the ec2-notifications topic you created earlier.

     

4. Click Create to finalize the rule.

   

5. EC2 Start Email Notification:

   

6. EC2 Stop Email Notification:

   

Conclusion

By combining AWS Lambda, EventBridge scheduling, and SNS notifications, you now have a fully serverless, automated framework for managing your EC2 instance lifecycle:

• Cost optimization: Instances only run during business hours, reducing your hourly compute charges.

• Operational visibility: SNS alerts notify you immediately whenever an instance starts or stops.

• Scalability & maintainability: EventBridge cron rules and Lambda functions require no servers to manage, and you can adjust tags or schedules without changing infrastructure.

Next steps and best practices:

1. Review your CloudWatch Logs and SNS subscription metrics to validate that notifications are delivered as expected.

2. Implement IAM least-privilege for your Lambda execution role and refine tag-based permissions.

3. Consider adding CloudWatch alarms on Lambda errors or unexpected instance-state counts to proactively detect issues.

With these measures in place, you’ll maintain tight cost control and real-time awareness of your EC2 fleet. If you’ve adopted a similar pattern or have other ideas for optimizing AWS resource usage, I’d welcome your insights—let’s continue sharing best practices.

• Deploy this: Two-tier application

Ans:

1. Created AWS RDS with MySQL:

   

2. Created EC2 template with following user data:

    #!/bin/bash
   
    # Update system
    sudo apt update -y
   
    # Install Docker & MySQL client
    sudo apt install docker.io mysql-client -y
   
    # Add 'ubuntu' user to Docker group
    sudo usermod -aG docker ubuntu
   
    # Enable Docker service
    sudo systemctl enable docker
    sudo systemctl start docker
   
    # Wait until the RDS instance is ready to accept connections (optional but useful)
    until mysql -u admin -h database-1.cdsswsgcuo0c.us-east-1.rds.amazonaws.com -P 3306 -pdevops2004 -e "SELECT 1;" &>/dev/null; do
      echo "Waiting for MySQL..."
      sleep 5
    done
   
    # Create the database if it doesn't exist
    mysql -u admin -h database-1.cdsswsgcuo0c.us-east-1.rds.amazonaws.com -P 3306 -pdevops2004 -e "CREATE DATABASE IF NOT EXISTS flaskdb;"
   
    # Pull your Flask app Docker image
    sudo docker pull amitabhdevops/aws-flask-app:latest
   
    # Run the Flask app
    sudo docker run -d \
      --name flaskapp \
      -e MYSQL_HOST=database-1.cdsswsgcuo0c.us-east-1.rds.amazonaws.com \
      -e MYSQL_USER=admin \
      -e MYSQL_PASSWORD=devops2004 \
      -e MYSQL_DB=flaskdb \
      -p 80:5000 \
      amitabhdevops/aws-flask-app:latest
   

3. Then, Created AWS AutoScaling Group with this template

4. After that, I created an IAM role for EC2 service with RDS and CloudWatch full access, and attached it to my EC2 Instance

5. Modified the RDS to connect with my Instance

6. Connected to first instance, which is created from ASG

7. and checked the databases content

8. output images:

   

   

   

1. Read about AWS RDS, DynamoDB, and AWS Lambda, and write a post on LinkedIn with an example in your own words.
   Ans:

   • AWS RDS: Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks.

   • AWS DynamoDB: Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. You can use Amazon DynamoDB to create a database table that can store and retrieve any amount of data, and serve any level of request traffic. Amazon DynamoDB automatically spreads the data and traffic for the table over a sufficient number of servers to handle the request capacity specified by the customer and the amount of data stored, while maintaining consistent and fast performance.

   • AWS Lambda: With AWS Lambda, you can run code without provisioning or managing servers. You pay only for the compute time that you consume—there's no charge when your code isn't running. You can run code for virtually any type of application or backend service—all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability.

2. You are part of a team responsible for migrating the database of an existing e-commerce platform to Amazon RDS. The goal is to improve scalability, performance, and manageability. The current setup uses a self-managed MySQL database on an on-premises server. 👇

   What needs to be done:

   • Set up and configure a MySQL database on AWS RDS, ensuring optimal performance.

   • Establish a connection between the RDS instance and your EC2 environment

Ans:

1. Created an RDS with connecting to an EC2 Instance

2. After the creation of RDS, connected to the EC2 instance that is attached to RDS and updated the system

3. After that, I created an IAM role for EC2 service with RDS and CloudWatch full access, and attached it to my EC2 Instance.

4. After that, I launched the EC2 and installed the MySQL client using the command sudo apt install mysql-client -y

5. and then ran this command and entered the password, and connected to my RDS MySQL command: mysql -u admin -h [database-1.c3w68q0ggwwn.eu-west-1.rds.amazonaws.com](<http://database-1.c3w68q0ggwwn.eu-west-1.rds.amazonaws.com/>) -P 3306 -p

   where database-1.c3w68q0ggwwn.eu-west-1.rds.amazonaws.com It is the EndPoint of my RDS DB.

   

6. Then I created a database ‘aws’ using the command create database aws;

7. To check whether the database is created or not, I ran this to show all databases: show databases;

8. Then I used that database using use aws

9. After that, I created the table using the command CREATE TABLE learners (learner_id INT, learner_name VARCHAR(50));

10. then i inserted two data into the table

    1. insert into learners (learner_id,learner_name) values (1,"shubham");

    2. insert into learners (learner_id,learner_name) values (2,"Amitabh");

    3. Output select * from learners;

       

Amazon Relational Database Service (RDS) is a managed service that simplifies the process of setting up, operating, and scaling relational databases in the cloud. It offers cost-efficient and scalable capacity while automating time-consuming database administration tasks, like backups, software patching, and monitoring. This allows developers to focus on their applications and business logic rather than managing the database infrastructure.

Key features of Amazon RDS:

• Managed Service:

  RDS handles many database administration tasks, including provisioning, backups, software updates, and patching.

• Scalable Capacity:

  You can easily scale the compute resources and storage capacity of your database instances to meet your needs.

• High Availability:

  RDS offers features like Multi-AZ deployments and read replicas to enhance database availability and improve performance.

• Security:

  RDS provides robust security features, including encryption at rest and in transit, network isolation, and access control.

• Database Engines:

  RDS supports various popular database engines, such as MySQL, PostgreSQL, Oracle, SQL Server, and MariaDB.

• Aurora:

  RDS also includes Amazon Aurora, a fully managed database engine that is built for the cloud and is compatible with MySQL and PostgreSQL.

• Cost-Effective:

  You pay only for the resources you use, with no upfront investments required.

• Automated Backup and Recovery:

  RDS automatically backs up your database, allowing you to restore to a previous point in time.

• Compatibility:

  You can use existing applications and tools with your RDS databases.

In essence, Amazon RDS provides a convenient and reliable way to manage relational databases in the cloud, allowing you to focus on application development and business goals while minimizing the effort required for database administration.

AWS DynamoDB:

AWS DynamoDB is a serverless, NoSQL database service provided by Amazon Web Services (AWS). It supports key-value and document data structures, enabling developers to build modern, scalable applications. DynamoDB is designed for high performance and can handle virtually any size of data.

Here's a more detailed breakdown:

• NoSQL Database:

  DynamoDB is a NoSQL database, meaning it doesn't rely on traditional relational database models like SQL. Instead, it uses a key-value or document-based data structure.

• Serverless:

  As a serverless service, AWS manages the underlying infrastructure, so developers don't need to worry about server provisioning, patching, or maintenance.

• Key-Value and Document Data Models:

  DynamoDB supports both key-value and document data models, allowing you to store data in a structured format.

• Scalable:

  DynamoDB can scale to handle a wide range of workloads, from small, single-user applications to large, globally distributed systems. It automatically scales horizontally to accommodate growing data volumes and traffic.

• High Performance:

  DynamoDB is designed for fast data access, with single-digit millisecond response times.

• Managed Service:

  AWS manages the database, so developers can focus on building applications instead of managing the database itself.

• Global Tables:

  DynamoDB supports Global Tables, allowing for multi-Region replication of data, ensuring high availability and low latency for globally distributed applications.

• Integration with Other AWS Services:

  DynamoDB integrates well with other AWS services like AWS Lambda, Kinesis, and S3, enabling developers to build complete solutions.

In essence, DynamoDB is a powerful, scalable, and managed NoSQL database service that is ideal for building modern, serverless applications on AWS.

AWS Lambda:

AWS Lambda is a service that lets you run code without managing servers. It's an example of serverless computing, also known as Function as a Service (FaaS).

You can use Lambda to:

• Extend other AWS services

• Create your own backend services

• Process streams of data

• Call APIs

• Integrate with other AWS services

• Run code for applications that need to scale up and down

You can write Lambda functions in languages like Node.js, Python, Go, and Java. You can use tools like AWS SAM or Docker CLI to build, test, and deploy your functions.

AWS Lambda is part of Amazon Web Services (AWS).

AWS Lambda is a serverless compute service that lets you run code without managing servers. Lambda functions are pieces of code that perform specific tasks, such as processing data streams or responding to HTTP requests. You can trigger these functions with various events, like changes in data, user actions, or scheduled tasks. Lambda manages the underlying infrastructure, including scaling and maintenance, allowing you to focus on your code.

Here's a more detailed explanation:

• Serverless Computing:

  Lambda eliminates the need to provision and manage servers yourself. You only pay for the compute time your code uses, and there are no charges when your code is not running.

• Event-Driven:

  Lambda functions are triggered by events, such as a file being uploaded to an S3 bucket, a message arriving in an SNS queue, or a request being sent to an API Gateway endpoint.

• Scalability:

  Lambda automatically scales your code up or down based on demand, ensuring that it can handle a wide range of workloads without manual intervention.

• Code Execution:

  You upload your code (in a variety of supported languages like Node.js, Python, Java, and Go) to Lambda, and it's executed in response to the triggered events.

• Infrastructure Management:

  Lambda handles all the underlying infrastructure management, including server maintenance, capacity provisioning, and automatic scaling.

• Versatility:

  You can use Lambda for a wide range of applications, from simple backend logic to complex data processing pipelines.

Key Features and Concepts:

• Object Storage: S3 stores data as objects, which are essentially files along with optional metadata.

• Buckets: S3 uses buckets as containers for storing objects.

• Scalability: S3 is designed to handle massive amounts of data and traffic.

• Data Durability: S3 offers high durability, with a stated goal of 11 nines (99.999999999%) data durability.

• Data Availability: S3 is designed for high availability, with a stated goal of 99.99%.

• Security: S3 provides various security features, including encryption, access control lists (ACLs), and IAM policies.

• Storage Classes: S3 offers different storage classes to optimize costs and performance based on access frequency and data lifecycle.

• Access Points: S3 Access Points allow you to create virtualized endpoints for accessing S3 data, which can improve performance and simplify application architecture.

• Lifecycle Management: S3 provides lifecycle management features to automate object transitions between storage classes and deletion.

• Replication: S3 supports cross-region replication, allowing you to replicate data across different AWS regions for redundancy and disaster recovery.

• Data Residency: S3 offers features for data residency, allowing you to store data in specific data perimeters.

Use Cases:

• Storing and retrieving data for various applications:

  S3 is used for storing and retrieving data for a wide range of applications, including cloud applications, dynamic websites, content distribution, mobile and gaming applications, and big data analytics.

• Building data lakes:

  S3 can be used to build data lakes, which are centralized repositories for storing structured and unstructured data at any scale.

• Storing static website content:

  S3 can be used to store static website content, which can then be served to users.

• Storing backups and archives:

  S3 can be used as a cost-effective storage solution for backups and archives.

Getting Started with S3:

1. Create an AWS Account: You'll need an AWS account to use S3.

2. Create an S3 Bucket: Create a bucket to store your objects.

3. Upload Objects: Upload your files or data to the bucket.

4. Configure Access: Configure access control to manage who can access your objects.

5. Choose a Storage Class: Select the appropriate storage class for your data.

6. Explore S3 Features: Learn about and utilize other S3 features like lifecycle management, replication, and access points.