Tips ==== https://github.com/boto/boto https://github.com/bitly/asyncdynamo https://pypi.python.org/pypi/dynamodb-mapper/1.1.0 https://pypi.python.org/pypi/ddbmock http://boto.readthedocs.org/en/latest/dynamodb2_tut.html Amazon upload ------------- http://stackoverflow.com/questions/670442/asynchronous-file-upload-to-amazon-s3-with-django https://github.com/jezdez/django-queued-storage https://github.com/sbc/django-uploadify-s3 https://github.com/burgalon/plupload-s3mixin http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html https://aws.amazon.com/items/1434?externalID=1434 https://django-storages.readthedocs.org/en/latest/backends/amazon-S3.html Django S3 ---------- https://github.com/etianen/django-s3-storage https://django-storages.readthedocs.org/en/latest/index.html Time Limited Signed UR ---------------------- http://www.bucketexplorer.com/documentation/amazon-s3--how-to-generate-url-for-amazon-s3-files.html http://stackoverflow.com/questions/17831535/how-to-generate-file-link-without-expiry AWS SDK for Python (Boto) ------------------------- http://aws.amazon.com/sdk-for-python/ http://boto.readthedocs.org/en/latest/index.html http://aws.amazon.com/python/ http://stackoverflow.com/questions/4993439/how-can-i-access-s3-files-in-python-using-urls http://sendapatch.se/projects/simples3/ http://stackoverflow.com/questions/11026719/is-there-a-way-to-serve-s3-files-directly-to-the-user-with-a-url-that-cant-be-s sign URLs with an IP -------------------- CloudFront http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/GettingStarted.html http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PrivateContent.html http://improve.dk/how-to-set-up-and-serve-private-content-using-s3/ session based authorization http://stackoverflow.com/questions/12279056/rails-allow-download-of-files-stored-on-s3-without-showing-the-actual-s3-url-to download private file https://medium.com/@hiromitz/generate-expiring-amazon-s3-link-with-custom-file-name-c277975c3b8d https://gist.github.com/hiromitz/9321852 https://pypi.python.org/pypi/Ax_Handoff/1.1.3 https://pypi.python.org/pypi/s3url/0.1.6 Boto ----- http://boto.readthedocs.org/en/latest/index.html http://aws.amazon.com/developers/getting-started/python/ http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls-overview.html `http://www.networkautomation.com/automate/urc/resources/livedocs/am/10/Technical_Reference/Actions___Activities/Amazon_S3/S3_-_Get_Predesigned_URL.htm` Amazon EC2 ---------- Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. Amazon EBS ---------- Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances in the AWS Cloud. Each Amazon EBS volume is automatically replicated within its Availability Zone to protect you from component failure, offering high availability and durability. Amazon EBS volumes offer the consistent and low-latency performance needed to run your workloads. With Amazon EBS, you can scale your usage up or down within minutes – all while paying a low price for only what you provision. http://alestic.com/2012/01/ec2-ebs-boot-recommended http://tiger-fish.com/blog/how-boot-amazon-ec2-instance-ebs-volume http://thomas.broxrost.com/2008/08/21/persistent-django-on-amazon-ec2-and-ebs-the-easy-way/ http://stackoverflow.com/questions/10390244/how-to-set-up-a-django-project-with-django-storages-and-amazon-s3-but-with-diff https://github.com/mstarinteractive/django-s3storage https://github.com/mstarinteractive/django-s3storage/blob/master/example_settings.py http://tartarus.org/james/diary/2013/07/18/fun-with-django-storage-backends http://djangotricks.blogspot.de/2013/12/how-to-store-your-media-files-in-amazon.html https://github.com/pcraciunoiu/django-s3sync How to serve your media files via Amazon's Simple Storage Service http://stackoverflow.com/questions/11403063/setting-media-url-for-django-heroku-app-amazon-s3 https://github.com/django-compressor/django-compressor http://stackoverflow.com/questions/11403063/setting-media-url-for-django-heroku-app-amazon-s3 http://stackoverflow.com/questions/10390244/how-to-set-up-a-django-project-with-django-storages-and-amazon-s3-but-with-diff http://www.caktusgroup.com/blog/2014/11/10/Using-Amazon-S3-to-store-your-Django-sites-static-and-media-files/ http://martinbrochhaus.com/s3.html http://stackoverflow.com/questions/9464038/redis-celery-configuration-over-amazon-ec2 http://stackoverflow.com/questions/14283021/how-to-use-django-celery-rq-worker-to-execute-a-video-filetype-conversion-ffm http://django-storages.readthedocs.org/en/latest/ https://docs.djangoproject.com/en/1.7/howto/static-files/deployment/#staticfiles-from-cdn http://www.caktusgroup.com/blog/2014/11/10/Using-Amazon-S3-to-store-your-Django-sites-static-and-media-files/ http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html http://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html https://aws.amazon.com/items/1434?externalID=1434 AWS RDS Postgres DB instance ---------------------------- http://aws.amazon.com/rds/postgresql/ http://aws.amazon.com/about-aws/whats-new/2013/12/11/aws-elastic-beanstalk-adds-background-task-handling-and-rds-postgresql-support/ http://stackoverflow.com/questions/26043706/how-to-use-boto-to-launch-an-elastic-beanstalk-with-an-rds-resource http://stackoverflow.com/questions/25946723/aws-cli-create-rds-with-elasticbeanstalk-create-environment/25963800#25963800 http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_flask.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python.rds.html http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html http://stackoverflow.com/questions/13424267/setting-up-django-and-postgresql-on-two-different-ec2-instances http://stackoverflow.com/questions/12850550/postgresql-for-django-on-elastic-beanstalk http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances http://www.holovaty.com/writing/aws-notes/ http://stackoverflow.com/questions/22599367/deploy-django-using-mysql-to-aws-ec2-and-rds http://stackoverflow.com/questions/20914706/aws-elastic-beanstalk-hosting-postresql-on-deployed-ec2-server-with-django http://www.quora.com/If-I-have-an-AWS-RDS-Postgres-DB-instance-do-I-also-need-to-install-Postgres-in-the-EC2-instance-that-has-my-Django-application-in-it http://stackoverflow.com/questions/25740502/aws-can-a-beanstalk-instance-be-deployed-with-a-postgres-rds http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.RDS.html https://github.com/tornadoweb/tornado/wiki/Links http://stackoverflow.com/questions/11638135/amazon-aws-python-webframework-dynamodb http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.loggingS3.title.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html#command-options-python http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_custom_container.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.platforms.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.concepts.architecture.html http://docs.aws.amazon.com/general/latest/gr/rande.html?r=1166 http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_SettingUpUser.html http://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.html http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPractices.html http://aws.amazon.com/code/6752709412171743 Deploying a Django app on Amazon EC2 instance --------------------------------------------- http://agiliq.com/blog/2014/08/deploying-a-django-app-on-amazon-ec2-instance/ http://thomas.broxrost.com/2008/08/21/persistent-django-on-amazon-ec2-and-ebs-the-easy-way/ http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_django.html https://ashokfernandez.wordpress.com/2014/03/11/deploying-a-django-app-to-amazon-aws-with-nginx-gunicorn-git/ https://github.com/ashokfernandez/Django-Fabric-AWS---amazon_app http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_flask.html Amazon ECS ---------- http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html Identity and Access Management ------------------------------ https://console.aws.amazon.com/iam/home#home Before the Amazon ECS agent can register container instance into a cluster, the agent must know which account credentials to use. You can create an IAM role that allows the agent to know which account it should register the container instance with. When you launch an instance with the Amazon ECS-optimized AMI provided by Amazon using this role, the agent automatically registers the container instance into your default cluster. The Amazon ECS container agent also makes calls to the Amazon EC2 and Elastic Load Balancing APIs on your behalf, so container instances can be registered and deregistered with load balancers. Before you can attach a load balancer to an Amazon ECS service, you must create an IAM role for your services to use before you start them. This requirement applies to any Amazon ECS service that you plan to use with a load balancer. http://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html https://console.aws.amazon.com/iam/home#roles Amazon EC2 Role for EC2 Container Service Role to allow EC2 instances in an Amazon ECS cluster to access Amazon ECS. https://console.aws.amazon.com/ec2/ http://www.prokerala.com/travel/distance/from-california/to-vancouver-usa/ Distance To Vancouver From Oregon is: 1692 miles / 2723.01 km / 1470.31 nautical miles Distance To Virginia From Vancouver is: 1725 miles / 2776.12 km / 1498.98 nautical miles Distance To Vancouver From California is: 2403 miles / 3867.25 km / 2088.15 nautical miles http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html#create-an-iam-user http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html http://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_GetStarted.html http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html https://us-west-2.console.aws.amazon.com/ecs/home?region=us-west-2#/firstRun Virginia --------- https://console.aws.amazon.com/ecs/home?region=us-east-1#/firstRun https://aws.amazon.com/ecr/getting-started/ ECR --- http://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_GetStarted.html https://console.aws.amazon.com/ecs/home?region=us-east-1#/repositories http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html#cli-signup https://console.aws.amazon.com/ecs/home?region=us-east-1#/repositories/create aws configure aws ecr get-login --region us-east-1 http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html http://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-pull-ecr-image.html https://docs.docker.com/mac/step_six/ https://docs.docker.com/engine/reference/commandline/tag/ http://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_AWSCLI.html http://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html https://aws.amazon.com/blogs/aws/ec2-container-registry-now-generally-available/ Effective today, Amazon ECR is available in US East (Northern Virginia) with more regions on the way soon! Your Amazon ECS tasks run on container instances (Amazon EC2 instances that are running the ECS container agent). http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_defintions.html http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html A service lets you specify how many copies of your task definition to run. You could also use Elastic Load Balancing to distribute incoming traffic to your tasks. Amazon ECS keeps that number of tasks running and coordinates task scheduling with the load balancer. http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_ecstutorial.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_ecs.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.html https://aws.amazon.com/about-aws/whats-new/2015/03/aws-elastic-beanstalk-supports-multi-container-docker-environments/ http://cloudacademy.com/blog/amazon-ec2-container-service-docker-aws/ Task definitions specify the container information for your application, such as how many containers are part of your task, what resources they will use, how they are linked together, and which host ports they will use http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose.html https://aws.amazon.com/about-aws/whats-new/2015/10/introducing-the-amazon-ec2-container-service-cli-with-support-for-docker-compose/ http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-service.html After you create a cluster, you can launch container instances, and then run tasks http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI.html http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_tutorial.html http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html https://aws.amazon.com/blogs/aws/ec2-container-service-ecs-update-access-private-docker-repos-mount-volumes-in-containers/ http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html RDS --- http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html http://aws.amazon.com/rds/details/multi-az/ If a storage volume on your primary fails in a Multi-AZ deployment, Amazon RDS automatically initiates a failover to the up-to-date standby. Compare this to a Single-AZ deployment: in case of a Single-AZ database failure, a user-initiated point-in-time-restore operation will be required. This operation can take several hours to complete, and any data updates that occurred after the latest restorable time (typically within the last five minutes) will not be available. EC2 Container Service --------------------- .. code-block:: bash $ sudo apt-cache search awscli awscli - Universal Command Line Environment for AWS $ sudo apt-get install awscli $ aws --version aws-cli/1.10.1 Python/3.5.1+ Linux/4.4.0-1-amd64 botocore/1.3.23 $ aws configure AWS Access Key ID []: **************** AWS Secret Access Key []: **************** Default region name [oregon]: us-west-2 Default output format [json]: $ aws iam list-users $ aws ecs create-cluster help $ aws ecs list-container-instances help $ aws ecs create-cluster --cluster-name demo-01 { "cluster": { "pendingTasksCount": 0, "runningTasksCount": 0, "clusterName": "demo-01", "status": "ACTIVE", "clusterArn": "arn:aws:ecs:us-west-2:642913345125:cluster/demo-01", "activeServicesCount": 0, "registeredContainerInstancesCount": 0 } } $ aws ecs list-container-instances --cluster demo-01 Within ECS, you create task definitions, which are very similar to a docker-compose.yml file. A task definition is a collection of container definitions, each of which has a name, the Docker image to run, and options to override the image’s entrypoint and command. The container definition is also where you define environment variables, port mappings, volumes to mount, memory and CPU allocation, and whether or not the specific container should be considered essential, which is how ECS knows whether the task is healthy or needs to be restarted. You can set up multiple container definitions within the task definition for multi-container applications. ECS knows how to pull from the Official Docker Hub by default and can be configured to pull from private registries as well. Private registries, however, require additional configuration for the Docker client installed on the EC2 host instances. Once you have a task definition, you can create a service from it. A service allows you to define the number of tasks you want running and associate with an Elastic Load Balancer (ELB). When a task maps to particular ports, like 443, only one task instance can be running per EC2 instance in in the ECS cluster. Therefore, you cannot run more tasks than you have EC2 instances. In fact, you’ll want to make sure you run at least one less task than the number of EC2 instances in order to take advantage of blue-green deployments. Task definitions are versioned, and Services are configured to use a specific version of a task definition. https://blog.codeship.com/easy-blue-green-deployments-on-amazon-ec2-container-service/#comments Kubernetes groups containers into units called pods, which run on physical or virtual hosts called nodes. Collections of nodes that work together to support a pod deployment, and its associated applications, are called clusters. he core components of Kubernetes Kubernetes master components include: Kube-apiserver. The front end of the control plane that exposes Kubernetes APIs to cluster nodes and applications. Etcd. The Kubernetes data plane, in the form of a key-value store that manages cluster-specific but not application data. Kube-scheduler. Monitors resource usage on a cluster and assigns workloads, in the form of Kubernetes pods, to one or more worker nodes based on specified policies about hardware usage, node-pod affinity, security and workload priority. Kube-controller-manager. Runs the controller processes responsible for node monitoring, replication, container deployment and security policy enforcement. Cloud-controller-manager. A feature that primarily service providers use to run cloud-specific control processes. Kubernetes worker node components include: Kubelet. An agent that runs on each worker node. Kube-proxy. Manages network communication between cluster nodes. Container runtime. The engine that runs containers and maintains workload isolation within the OS. https://www.techtarget.com/searchitoperations/tip/Ensure-Kubernetes-high-availability-with-master-node-planning https://www.techtarget.com/searchaws/tip/2-options-to-deploy-Kubernetes-on-AWS-EKS-vs-self-managed https://medium.com/the-programmer/aws-eks-fundamentals-core-components-for-absolute-beginners-part1-9b16e19cedb3 https://bluexp.netapp.com/blog/aws-cvo-blg-aws-eks-architecture-clusters-nodes-and-networks https://spacelift.io/blog/kubernetes-secrets https://github.com/shuaibiyy/awesome-terraform https://github.com/bregman-arie/devops-exercises/blob/master/topics/terraform/README.md aws instance types ------------------ https://aws.amazon.com/ec2/instance-types/ https://instances.vantage.sh/ Max number of pods per node =========================== https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt Background Jobs with AWS ------------------------ https://joshmanderson.com/blog/serverless-background-jobs https://medium.com/fasal-engineering/background-job-processing-at-scale-using-aws-lambda-and-sqs-da7f512af767 https://medium.com/cheesecake-labs/asynchronous-task-queue-with-django-celery-and-aws-sqs-d68733ccfdb https://www.yippeecode.com/topics/aws-sqs-message-broker-for-python-celery/ https://www.yippeecode.com/topics/aws-sqs-message-broker-for-python-celery/ https://www.unitygroup.com/blog/performing-saas-application-background-tasks-with-aws-batch-and-aws-step-functions-6/ https://blog.datumbrain.com/2021/12/06/background-process-in-lambda-using-sqs.html https://stackoverflow.com/questions/67013095/what-is-the-best-practice-to-architect-tasks-processing-using-aws https://stackoverflow.com/questions/42637937/django-background-task-always-on#comment72427247_42637937 https://learn.microsoft.com/en-us/azure/architecture/best-practices/background-jobs RDS DB Parameter Group Family ***************************** .. code-block:: bash $ aws rds describe-db-engine-versions --query "DBEngineVersions[].DBParameterGroupFamily" Available PostgreSQL database versions ************************************** .. code-block:: bash aws rds describe-db-engine-versions --default-only --engine postgres https://aws.amazon.com/rds/faqs/#versioningGuidance https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html#Concepts.DBInstanceClass.Support https://aws.amazon.com/rds/instance-types/ Kubernetes storage provisioner ****************************** The difference lies in the type of Kubernetes storage provisioner used: 1. `provisioner="kubernetes.io/aws-ebs"`: This is the provisioner for the AWS Elastic Block Store (EBS) volumes using the in-tree volume plugin. In older versions of Kubernetes, this in-tree plugin was the standard way to provision EBS volumes. However, in recent Kubernetes versions (1.19 and later), in-tree plugins are being deprecated, and users are encouraged to use CSI (Container Storage Interface) drivers. 2. `provisioner: ebs.csi.aws.com`: This is the provisioner for the AWS EBS volumes using the CSI driver. CSI is a standardized interface that allows external storage providers to implement storage plugins for Kubernetes. With the CSI driver, storage vendors can develop and maintain their plugins independently, without relying on in-tree plugins, making it more extensible and easier to maintain. In summary, `provisioner="kubernetes.io/aws-ebs"` is the legacy in-tree provisioner for AWS EBS, while `provisioner: ebs.csi.aws.com` is the CSI driver provisioner for AWS EBS, which is the recommended approach for newer Kubernetes versions. If possible, it's better to use CSI drivers as they offer better flexibility, compatibility, and future-proofing for your Kubernetes storage needs. CSI driver for Amazon EBS https://github.com/kubernetes-sigs/aws-ebs-csi-driver https://aws.amazon.com/ebs/ CSI Driver for Amazon EFS https://github.com/kubernetes-sigs/aws-efs-csi-driver https://aws.amazon.com/efs/ Amazon Elastic Cache Security Groups ************************************ Q: How do I control access to Amazon ElastiCache? When not using VPC, Amazon ElastiCache allows you to control access to your clusters through Cache Security Groups. A Security Group acts like a firewall, controlling network access to your cluster. By default, network access is turned off to your clusters. If you want your applications to access your cluster, you must explicitly enable access from hosts in specific EC2 security groups. This process is called ingress. https://aws.amazon.com/elasticache/faqs/#Can_I_access_Amazon_ElastiCache_from_outside_AWS https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/SecurityGroups.html Amazon EKS IAM Policies, Roles, and Permissions *********************************************** https://docs.aws.amazon.com/eks/latest/userguide/IAM_policies.html Fluent Bit CloudWatch plugin **************************** cloudwatch_logs https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch Images: - https://github.com/aws/aws-for-fluent-bit - https://hub.docker.com/r/amazon/aws-for-fluent-bit/tags - https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch#aws-for-fluent-bit Configuration: - https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml - https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file Deprecated: - cloudwatch - https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit - https://github.com/aws/aws-for-fluent-bit#plugins Access logs for Application Load Balancer ***************************************** https://docs.aws.amazon.com/elasticloadbalancing/latest/application/enable-access-logging.html https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html NODE AMI ID (Images) ******************** Amazon EKS optimized Amazon Linux AMI IDs (Amazon Machine Image) https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html https://cloud-images.ubuntu.com/locator/ https://cloud-images.ubuntu.com/locator/ec2/ Ubuntu on Amazon Elastic Kubernetes Service (EKS) These images are customised specifically for the EKS service, and are not intended as general OS images. https://cloud-images.ubuntu.com/aws-eks/ https://cloud-images.ubuntu.com/docs/aws/eks/ Resource limit ************** http://aws.amazon.com/contact-us/ec2-request Difference between Cluster Security Group vs Node Security Group ***************************************************************** In Amazon EKS, there are two types of Security Groups are typically associated with an EKS cluster: * ClusterSecurityGroup: This is attached to the control plane of the EKS cluster. It's used to manage access to the Kubernetes API Server. * NodeSecurityGroup: This is attached to the EKS worker nodes (and managed node groups). It controls network access to worker nodes in the cluster. These two different types of security groups provide different levels of control over the network access for your EKS cluster and worker nodes. Why Use Fluent Bit for Sending Logs to Loki? ******************************************** https://calyptia.com/blog/how-to-send-logs-to-loki-using-fluent-bit Send SES emails from a specific sender using AWS IAM Policy ************************************************************* Restricting the "From" Address .. code-block:: json { "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Action":[ "ses:SendEmail", "ses:SendRawEmail" ], "Resource":"*", "Condition":{ "StringEquals":{ "ses:FromAddress":"marketing@example.com" } } } ] } https://docs.aws.amazon.com/ses/latest/dg/sending-authorization-policy-examples.html https://docs.aws.amazon.com/ses/latest/dg/control-user-access.html Grant Access To Only One S3 Bucket using AWS IAM Policy ******************************************************* .. code-block:: json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListAllMyBuckets" ], "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::MY-BUCKET", "arn:aws:s3:::MY-BUCKET/*" ] } ] } https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.Authorizing.IAM.S3CreatePolicy.html ELB ---- Internet-facing Classic Load Balancers https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-internet-facing-load-balancers.html Why does ELB need one public IP address for each public subnet? https://repost.aws/questions/QUHNazu9Y5ThK5BUAkJLGVyA/why-does-elb-need-one-public-ip-address-for-each-public-subnet#ANZv3fDNgKQ7S7vZq1PWpMZw What's the source IP address of the traffic that Elastic Load Balancing sends to my web servers? https://repost.aws/knowledge-center/elb-find-load-balancer-ip CloudWatch ----------- .. code-block:: bash fields @timestamp, @message | filter @logStream = 'app-web' | filter @message like "1.2.3.4" | parse @message 'log":"* - - [*] \"* * *\" * * * \"*",' as ip, time, method, path, http, status, _, __,ua | DISPLAY ip, time, method, path, http, status, ua | sort @timestamp desc | limit 25 .. code-block:: bash fields @timestamp, @message | parse @message 'log":"* - - [*] \"* * *\" * * * \"*",' as ip, time, method, path, http, status, _, __,ua | filter ispresent(ip) | stats count() as count by ip | sort count desc .. code-block:: bash fields @timestamp, @message | parse @message 'log":"* - - [*] \"* * *\" * * * \"*",' as ip, time, method, path, http, status, _, __,ua | stats count() as count by status, method, path | sort count desc .. code-block:: bash fields @timestamp, @message | parse @message 'log":"* - - [*] \"* * *\" * * * \"*",' as ip, time, method, path, http, status, _, __,ua | filter ispresent(status) | filter status == "500" | DISPLAY @logStream, ip, time, method, path, http, status, ua | sort status desc WAF2 ---- https://aws.amazon.com/about-aws/whats-new/2023/07/aws-waf-uri-path-aggregation-key-rate-based-rules/ https://docs.aws.amazon.com/waf/latest/APIReference/API_UriPath.html https://docs.aws.amazon.com/waf/latest/APIReference/API_FieldToMatch.html