Tips

https://github.com/boto/boto

https://github.com/bitly/asyncdynamo

https://pypi.python.org/pypi/dynamodb-mapper/1.1.0

https://pypi.python.org/pypi/ddbmock

http://boto.readthedocs.org/en/latest/dynamodb2_tut.html

Boto

http://boto.readthedocs.org/en/latest/index.html

http://aws.amazon.com/developers/getting-started/python/

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls-overview.html

http://www.networkautomation.com/automate/urc/resources/livedocs/am/10/Technical_Reference/Actions___Activities/Amazon_S3/S3_-_Get_Predesigned_URL.htm

Amazon EC2

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud.

It is designed to make web-scale cloud computing easier for developers.

Amazon EBS

Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes

for use with Amazon EC2 instances in the AWS Cloud.

Each Amazon EBS volume is automatically replicated within its Availability Zone to protect you from component failure,

offering high availability and durability. Amazon EBS volumes offer the consistent

and low-latency performance needed to run your workloads.

With Amazon EBS, you can scale your usage up or down within minutes –

all while paying a low price for only what you provision.

http://alestic.com/2012/01/ec2-ebs-boot-recommended

http://tiger-fish.com/blog/how-boot-amazon-ec2-instance-ebs-volume

http://thomas.broxrost.com/2008/08/21/persistent-django-on-amazon-ec2-and-ebs-the-easy-way/

http://stackoverflow.com/questions/10390244/how-to-set-up-a-django-project-with-django-storages-and-amazon-s3-but-with-diff

https://github.com/mstarinteractive/django-s3storage

https://github.com/mstarinteractive/django-s3storage/blob/master/example_settings.py

http://tartarus.org/james/diary/2013/07/18/fun-with-django-storage-backends

http://djangotricks.blogspot.de/2013/12/how-to-store-your-media-files-in-amazon.html

https://github.com/pcraciunoiu/django-s3sync

How to serve your media files via Amazon’s Simple Storage Service

http://stackoverflow.com/questions/11403063/setting-media-url-for-django-heroku-app-amazon-s3

https://github.com/django-compressor/django-compressor

http://stackoverflow.com/questions/11403063/setting-media-url-for-django-heroku-app-amazon-s3

http://stackoverflow.com/questions/10390244/how-to-set-up-a-django-project-with-django-storages-and-amazon-s3-but-with-diff

http://www.caktusgroup.com/blog/2014/11/10/Using-Amazon-S3-to-store-your-Django-sites-static-and-media-files/

http://martinbrochhaus.com/s3.html

http://stackoverflow.com/questions/9464038/redis-celery-configuration-over-amazon-ec2

http://stackoverflow.com/questions/14283021/how-to-use-django-celery-rq-worker-to-execute-a-video-filetype-conversion-ffm

http://django-storages.readthedocs.org/en/latest/

https://docs.djangoproject.com/en/1.7/howto/static-files/deployment/#staticfiles-from-cdn

http://www.caktusgroup.com/blog/2014/11/10/Using-Amazon-S3-to-store-your-Django-sites-static-and-media-files/

http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html

http://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html

https://aws.amazon.com/items/1434?externalID=1434

AWS RDS Postgres DB instance

http://aws.amazon.com/rds/postgresql/

http://aws.amazon.com/about-aws/whats-new/2013/12/11/aws-elastic-beanstalk-adds-background-task-handling-and-rds-postgresql-support/

http://stackoverflow.com/questions/26043706/how-to-use-boto-to-launch-an-elastic-beanstalk-with-an-rds-resource

http://stackoverflow.com/questions/25946723/aws-cli-create-rds-with-elasticbeanstalk-create-environment/25963800#25963800

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_flask.html

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python.rds.html

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html

http://stackoverflow.com/questions/13424267/setting-up-django-and-postgresql-on-two-different-ec2-instances

http://stackoverflow.com/questions/12850550/postgresql-for-django-on-elastic-beanstalk

http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances

http://www.holovaty.com/writing/aws-notes/

http://stackoverflow.com/questions/22599367/deploy-django-using-mysql-to-aws-ec2-and-rds

http://stackoverflow.com/questions/20914706/aws-elastic-beanstalk-hosting-postresql-on-deployed-ec2-server-with-django

http://www.quora.com/If-I-have-an-AWS-RDS-Postgres-DB-instance-do-I-also-need-to-install-Postgres-in-the-EC2-instance-that-has-my-Django-application-in-it

http://stackoverflow.com/questions/25740502/aws-can-a-beanstalk-instance-be-deployed-with-a-postgres-rds

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.RDS.html

https://github.com/tornadoweb/tornado/wiki/Links

http://stackoverflow.com/questions/11638135/amazon-aws-python-webframework-dynamodb

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.loggingS3.title.html

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html#command-options-python

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_custom_container.html

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.platforms.html

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.concepts.architecture.html

http://docs.aws.amazon.com/general/latest/gr/rande.html?r=1166

http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_SettingUpUser.html

http://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.html

http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPractices.html

http://aws.amazon.com/code/6752709412171743

Identity and Access Management

https://console.aws.amazon.com/iam/home#home

Before the Amazon ECS agent can register container instance into a cluster, the agent must know which account credentials to use.

You can create an IAM role that allows the agent to know which account it should register the container instance with.

When you launch an instance with the Amazon ECS-optimized AMI provided by Amazon using this role, the agent automatically registers the container instance into your default cluster.

The Amazon ECS container agent also makes calls to the Amazon EC2 and Elastic Load Balancing APIs on your behalf, so container instances can be registered and deregistered with load balancers. Before you can attach a load balancer to an Amazon ECS service, you must create an IAM role for your services to use before you start them.

This requirement applies to any Amazon ECS service that you plan to use with a load balancer.

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html

https://console.aws.amazon.com/iam/home#roles

Amazon EC2 Role for EC2 Container Service Role to allow EC2 instances in an Amazon ECS cluster to access Amazon ECS.

https://console.aws.amazon.com/ec2/

http://www.prokerala.com/travel/distance/from-california/to-vancouver-usa/

Distance To Vancouver From Oregon is: 1692 miles / 2723.01 km / 1470.31 nautical miles

Distance To Virginia From Vancouver is: 1725 miles / 2776.12 km / 1498.98 nautical miles

Distance To Vancouver From California is: 2403 miles / 3867.25 km / 2088.15 nautical miles

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html#create-an-iam-user

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

http://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_GetStarted.html

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html

https://us-west-2.console.aws.amazon.com/ecs/home?region=us-west-2#/firstRun

ECR

http://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_GetStarted.html

https://console.aws.amazon.com/ecs/home?region=us-east-1#/repositories

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html#cli-signup

https://console.aws.amazon.com/ecs/home?region=us-east-1#/repositories/create

aws configure aws ecr get-login –region us-east-1

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

http://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-pull-ecr-image.html https://docs.docker.com/mac/step_six/

https://docs.docker.com/engine/reference/commandline/tag/

http://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_AWSCLI.html

http://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html

https://aws.amazon.com/blogs/aws/ec2-container-registry-now-generally-available/

Effective today, Amazon ECR is available in US East (Northern Virginia) with more regions on the way soon!

Your Amazon ECS tasks run on container instances (Amazon EC2 instances that are running the ECS container agent).

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_defintions.html

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html

A service lets you specify how many copies of your task definition to run. You could also use Elastic Load Balancing to distribute incoming traffic to your tasks. Amazon ECS keeps that number of tasks running and coordinates task scheduling with the load balancer.

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_ecstutorial.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_ecs.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.html https://aws.amazon.com/about-aws/whats-new/2015/03/aws-elastic-beanstalk-supports-multi-container-docker-environments/

http://cloudacademy.com/blog/amazon-ec2-container-service-docker-aws/

Task definitions specify the container information for your application, such as how many containers are part of your task, what resources they will use, how they are linked together, and which host ports they will use

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose.html

https://aws.amazon.com/about-aws/whats-new/2015/10/introducing-the-amazon-ec2-container-service-cli-with-support-for-docker-compose/

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-service.html

After you create a cluster, you can launch container instances, and then run tasks

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI.html

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_tutorial.html

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html

https://aws.amazon.com/blogs/aws/ec2-container-service-ecs-update-access-private-docker-repos-mount-volumes-in-containers/

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html

RDS

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html

http://aws.amazon.com/rds/details/multi-az/

If a storage volume on your primary fails in a Multi-AZ deployment, Amazon RDS automatically initiates a failover to the up-to-date standby. Compare this to a Single-AZ deployment: in case of a Single-AZ database failure, a user-initiated point-in-time-restore operation will be required. This operation can take several hours to complete, and any data updates that occurred after the latest restorable time (typically within the last five minutes) will not be available.

EC2 Container Service

$ sudo apt-cache search awscli
awscli - Universal Command Line Environment for AWS

$ sudo apt-get install awscli

$ aws --version
aws-cli/1.10.1 Python/3.5.1+ Linux/4.4.0-1-amd64 botocore/1.3.23

$ aws configure
AWS Access Key ID []: ****************
AWS Secret Access Key []: ****************
Default region name [oregon]: us-west-2
Default output format [json]:

$ aws iam list-users

$ aws ecs create-cluster help
$ aws ecs list-container-instances help

$ aws ecs create-cluster --cluster-name demo-01
{
    "cluster": {
        "pendingTasksCount": 0,
        "runningTasksCount": 0,
        "clusterName": "demo-01",
        "status": "ACTIVE",
        "clusterArn": "arn:aws:ecs:us-west-2:642913345125:cluster/demo-01",
        "activeServicesCount": 0,
        "registeredContainerInstancesCount": 0
    }
}

$ aws ecs list-container-instances --cluster demo-01

Within ECS, you create task definitions, which are very similar to a docker-compose.yml file. A task definition is a collection of container definitions, each of which has a name, the Docker image to run, and options to override the image’s entrypoint and command. The container definition is also where you define environment variables, port mappings, volumes to mount, memory and CPU allocation, and whether or not the specific container should be considered essential, which is how ECS knows whether the task is healthy or needs to be restarted.

You can set up multiple container definitions within the task definition for multi-container applications. ECS knows how to pull from the Official Docker Hub by default and can be configured to pull from private registries as well. Private registries, however, require additional configuration for the Docker client installed on the EC2 host instances.

Once you have a task definition, you can create a service from it. A service allows you to define the number of tasks you want running and associate with an Elastic Load Balancer (ELB). When a task maps to particular ports, like 443, only one task instance can be running per EC2 instance in in the ECS cluster. Therefore, you cannot run more tasks than you have EC2 instances. In fact, you’ll want to make sure you run at least one less task than the number of EC2 instances in order to take advantage of blue-green deployments. Task definitions are versioned, and Services are configured to use a specific version of a task definition.

https://blog.codeship.com/easy-blue-green-deployments-on-amazon-ec2-container-service/#comments

Kubernetes groups containers into units called pods, which run on physical or virtual hosts called nodes. Collections of nodes that work together to support a pod deployment, and its associated applications, are called clusters.

he core components of Kubernetes

Kubernetes master components include:

Kube-apiserver. The front end of the control plane that exposes Kubernetes APIs to cluster nodes and applications. Etcd. The Kubernetes data plane, in the form of a key-value store that manages cluster-specific but not application data. Kube-scheduler. Monitors resource usage on a cluster and assigns workloads, in the form of Kubernetes pods, to one or more worker nodes based on specified policies about hardware usage, node-pod affinity, security and workload priority. Kube-controller-manager. Runs the controller processes responsible for node monitoring, replication, container deployment and security policy enforcement. Cloud-controller-manager. A feature that primarily service providers use to run cloud-specific control processes.

Kubernetes worker node components include:

Kubelet. An agent that runs on each worker node. Kube-proxy. Manages network communication between cluster nodes. Container runtime. The engine that runs containers and maintains workload isolation within the OS.

https://www.techtarget.com/searchitoperations/tip/Ensure-Kubernetes-high-availability-with-master-node-planning

https://www.techtarget.com/searchaws/tip/2-options-to-deploy-Kubernetes-on-AWS-EKS-vs-self-managed

https://medium.com/the-programmer/aws-eks-fundamentals-core-components-for-absolute-beginners-part1-9b16e19cedb3

https://bluexp.netapp.com/blog/aws-cvo-blg-aws-eks-architecture-clusters-nodes-and-networks

https://spacelift.io/blog/kubernetes-secrets

https://github.com/shuaibiyy/awesome-terraform

https://github.com/bregman-arie/devops-exercises/blob/master/topics/terraform/README.md

Background Jobs with AWS

https://joshmanderson.com/blog/serverless-background-jobs

https://medium.com/fasal-engineering/background-job-processing-at-scale-using-aws-lambda-and-sqs-da7f512af767

https://medium.com/cheesecake-labs/asynchronous-task-queue-with-django-celery-and-aws-sqs-d68733ccfdb

https://www.yippeecode.com/topics/aws-sqs-message-broker-for-python-celery/

https://www.yippeecode.com/topics/aws-sqs-message-broker-for-python-celery/

https://www.unitygroup.com/blog/performing-saas-application-background-tasks-with-aws-batch-and-aws-step-functions-6/

https://blog.datumbrain.com/2021/12/06/background-process-in-lambda-using-sqs.html

https://stackoverflow.com/questions/67013095/what-is-the-best-practice-to-architect-tasks-processing-using-aws

https://stackoverflow.com/questions/42637937/django-background-task-always-on#comment72427247_42637937

https://learn.microsoft.com/en-us/azure/architecture/best-practices/background-jobs

RDS DB Parameter Group Family

$ aws rds describe-db-engine-versions --query "DBEngineVersions[].DBParameterGroupFamily"

Kubernetes storage provisioner

The difference lies in the type of Kubernetes storage provisioner used:

  1. provisioner=”kubernetes.io/aws-ebs”: This is the provisioner for the AWS Elastic Block Store (EBS) volumes using the in-tree volume plugin.

In older versions of Kubernetes, this in-tree plugin was the standard way to provision EBS volumes. However, in recent Kubernetes versions (1.19 and later), in-tree plugins are being deprecated, and users are encouraged to use CSI (Container Storage Interface) drivers.

  1. provisioner: ebs.csi.aws.com: This is the provisioner for the AWS EBS volumes using the CSI driver.

CSI is a standardized interface that allows external storage providers to implement storage plugins for Kubernetes. With the CSI driver, storage vendors can develop and maintain their plugins independently, without relying on in-tree plugins, making it more extensible and easier to maintain.

In summary, provisioner=”kubernetes.io/aws-ebs” is the legacy in-tree provisioner for AWS EBS, while provisioner: ebs.csi.aws.com is the CSI driver provisioner for AWS EBS, which is the recommended approach for newer Kubernetes versions. If possible, it’s better to use CSI drivers as they offer better flexibility, compatibility, and future-proofing for your Kubernetes storage needs.

CSI driver for Amazon EBS

https://github.com/kubernetes-sigs/aws-ebs-csi-driver

https://aws.amazon.com/ebs/

CSI Driver for Amazon EFS

https://github.com/kubernetes-sigs/aws-efs-csi-driver

https://aws.amazon.com/efs/

Amazon Elastic Cache Security Groups

Q: How do I control access to Amazon ElastiCache?

When not using VPC, Amazon ElastiCache allows you to control access to your clusters through Cache Security Groups. A Security Group acts like a firewall, controlling network access to your cluster. By default, network access is turned off to your clusters. If you want your applications to access your cluster, you must explicitly enable access from hosts in specific EC2 security groups. This process is called ingress.

https://aws.amazon.com/elasticache/faqs/#Can_I_access_Amazon_ElastiCache_from_outside_AWS

https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/SecurityGroups.html

Amazon EKS IAM Policies, Roles, and Permissions

https://docs.aws.amazon.com/eks/latest/userguide/IAM_policies.html

NODE AMI ID

Amazon EKS optimized Amazon Linux AMI IDs (Amazon Machine Image)

https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html