Powered By Blogger

Thursday, April 20, 2017

AWS Cloud Formation


Amazon Web Services CloudFormationis a free service that provides Amazon Web Service (AWS) customers with the tools they need to create and manage the infrastructure a particular software application requires to run on Amazon Web Services


CloudFormation has two parts: templates and stacks. A template is a JavaScript Object Notation (JSON) text file. The file, which is declarative and not scripted, defines what AWS resources or non-AWS resources are required to run the application. ( I will walk you through with the example at the end of this article ) 
When the template is submitted to the service, CloudFormation creates the necessary resources in the customer's account and builds a running of the template, putting dependencies and data flows in the right order automatically. The running instance is called a stack.
We  can make changes to the stack after it’s been deployed by using CloudFormation tools and an editing process that is similar to version control. When a stack is deleted, all related resources are deleted automatically as well.


advantage of CloudFormation is that it allows developers to automate service provisioning steps in a fairly simple way. There is no extra charge for AWS CloudFormation; customers only pay for the AWS resources that are required to run their applications.

Example
Below is self explanatory example of sample template 

{
  "AWSTemplateFormatVersion" : "2010-09-09",
 ##This is mandatory 

  "Description" : "AWS CloudFormation Sample Template EC2_Instance_With_Ephemeral_Drives: Example to show how to attach ephemeral drives using EC2 block device mappings. **WARNING** This template creates an Amazon EC2 instance. You will be billed for the AWS resources used if you create a stack from this template.",

  "Parameters" : {
    "KeyName": {
      "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the web server",
      "Type": "AWS::EC2::KeyPair::KeyName",
      "ConstraintDescription" : "must be the name of an existing EC2 KeyPair."
    },
##This is parameter for spinningup new EC2-Instance and its pre-requsites 
    "InstanceType" : {
      "Description" : "WebServer EC2 instance type",  
      "Type" : "String",
      "Default" : "t2.small", ## Instance type , there are lot more instance type but I choose t2.small for example 
     "ConstraintDescription" : "must be a valid EC2 instance type."
    },

    "SSHLocation": {
      "Description": "Lockdown SSH access to the bastion host (default can be accessed from anywhere)",
      "Type": "String",
      "MinLength": "9",
      "MaxLength": "18",
      "Default": "0.0.0.0/0", ##Mandatory field as we need to access server from IP range , 0.0.0.0/0 means over the internet and nyone can access it 
      "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
      "ConstraintDescription": "must be a valid CIDR range of the form x.x.x.x/x."
    }
  },

  "Mappings" : {
    "AWSInstanceType2Arch" : {
      "t1.micro"    : { "Arch" : "PV64"   }
      
    },
    

    "AWSRegionArch2AMI" : {  #provide region and their respectvie images (AMID with ID) 
      "us-east-1"        : {"PV64" : "ami-2a69aa47", "HVM64" : "ami-6869aa05", "HVMG2" : "ami-920f8984"},
      "us-west-2"        : {"PV64" : "ami-7f77b31f", "HVM64" : "ami-7172b611", "HVMG2" : "ami-54d44234"}
  },

  "Resources" : {
    "EC2Instance" : {
      "Type" : "AWS::EC2::Instance", 
      "Properties" : {
        "KeyName" : { "Ref" : "KeyName" },
        "InstanceType" : { "Ref" : "InstanceType" },
        "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" },
                          { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
        "SecurityGroups" : [{ "Ref" : "EC2SecurityGroup" }],
        "BlockDeviceMappings" : [
          {
            "DeviceName"  : "/dev/sdc",
            "VirtualName" : "ephemeral0"
          }
        ]
      }
    },

    "EC2SecurityGroup" : {
      "Type" : "AWS::EC2::SecurityGroup", #this will create secutiry group with given ingress rule port number and cidr range 
      "Properties" : {
        "GroupDescription" : "SSH access",
        "SecurityGroupIngress" : [{ "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : { "Ref" : "SSHLocation" }}]
      }
    }
  },

  "Outputs" : {
    "Instance" : {
      "Value" : { "Fn::GetAtt" : [ "EC2Instance", "PublicDnsName" ] }, 
      "Description" : "DNS Name of the newly created EC2 instance"
    }
  }
}

Cloud Computing

Cloud computing is usually described in one of two ways. Either based on the cloud location, or on the service that the cloud is offering.
Based on a cloud location, we can classify cloud as:
  • public,
  • private,
  • hybrid
  • community cloud
Based on a service that the cloud is offering, we are speaking of either:
  • IaaS (Infrastructure-as-a-Service)
  • PaaS (Platform-as-a-Service)
  • SaaS (Software-as-a-Service)
  • or, Storage, Database, Information, Process, Application, Integration, Security, Management, Testing-as-a-service
Previously, we have explained how cloud works. Basically, programs that are needed to run a certain application are now more popularly located on a remote machine, owned by another company. This is done in order not to lose on the quality performance due to processing power of your own computer, to save money on IT support, and yet remain advantageous on the market. These computers that run the applications, store the data, and use a server system, are basically what we call “the cloud”.

Where Do I Pull the Switch: Cloud Location

When we talk about public cloud, we mean that the whole computing infrastructure is located on the premises of a cloud computing company that offers the cloud service. The location remains, thus, separate from the customer and he has no physical control over the infrastructure.
As public clouds use shared resources, they do excel mostly in performance, but are also most vulnerable to various attacks.
Private cloud means using a cloud infrastructure (network) solely by one customer/organization. It is not shared with others, yet it is remotely located. If the cloud is externally hosted. The companies have an option of choosing an on-premise private cloud as well, which is more expensive, but they do have a physical control over the infrastructure.
The security and control level is highest while using a private network. Yet, the cost reduction can be minimal, if the company needs to invest in an on-premise cloud infrastructure.
Hybrid cloud, of course, means, using both private and public clouds, depending on their purpose.
For example, public cloud can be used to interact with customers, while keeping their data secured through a private cloud.
private_public_cloud
Image 1 – Private vs Public Cloud (Image Source: TalkCloudComputing)
Community cloud implies an infrastructure that is shared between organizations, usually with the shared data and data management concerns. For example, a community cloud can belong to a government of a single country. Community clouds can be located both on and off the premises.

What Can I Do With It: Cloud Service

The most popular services of the cloud are that of either infrastructureplatformsoftware, or storage.
As explained before, the most common cloud service is that one offering data storage disks and virtual servers, i.e. infrastructure. Examples of Infrastructure-as-a-Service (IaaS) companies are Amazon, Rackspace, Flexiscale.
If the cloud offers a development platform, and this includes operating system, programming language execution environment, database, and web server, the model is known as Platform-as-a-Service (PaaS), examples of which are Google App Engine, Microsoft Azure, Salesforce. Operating system can be frequently upgraded and developed with PaaS, services can be obtained from diverse sources, and programming can be worked in teams (geographically distributed).
Software-as-a-Service (SaaS), finally, means that users can access various software applications on a pay-per-use basis. As opposed to buying licensed programs, often very expensive. Examples of such services include widely used GMail, or Google Docs.
Image 2 – Cloud Services Types and Examples (Image Source: TheGadgetSquare)
The longer list will include Storage as a service (STaaS)Security as a service (SECaaS)Data as a service (DaaS)Test environment as a service (TEaaS)Desktop as a service (DaaS)API as a service (APIaaS).
Once you have understood the types of cloud computing, based on location and services, the most important step is to choose the right type of cloud and service, for a specific task with your clients.

AWS RDS Upgrade

Upgrading the MySQL DB Engine

When Amazon Relational Database Service (Amazon RDS) supports a new version of a database engine, you can upgrade your DB instances to the new version. There are two kinds of upgrades: major version upgrades and minor version upgrades.

Major Version Upgrades for MySQL

Amazon RDS supports the following in-place upgrades for major versions of the MySQL database engine:
  • MySQL 5.5 to MySQL 5.6
  • MySQL 5.6 to MySQL 5.7
Note
You can only create MySQL version 5.7 DB instances with current generation DB instance classes and the M3 previous generation DB instance class. If you want to upgrade a MySQL version 5.6 DB instance running on a previous generation DB instance class (other than M3) to a MySQL version 5.7 DB instance, you must first modify the DB instance to use a current generation DB instance class. After the DB instance has been modified to use a current generation DB instance class, you can then modify the DB instance to use the MySQL version 5.7 database engine. For information on Amazon RDS DB instance classes, see DB Instance Class.
Major version upgrades can contain database changes that are not backward-compatible with existing applications. As a result, Amazon Relational Database Service (Amazon RDS) doesn't apply major version upgrades automatically; you must manually modify your DB instance. You should thoroughly test any upgrade before applying it to your production instances.
To perform a major version upgrade for a MySQL version 5.5 DB instance on Amazon RDS to MySQL version 5.6 or later, you should first perform any available OS updates. After OS updates are complete, you must upgrade to each major version: 5.5 to 5.6, and then 5.6 to 5.7. MySQL DB instances created before April 24, 2014, show an available OS update until the update has been applied. For more information on OS updates, see Updating the Operating System for a DB Instance or DB Cluster.
During a major version upgrade of MySQL, Amazon RDS runs the MySQL binary mysql_upgrade to upgrade tables, if required. Also, Amazon RDS empties the slow_log and general_log tables during a major version upgrade. To preserve log information, save the log contents before the major version upgrade.
MySQL major version upgrades typically complete in about 10 minutes. Some upgrades might take longer because of the DB instance class size or because the instance doesn't follow certain operational guidelines in Best Practices for Amazon RDS. If you upgrade a DB instance from the Amazon RDS console, the status of the DB instance indicates when the upgrade is complete. If you upgrade using the AWS Command Line Interface (AWS CLI), use the describe-db-instances command and check the Status value.

Upgrades to MySQL Version 5.7 Might Be Slow

MySQL version 5.6.4 introduced a new date and time format for the datetimetime, and timestamp columns that allows fractional components in date and time values. When upgrading a DB instance to MySQL version 5.7, MySQL will force the conversion of all date and time column types to the new format. Because this conversion rebuilds your tables, it might take a considerable amount of time to complete the DB instance upgrade. The forced conversion will occur for any DB instances that are running a version prior to MySQL version 5.6.4, and also any DB instances that were upgraded from a version prior to MySQL version 5.6.4 to a version other than 5.7.

If your DB instance is running a version prior to MySQL version 5.6.4, or was upgraded from a version prior to MySQL version 5.6.4, then we recommend that you convert the datetimetime, and timestamp columns in your database before upgrading your DB instance to MySQL version 5.7. This conversion can significantly reduce the amount of time required to upgrade the DB instance to MySQL version 5.7. To upgrade your date and time columns to the new format, issue the ALTER TABLE <table_name> FORCE; command for each table that contains date or time columns. Because altering a table locks the table as read-only, we recommend that you perform this update during a maintenance window.

Upgrading a MySQL Database with Reduced Downtime

If your MySQL DB instance is currently in use with a production application, you can use the following procedure to upgrade the database version for your DB instance and reduce the amount of downtime for your application. This procedure shows an example of upgrading from MySQL version 5.5 to MySQL version 5.6.
To upgrade an MySQL database while a DB instance is in use
  1. Sign in to the AWS Management Console and open the Amazon RDS console athttps://console.aws.amazon.com/rds/.
  2. Create a Read Replica of your MySQL 5.5 DB instance. This process creates an upgradable copy of your database.
    1. On the console, choose Instances, and then choose the DB instance that you want to upgrade.
    2. Choose Instance Actions, and then choose Create Read Replica.
    3. Provide a value for DB Instance Identifier for your Read Replica and ensure that the DB instance Class and other settings match your MySQL 5.5 DB instance.
    4. Choose Yes, Create Read Replica.
  3. When the Read Replica has been created and Status shows available, upgrade the Read Replica to MySQL 5.6.
    1. On the console, choose Instances, and then choose the Read Replica that you just created.
    2. Choose Instance Actions, and then choose Modify.
    3. For DB Engine Version, choose the MySQL 5.6 version to upgrade to, and then choose Apply Immediately. Choose Continue.
    4. Choose Modify DB Instance to start the upgrade.
  4. When the upgrade is complete and Status shows available, verify that the upgraded Read Replica is up to date with the master MySQL 5.5 DB instance. You can do this by connecting to the Read Replica and issuing the SHOW SLAVE STATUS command. If the Seconds_Behind_Master field is 0, then replication is up to date.
  5. Make your MySQL 5.6 Read Replica a master DB instance.
    Important
    When you promote your MySQL 5.6 Read Replica to a standalone, single-AZ DB instance, it will no longer be a replication slave to your MySQL 5.5 DB instance. We recommend that you promote your MySQL 5.6 Read Replica during a maintenance window when your source MySQL 5.5 DB instance is in read-only mode and all write operations are suspended. When the promotion is completed, you can direct your write operations to the upgraded MySQL 5.6 DB instance to ensure that no write operations are lost.
    In addition, we recommend that before promoting your MySQL 5.6 Read Replica you perform all necessary data definition language (DDL) operations, such as creating indexes, on the MySQL 5.6 Read Replica. This approach avoids negative effects on the performance of the MySQL 5.6 Read Replica after it has been promoted. To promote a Read Replica, use this procedure:
    1. On the console, choose Instances, and then choose the Read Replica that you just upgraded.
    2. Choose Instance Actions, and then choose Promote Read Replica.
    3. Enable automated backups for the Read Replica instance. For more information, see Working With Backups.
      Choose Continue.
    4. Choose Yes, Promote Read Replica.
  6. You now have an upgraded version of your MySQL database. At this point, you can direct your applications to the new MySQL 5.6 DB instance, add Read Replicas, set up Multi-AZ support, and so on.

AWS Management Console

To upgrade the engine version of a DB instance by using the AWS Management Console
  1. Sign in to the AWS Management Console and open the Amazon RDS console athttps://console.aws.amazon.com/rds/.
  2. In the navigation pane, choose Instances.
  3. Choose the check box for the DB instance that you want to upgrade.
  4. Choose Instance Actions, and then choose Modify.
  5. For DB Engine Version, choose the new version.
  6. To upgrade immediately, select Apply Immediately. To delay the upgrade to the next maintenance window, clear Apply Immediately.
  7. Choose Continue.
  8. Review the modification summary information. To proceed with the upgrade, choose Modify DB Instance. To cancel the upgrade, choose Cancel or Back.

CLI

To upgrade the engine version of a DB instance, use the AWS CLI modify-db-instance command. Specify the following parameters:
  • --db-instance-identifier – the name of the db instance.
  • --engine-version – the version number of the database engine to upgrade to.
  • --allow-major-version-upgrade – to to upgrade major version.
  • --no-apply-immediately – apply changes during the next maintenance window. To apply changes immediately, use --apply-immediately.
Example
For Linux, OS X, or Unix:
Copy
aws rds modify-db-instance \ --db-instance-identifier <mydbinstance> \ --engine-version <new_version> \ --allow-major-version-upgrade \ --apply-immediately