Cloud Computing Info

AWS Price Decrease

Upcoming Price Changes

Effective November 1, 2009, we will be lowering prices for all On-Demand instances. The tables below show the existing and future On-Demand prices.

How often does a vendor REDUCE their prices, and thereby lowering your bill, without some nasty contract renegotiation? In my experience, never. One more reason to really like Amazon’s web services.
Starting November 1, 2009, EC2 prices are dropping 15% across the board (for linux AMIs). For a small image, that means a drop from $0.10/hour to $0.085/hour, large is going from $0.4/hour to $0.34/hour and the extra large are going from $0.8/hour to $0.68/hour.

Yep, looks like Amazon finally clued in to the fact that SimpleDB is pretty much useless for any mission critical work. They’ve added a new web services, Relational Database Service, abbreviated RDS.

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business.
Amazon RDS gives you access to the full capabilities of a familiar MySQL database.

Generate Your Keys

Now that you have chosen your instance, but before starting you actually start your instance, you need to generate your key pairs. The keypairs are SSH keypairs. A later post will explain SSH in greater detail but the keys come in a pair because there is both public and private components.

SSH is a Secure SHell. This is a command prompt like a DOS box or a telnet connection. However, unlike DOS and Telnet, it is very secure. The private key is the local machine’s secret password. The public key is shared to any host that the local machine will connect to.

Choose an AMI

Amazon, and Amazon clients, are providing a huge variation of machine images. The short story is that you can choose between MS-Windows, Linux and Sun Solaris for your OS. The real story is that it is a bit more complicated than that.

The real question is what applications do you plan to run and what expertise do you have on hand or plan to hire? A quick example is a database like MySQL. MySQL runs on various operating systems. If you have Windows expertise, you may want to stick with windows.

Choose Your Tool

When working with AWS, you have a choice of tools. You should try several tools and use the one that works best for your needs. Some tools are provided by Amazon and others are provided by third party developers. I cover seven tools in chapters that follow this one but that list is not a comprehensive list. It’s just the tools that I have used myself and that I know for a fact do work.

Some services are more programming tools that anything else. SQS is like that. It is a queuing service that you will plug into your applications.

AWS Security

Data Center Security

Amazon is a well known entity and works to provide an extremely secure environment for your applications ans your data. Amazon is pursuing Sabanes-Oxley certification (by an external auditing agency) and SAS-70 Type II certification.

Amazon does not broadcast the locations of their data centers and physical security is a top concern for them. They have military grade external protections. Physical access to Amazon data centers controlled by a two-factor authentication and only those Amazon employees with an actual need are ever given access.

Hardware access is provided only to those administrators who directly require it and they must use their own SSH keys to access bastion hosts (kind of like cloud overseers).

Sign Up For AWS

First things first, if you don’t have an Amazon.com account, go get one. If you do have one, you can use the one you already have. Amazon offers personal and corporate accounts. A person may have both accounts and can choose which to use when purchasing items.

It also may make sense that all employees have a business only account that uses their work email to log into the service. That way you never have an issue where purchases or billing can go to the wrong place.

Or, you may do like I have done in the past, put all expenses on a personal card and expense them back to the company.

Amazon DevPay

Amazon DevPay is an easy to use billing system for AWS developers. Build your cloud application, allow users to sign up and use your application and let Amazon bill them for you.

DevPay is “AWS-Aware” in that it ties into the billing of AWS services. Instead of a user having to sign up for AWS and be billed separately, you can add in the AWS costs to your costs and just bill the users directly.

DevPay is web based and uses Amazon Payments. The web interface allows you to register your application and set your pricing.

Using and Managing Amazon Web Services (AWS)

I personally believe that AWS is perfect for any development and testing environment. Regardless of how sensitive your data is, you can build your applications and test them in a cloud environment using bogus data.

For production environments, the choice is much harder. Does the country(ies) you operate in have strict privacy, or data on-shoring, laws that would be impact your applications? If you can easily offshore your applications, you can easily use cloud computing.

Does the area where you work have reliable infrastructure?

Mechanical Turk

Mechanical Turk is an odd service. It’s called an “on-demand workforce” or peopleware. For large tasks that need to be automated but also require human intelligence, Mechanical Turk is the tool.

One of the examples Amazon uses is if you have 1,000,000 (one million) images that need to be tagged and categorized, you can use Mechanical Turk to “hire” 10,000 employees. You get to pick what you will pay and only those “turks” who want the work will sign up.

Amazon picks up 10% (additive) to whatever you pay someone.

Amazon FWS

FWS is the Amazon Fulfillment Service.

SimpleDB

SimpleDB was Amazon’s first available (in beta) web service. It is a neat feature but it has its downsides. First, SimpleDB is not a relational database. It is a name/value key pair. For simple lookups, it is highly reliable and scalable. For anything more complicated, it falls apart.

SimpleDB is not ACID compliant and has not referential integrity. For that matter, it has not schemas, tables or relationships. Amazon says that it “eliminates the administrative burden of data modeling”. Some things make me say, “Hmmmmm.”

SimpleDB structures data somewhat like a spreadsheet. Think of columns across and values down.

CloudFront

Amazon CloudFront is Amazon’s Content Delivery Network (CDN). A CDN puts very large servers with high throughput at the edge of the network. That means that a CDN provider put cached data in multiple locations through out the network (internet). Requests for data are routed to a local server cache instead of the main server at a host. This improves performance, customer experience and possibly even costs (via lower bandwidth requirements).

An example would be a company that serves many pages to many users. Rather than have all of the pages stored in a central location and be accessed by many people all at once, the pages are distributed throughout the network and sit on many different servers.

Amazon Flexible Payments

Amazon Flexible Payments Service (FPS) is a set of web services that allow businesses or developers to bill users using the Amazon payment infrastructure (like a PayPal or Google Checkout). As a seller or a buyer, you can set limits on usage either globally or for specific senders and/or receivers. A gatekeeper component enforces the rules.

As a sender you can limit the number of transactions, transaction dates, dollar amounts, recipients and daily, weekly or monthly spending limits. Recipients can specify all of those and can specify allowable payment methods (credit card, bank transfer and amazon payments) and who pays the transaction fee.

One of the goals of FPS is to make micropayments effective and financially cost effective.

Simple Queue Service (SQS)

SQS is Amazon’s message queuing service. It works much like IBM’s MQ Series, JMS or Oracle AQ. Pop in a message and one or more recipients can pop it out. SQS is completely open so any internet connected computer can call a web service and add or remove a message.

Because SQS is API based, you can write an interface to it in the language of your choice. There are several free Java, Ruby and PHP interfaces available (that I know of) with more coming.

Elastic Compute Cloud (EC2)

Elastic Block Storage (EBS)

For most of its life in beta, EC2 offered only two kinds of storage, AMI based transient storage and S3. The transient storage was mounted as a filesystem and S3 was used for backup. To save data during downtime for instances, data had to first be saved off to S3 and the instance brought down. When the instance was brought back up, data was restored from S3. It was a painful process.

Enter EBS, the Elastic Block Store.

Simple Storage Service (S3)

Cost

Storage is cheaper in the US than in Europe. If you are based in Europe, you may want to decide which is more important when getting or adding data: price or latency.

Table 3: S3 Storage Costs

Table 4: S3 Data Transfer Costs

Table 5: S3 Request Costs

These prices are accurate as of the time of writing them.

Simple Storage Service (S3)

Security

Write and delete access to buckets and objects is controlled via Access Control Lists (ACL). You can assign read permissions to any object to specific users. You can also make an object public to grant access to anyone.

Transfer into and out of S3 can utilize SSH which will encrypt data. This prevents any “over the wire” interception of your data. Data at rest is not encrypted and Amazon recommends that users encrypt any sensitive data with their encryption tool of choice.

Simple Storage Service (S3)

The AWS S3 service is an API driven storage service. The API provides get, put and delete. Data is stored using a bucket concept that is not unlike directories and sub-directories. A bucket can hold one or more buckets and one or more objects (i.e. files). You can nest buckets as many levels deep as required by your application or other needs. Objects can be up to 5GB per and you can store an unlimited number of objects.

At the top level is a global bucket. All S3 accounts share the global bucket.

Sizing and Costs

EC2, like the other services in AWS are pay as you go, pay for what you use, services. As I mentioned above, you basically pay for the power you use which is a CPU per hour charge, bandwidth and storage. Linux and Windows guests have a different pricing menu. I am listing the prices current as of Dec 2008. I recommend you always check at aws.amazon.com to verify current pricing before making a commitment.

Instead of buying or leasing a specific type of hardware (that you would then be responsible for upgrading over time), AWS computing power is based on an EC2 compute unit.

Cloud Computing Info

Elastic Compute Cloud (EC2)

Transient Storage

The storage that comes with an AMI is called Transient Storage. That means that when the instance is stopped, the storage goes away. Any data or files saved when the instance was running is lost. This is by design.

To persist your data between sessions, you have two options. During most of the beta period, the Simple Storage Service (S3) was the only internal method of persisting data. S3 cannot be mounted as file system so it served as a backup service only.

Shortly before the beta period ended, Amazon added the Elastic Block Store (EBS).

Cloud Computing Info

Elastic Compute Cloud (EC2)

Security and Security Groups

Security is one of the most important, if not the most important, aspects of any important application. If you are thinking about running any kind of a mission critical application in the cloud, security should be a large part of your research.

AWS has been independently certified as Sarbanes-Oxley compliant and has passed a SAS70 audit. Amazon’s physical data center security follows established norms and is routinely audited.

On the software side, Amazon maintains a separation between host operating systems (those that Amazon are responsible for) and guest operating systems (the AMIs).

Cloud Computing Info

Elastic Compute Cloud (EC2)

Elastic IPs

Elastic IPs are a very neat feature of EC2. An Elastic IP is a configurable IP address assigned to your account. Unlike the rest of the features in AWS, you pay for any elastic IPs that you have allocated but not used. Amazon doesn’t want customers hoarding, and thus wasting, this resource. If you set aside the IP and use it, it’s free. If you set it aside and just keep it hanging around unused, you’ll pay a small monthly fee.

These IP addresses are connected to your account so you can assign them to any instance you might be running.

Cloud Computing Info

Elastic Compute Cloud (EC2)

EC2 is the computing part of the Amazon services. EC2 provides the CPU, memory, operating system and transient storage. EC2 is the equivalent of a barebones PC. You get to pick the amount of RAM you need (from a predefined list of configurations), the amount of transient storage you need (also from a list) and the number of CPUs you need (from a series of compute options). For the operating system, you can choose from various flavors of Linux, Solaris or Microsoft Windows Server.

If you can use a web browser and understand basic computer technology, you can use EC2.

Infrastructure as a Service

Amazon AWS is the largest of all the IaaS providers. Where SaaS offers a complete application as service and PaaS offers the ability to develop an application, IaaS doesn’t care about the application at all. IaaS provides the underlying hardware and operating system resources to do anything you want. IaaS has also been referred to as Everything as a Service.

IaaS offers CPU, memory, storage, networking and security as a package. IaaS is the virtual machine in the sky. In general, with IaaS, you choose from a range of operating systems (usually some flavor of open source), a size for your hardware (number of CPUs and CPU power) and an amount of storage.

There are a number of successful IaaS providers: Amazon, Joyent, GoGrid and FlexiScale.

Desktops as a Service

Falling some somewhere between software and a platform are Cloud Desktops (also called a Cloud OS). These desktops run inside a browser and are accessible from any desktop with an internet connection.

A cloud desktop offers word processing, spreadsheets, development tools, networking tools and more. While relatively immature at this time, we can expect this market to grow significantly in the coming years, especially as more and more smart phones and ultraportables hit the real world.

Microsoft LiveMesh offers free storage, machine synchronization and a cloud based desktop.