This morning I saw a post by Om Malik at GigaOm, OpenStack Wants to Be Android of The Cloud. Very interesting article and it’s the first I’ve heard of OpenStack.

I’m a huge AWS fan and I think it is still be best product for the public cloud. However, it doesn’t do much in the private arena.

I’ve been following Cloud OSes for a while but that is a completely different than cloud infrastructure. I’ve also been following Eucalyptus and a few minor projects.

Eucalyptus has a lot of promise, but I am liking the true open source commitment of OpenStack. Not open source as a label and a sales model, not open core. Real open source.

With support from rackspace (deep commitment there), NASA (another deep commitment), dell, AMD, Intel, and more, this has the potential to be the linux of the open source cloud stack.

The project is being released with an Apache 2.0 license. Plastered all over the site are the words “open” and “freedom”. I think they, unlike some other companies, will go beyond the words.

Like the OpenStack blog says:

What does “openness” mean to us? “Open” and “open source” are thrown around a lot, so its worth specifically defining our commitment to the community:

COMMITMENT #1: We are producing truly open source software. No artificial limits will be placed or performance limitations maintained. No licensing model – one free, one paid – will be introduced. We are releasing the code under the Apache 2.0 license which allows the community to do with the software as they see fit, including implement into other distributions or “for fee” offerings.

COMMITMENT #2: We are committed to an open design process. Rackspace will provide dedicated project leads to guide the roadmap on behalf of the community. We will hold regular design summits—open to anyone—which will produce a roadmap to guide development.

COMMITMENT #3: All development will be done in the open. We will maintain a publicly available source code repository to simplify participation.

COMMITMENT #4: We will maintain an open community. Healthy, vibrant developer and user communities are the basis of any open source project. Most decisions will be made using a “lazy consensus” model. All processes will be documented, open and transparent.

You can get more info on OpenStack by following them on twitter, @openstack.

I know I plan to keep an eye on them.

LewisC

Ok, now they’ve gone and done it. Amazon just doesn’t know when to quit. Is there a web service they don’t want to own? I josh. I like Amazon and like to see them put new and useful services out there and I am a big proponent of competition. I think competition is good for everyone as long as it doesn’t become predatory.

Now, Amazon has been going after Paypal for a while now with Amazon Checkout. But now, they have made this a super simple, no login, purchase tool with PayPhrase – the easy-to-remember shortcut for paying on Amazon.com and other websites.

According to Amazon,

PayPhrase links your Amazon.com payment and shipping information with a simple phrase that you choose. With PayPhrase, you no longer have to register or share credit card information with multiple web sites.

With Amazon Checkout, my info is stored and some kind of token is exchanged with a vendor so my personal information doesn’t need to be. I like Paypal and use it fairly heavily when paying for things on the internet. I also like Amazon and even have an Amazon credit card. I think I buy something from them at least once a month. Now, I don’t even need to login.

The cool thing about PayPhrase is the control you can put on an account. You can give your kid access and set spending limits and have it send you order approval notifications. Man, I need that for XBox Live! You can use it to give your kids an allowance. Set them up with a PayPhrase using your credit information and then set a monthly limit. Sweet!

There is no way that PayPhrase is as widely supported as Paypal right now. Amazon CheckOut market share has got to be minimal compared to PayPal. Have to see how well it spreads over time. For a developer or merchant already on AWS, it’s kind of a no brainer to include this. They already have some merchants using PayPhrase: DKNY, Jockey, Patagonia, Buy.com, J&R Electronics, and Car Toys to name a few.

If you sign up, the system will generate a phrase for you. I didn’t like mine as there is no way I would ever remember it. They also list some suggestions but I didn’t like those. Almost every suggestion included the word “bread”. They trying to tell me something? I swear I’ve cut back on the carbs!

It must be at least two words and contain no numbers. This is NOT a password. It is a pass phrase.

Once you’ve chosen your phrase, you also have to enter a pin number (4 digits). When complete you verify your credit card and payment info and then you are the proud owner of a new PayPhrase.

Be interesting to see if they make any kind of a dent in PayPal. This is a service of Amazon Checkout which is a sub-service of Amazon Payments. I don’t believe PayPhrase is an additional fee on top of CheckOut. CheckOut has very reasonable pricing.

Take care,

LewisC

AWS Price Decrease

Upcoming Price Changes

Effective November 1, 2009, we will be lowering prices for all On-Demand instances. The tables below show the existing and future On-Demand prices.

How often does a vendor REDUCE their prices, and thereby lowering your bill, without some nasty contract renegotiation? In my experience, never. One more reason to really like Amazon’s web services.

Starting November 1, 2009, EC2 prices are dropping 15% across the board (for linux AMIs). For a small image, that means a drop from $0.10/hour to $0.085/hour, large is going from $0.4/hour to $0.34/hour and the extra large are going from $0.8/hour to $0.68/hour. For a business using several instances (usually in the large and extra-large capacities), this could be a significant savings over time. Think about – a 15% reduction and you don’t have to do anything to get it.

Data transfer and storage stay the same so it’s not a complete 15% reduction in your entire bill. I still think this is a significant poke at Microsoft and others getting into the cloud market. Windows AMIs are being reduced to but at a smaller percentage. For example, the Extra Large windows instance is dropping from $1.00/hour to $0.98/hour.

I need a bigger box!

Amazon has added some new server sizes to the farm. I love the names. Remember double, extra top secret (I think that was from Get Smart). Let’s try these names on for size. Amazon is added in the Double Extra Large High-Memory On-Demand Instance and Quadruple Extra Large High-Memory On-Demand Instance. Say that 5 times real fast.

These puppies are not cheap: $1.20/hour and $2.40/hour, respectively. They are, however, beefy!

High-Memory Double Extra Large Instance

34.2 GB of memory
13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each)
850 GB of instance storage
64-bit platform
I/O Performance: High

High-Memory Quadruple Extra Large Instance

68.4 GB of memory
26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High

Schwing!

Take care,

LewisC

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business.

Amazon RDS gives you access to the full capabilities of a familiar MySQL database. This means the code, applications, and tools you already use today with your existing MySQL databases work seamlessly with Amazon RDS. Amazon RDS automatically patches the database software and backs up your database, storing the backups for a user-defined retention period. You also benefit from the flexibility of being able to scale the compute resources or storage capacity associated with your relational database instance via a single API call. As with all Amazon Web Services, there are no up-front investments required, and you pay only for the resources you use.

This is pretty slick. I haven’t played with it yet as it was just announced but it seems to be an API driven mysql instance. For slightly more than a base instance, 0.11/hour RDS vs 0.10/hour base EC2 (this price is dropping 15% BTW) on a small server, you get a complete server with MySQL installed. You can create and manage your database instances via procedural call (the API) and you can scale to larger instances or additional storage fairly painlessly by also using those APIs. You also pay extra for your storage of course.

That’s about it from what I’ve read. I don’t see any automated replication (beyond the normal AWS safety features) and I don’t see any kind of clustering or sharding. This is not what most people would call a cloud database. It’s just an easy to configure, maintain and grow MySQL server. Not that that’s bad. For a small business with some technical savvy but not a lot of time, this is an awesome addition to AWS. I would be willing to bet that some kind of clustering will come, sooner or later.

Ooops, just stumbled across:

Coming Soon: High Availability Offering — For developers and business who want additional resilience beyond the automated backups provided by Amazon RDS at no additional charge. With the high availability offer, developers and business can easily and cost-effectively provision synchronously replicated DB Instances in multiple availability zones (AZ’s), to protect against failure within a single location.

One of the things I have always liked about AWS is that they really do make it simple. For the uses cases where SimpleDB is appropriate, using it is a no brainer, as is EC2 and S3. AWS even makes queuing simple. RDS keeps to that methodology.

Amazon RDS allows you to use a simple set of web services APIs to create, delete and modify relational database instances (DB Instances). You can also use the APIs to control access and security for your instance(s) and manage your database backups and snapshots. For a full list of the available Amazon RDS APIs, please see the Amazon RDS API Guide. Some of the most commonly used APIs and their functionality are listed below:

CreateDBInstance — Provision a new DB Instance, specifying DB Instance class, storage capacity and the backup retention policy you wish to use. This one API call is all that’s needed to give you access to a running MySQL database, with the software pre-installed and the available resource capacity you request.

ModifyDBInstance — Modify settings for a running DB Instance. This lets you use a single API call to scale the resources available to your DB Instance in response to the load on your database, or change how it is automatically backed up and maintained on your behalf.

DeleteDBInstance — Delete a running DB Instance. With Amazon RDS, you can terminate your DB Instance at any time and pay only for the resources you used.

CreateDBSnapshot — Generate a snapshot of your DB Instance. You can restore your DB Instance to these user-created snapshots at any point, even to reinstate a previously deleted DB Instance.

RestoreDBInstanceToPointInTIme — Create a new DB Instance from a point-in-time backup. You can restore to any point within the retention period you specified, usually up to the last five minutes of your database’s usage.

This is a very cool addition to AWS. I am looking forward to playing with it. It’s important to note that if you are capable of administering your own server and database, you can save money by running a base EC2 instance and DIY. If you want to run any database other than MySQL, you have to do that anyway.

LewisC

Generate Your Keys

Now that you have chosen your instance, but before starting you actually start your instance, you need to generate your key pairs. The keypairs are SSH keypairs. A later post will explain SSH in greater detail but the keys come in a pair because there is both public and private components.

SSH is a Secure SHell. This is a command prompt like a DOS box or a telnet connection. However, unlike DOS and Telnet, it is very secure. The private key is the local machine’s secret password. The public key is shared to any host that the local machine will connect to.

The host is able to create a query after seeing the public key that only someone with the private key could answer. The private key is never shared but the host is convinced that it is talking to the person (or machine) that is says it is.

This may sound confusing but it is actually very secure. It’s is much better than passwords that can be hacked or accidentally given away.

Amazon supports SSH and secure communications out of the box. If you choose to revert to simple protocols such as telnet and ftp and to password authentication, you may do so. However, your first connection to any instance started through AWS will have to be via SSH. Amazon makes it easy to be secure but gives you the option of making it less secure.

So at least one pair of keys needs to be generated. Each tool set that you choose will create the files in a different way. If you are running the command line tools, you will run the ec2-add-keypair program. If running ElasticFox or CloudStudio, you will have a button on the GUI. However you create the keypair, the end result is that you will end up with a file that tends in a .pem format.

When running SSH (and the tools) from a Windows client, you will need to convert the .pem file to a PuTTY formatted key file. PuTTY, like SSH will be documented in greater detail in a near future post. Review that post for tips on Converting SSH to PuTTY.

You choose an instance’s keypair when you start it and you cannot change that after it is running. Generate your key pair and getting working first.

Choose an AMI

Amazon, and Amazon clients, are providing a huge variation of machine images. The short story is that you can choose between MS-Windows, Linux and Sun Solaris for your OS. The real story is that it is a bit more complicated than that.

The real question is what applications do you plan to run and what expertise do you have on hand or plan to hire? A quick example is a database like MySQL. MySQL runs on various operating systems. If you have Windows expertise, you may want to stick with windows. On the other hand, you can run some Linux instances with MySQL pre-installed and configured.

This about the stack that you want to run. I generally run Linux instances. They are a few cents cheaper per CPU hour and I am good enough with Linux that it doesn’t cause me any issues. I can run Oracle, MySQL and Postgres side-by-side. I occasionally do run Windows instances though just to compare offerings.

If you run SQL Server, you will need to run Windows. Almost any other software stack offers an option of OS. If you do run Windows, you will be running Windows Server 2003, in either 32 or 64 bit. SQL Server can be the Express Edition or the full blown commercial edition (which costs extra for licensing).

If you want to run Solaris, you currently have to register with Sun to get access to the OpenSolaris instance. It’s free but it requires registration. With OpenSolaris you get DTrace and ZFS, two selling points for many people.

You get OpenSolaris 2008.05 or Solaris Community Edition and pricing is the same as a Linux install. You can AMIs with AMP preinstalled as well as stacks like Drupal and MySQL.

For Linux installs, the choices are almost limitless: Fedora, CentOS, Ubuntu, Oracle Unbreakable Linux, RedHat. You name it, it’s probably there. Many of these come with pre-installed software stacks. No download and configure, just run.

Plan to try many instance types. You may even end up with a RightScale instance.

With terabyte datasets becoming the norm, and petabyte on the way, I knew Amazon would eventually address this in some way. They did it faster than I thought they would.

You’ll pay per device and per load hour in addition to normal S3 storage and calls. You won’t pay any transfer fees.

This will be huge for people who want to make large data sets available (internally or externally for pay) and for CDN users.

The import feature requires some advance coordination with Amazon but BucketExplorer and S3Fox already support it.

Export is TBD but I imagine you’ll send them a device and manifest and they’ll offload the bucket for you.

Read the rest of the details on the amazon blog: AWS Import/Export: Ship Us That Disk!.

LewisC

Choose Your Tool

When working with AWS, you have a choice of tools. You should try several tools and use the one that works best for your needs. Some tools are provided by Amazon and others are provided by third party developers. I cover seven tools in chapters that follow this one but that list is not a comprehensive list. It’s just the tools that I have used myself and that I know for a fact do work.

Some services are more programming tools that anything else. SQS is like that. It is a queuing service that you will plug into your applications. You can interface with SQS using PHP, C#, Ruby, Perl and many other languages. Actually, you can also write interfaces to S3 or EC2 using a language of your choice but how many of us really want to write an interface?

When choosing your tool, you need to think about how you will be using AWS. Will you primarily be an S3 user? In that case you will to choose a tool with robust S3 handling. If you don’t plan to use EC2 at all, you may want to skip the tools that provide EC2 functionality and stick with the S3 browsers like S3 Browser and S3 Organizer.

On the other hand, if your only use of S3 will be snapshot backups of you EC2 instances and EBS volumes, you will want to pick a tool that helps you choose an AMI, run it and monitor it. ElasticFox and Cloud Studio are ideal for that environment.

If you plan to use both EC2 and S3 fairly heavily, Cloud Studio provides nice S3 support while ElasticFox is lacks S3 support. If you are a Firefox user though, the combination of S3 Browser and ElasticFox will provide all of the functionality you’re likely to need.

For those individuals who like pain, Amazon provides the AWS Command Line Tool set. It’s not for everyone but for those who enjoy typing at a command prompt, more power to you. Actually, I am somewhat joking. Everyone should be a little bit familiar with the command line tools just in case something goes wrong with the GUI tools.

Time will add more tools, I’m sure. The point I am making here, is to explore the options and choose a tool that fits your needs.

If you are an S3 only user, your setup needs ends here. Once you have signed up and chosen a tool, you can get started working.

The remaining posts in this series are geared towards EC2 users. Before logging into your chosen tool, you need to put some thought into your instance security, storage and usage.

This post will (attempt) to explain what SSH and PuTTY are so that as a user you understand the terminology of AWS and so that you can be productive in the environment. This post will not attempt to make you an expert in SSH. For best practices in implementing SSH, I strongly recommend a book dedicated to hardening *nix (Linux, Unix, Solaris, etc).

SSH

In the early days, not that long ago really, of networking, very simple tools were used to work with remote computers: telnet as a console, ftp for file copying, rsh for remote command execution and others. These were easy to configure and use tools. They were client server in that a software component needed to run on both the local machine (client) and the remote machine (server).

While easy to use, they were very insecure. They made no pretense at verifying that the calling host really was the calling host. Everything was username/password based and both the username and the password were passed around the network in cleartext. If you intercepted the little data packages that were being routed around the network (with a sniffer for example), you would be able to extract the login credentials. Even if you encrypted all of your data, your credentials were still in the clear.

SSH is an attempt (quite successful) to fix those insecurities without making things anymore complex than they need to be. SSH stands for Secure SHell. However, SSH is not really a command shell, it is rather a protocol that encrypts communications. That means that programs that use SSH can work like telnet or ftp but will be more secure.

Note: Technically, SSH is also a tool. There is a client terminal program called SSH. It’s a non-graphical command line tool that provides a window which executes a command shell on the remote system.

SSH offers multiple modes of connecting but for the purposes of AWS, we will talk about key based access. To make things more secure, EC2 uses a key based authentication. Before starting an instance, you need to create a key pair.

Note: The below explanation of SSH is a gross over simplification. I am just trying to give you a feel for what is going on. If you really want to understand the technical details, I really do recommend that you purchase a book. My personal recommendation is SSH, The Secure Shell: The Definitive Guide from O’Reilly.

When an instance starts up for the first time, EC2 copies the ssh key that you created to the proper directory on the remote server. The remote server will be running the SSH Server software.

You will then use an SSH client to connect to the server. The client will ask for some information proving that the server really is who it says it is. The first time you connect to a server, the client won’t have that information available so it will prompt you to vertify that the server is legitimate.

You verify that information by comparing a thumbprint. Verifying a host is a bit beyond this book but do an internet search for for “ssh host thumbprint”. You’ll find a variety of articles explaining it in detail.

Once the client accepts the host, the client will send secret information to the host. This is your key data. If the host is able to make a match, it will authenticate you and let you login in. If the host then asks for a password, you key did not work and something is not configured properly. In my experience, it will probably be that your client key file is not in the place your client is expecting it to be.

What happens next depends on the tool you are using. If you are using a terminal program, ssh for example, you will now have a command prompt. If you are using sftp or scp, you will be able to copy files.

In addition to command line tools, there are GUI tools that use the SSH protocol. WinSCP is an excellent SCP client for Windows.

Regardless of the tools you use, SSH is busy encrypting everything you send over the wire. The SSH protocol has evolved over the years, and will probably evolve even more in the future, but it is currently running a very secure form of encryption.

If you are running Linux, you are pretty much finished at this point. SSH ships with every Linux distribution that I am aware of. If you are using Windows, however, you either need to install CyWin (a unix environment that runs in windows), or you’ll want to get PuTTY.

PuTTY

You can download all of the programs discussed in this section at:

http://www.chiark.greenend.org.uk/~sgtatham/putty/

I honestly have no idea why PuTTY is spelled PuTTY. I can figure the TTY part of it is from the Unix command that output a display. I’m not sure bout the Pu though.

I do know what PuTTY is though. PuTTY is a very simple implementation of an MS-Windows SSH terminal client. When I say it is simple, I mean that as a complement. This is a tool that does not get in the way.

You tell PuTTY to connect to a remote server and, as long as your keys are configured, it will connect you. If are not using keys, you can connect with passwords (if the host allows that). As a best practice, keys are recommends over passwords.

PuTTY is the terminal client but you can get a couple of other tools from the same author. PSFTP and PSCP offer secure file transfers. These tools are as easy to use as PuTTY and work pretty much the same way.

For command line syntax and configuration, take a look at the documentation at the link above.

A note about SSH keys and PuTTY, they are not compatible. This same web site offers a utility called PuTTYgen. When you create a key pair for EC2, you download that file to your local machine. PuTTYgen converts that file (a .pem file) to a private key file (a .ppk file).

PuTTY Key Generator

The tool is named puttygen.exe. Run the executable and the above window pops up. To convert an amazon key to a PuTTY key, use the menu option Conversions ? Import Key. Load the .pem file that you downloaded and press the Save Private Key button.

It will warn you about leaving the passphrase blank. That’s ok.

Save the file to the location that PuTTY has been configured to look in for it’s keys.

Data Center Security

Amazon is a well known entity and works to provide an extremely secure environment for your applications ans your data. Amazon is pursuing Sabanes-Oxley certification (by an external auditing agency) and SAS-70 Type II certification.

Amazon does not broadcast the locations of their data centers and physical security is a top concern for them. They have military grade external protections. Physical access to Amazon data centers controlled by a two-factor authentication and only those Amazon employees with an actual need are ever given access.

Hardware access is provided only to those administrators who directly require it and they must use their own SSH keys to access bastion hosts (kind of like cloud overseers). They can then escalate access to gain access to individual client hosts. All administrator access is logged and audited.

The network is monitored by Amazon security services. Due to Amazon IP security, an EC2 instance cannot spoof an IP address. An instance is not allowed to send traffic with a spoofed address. Also, Amazon monitors for port scanning. If they find port scanning, they block the incoming address.

Because all clients are running in virtual servers with virtual storage, there is no way for one client to gain access to another clients data or traffic. For all intents and purposes, each client is running in their own data center.

Data Security

Your data is secured when traveling over the wire by SSL. You can chose less secure methods once you have an image up and running but by default, an AMI will be very secure. If you choose to open your firewall (security group) to any and all traffic, you will be open to hacking. If you chose to use password security instead of SSH keys, you take your own risks.

There are several additional steps you can take to protect your data.

• Only present web servers to the internet. You have the option of not having a public IP address on every instance. If you have amulti-tier application, you can choose to have a public IP address on your web server and have just an internal IP address on your database server. To access the database server, you would have to log into the web server and then ssh from there to the databaseserver.

• Another option is to encrypt all of your stored data (or at least the sensitive portions of it). Amazon offers Linux, Windows and Sunvirtual machines and all of these operating systems offer very robust (at least via third party tools) encryption. A very good, freeoption on windows servers is TrueCrypt.

Data being stored in AWS applications (S3, SimpleDB and EBS) is automatically, redundantly stored in multiple physical locations. You do not pay for this additional storage. Amazon does this to ensure the integrity of your data (and that they meet their SLAs).

Yet another option is to use the encryption capabilities offered by the various databases that you might be using. Oracle provides Transparent Data Encryption for data at rest and offers Oracle Secure Backup via RMAN. Using Oracle Secure backup with the Cloud Module extension will allow you to encrypt your back ups and store your data on S3.

Authentication

AWS allows two different methods of authentication. When you submit a request, be it to create a new instance in EC2 or to upload a file to S3, AWS needs to know that you are allowed to to submit the request that you are submitting.

AWS recognizes two different types of request identifiers: a secret key or an x.509 certificate. The x.509 certificate can only be used with SOAP transactions and can only be used with certain EC2 and SQS requests. The secret key method can be used with all of the services and for all of the request types. For that reason, I will assume you have chosen to use the secret key method and that is the method I will be using here on the blog.

Amazon allows you to regenerate your key any time you decide that you need to. Remember that your access keys are what you will use to access AWS via any third party software or external vendors (such as RightScale).

To get to the Access Identifiers, choose Your Account ? Access identifiers from the menu shown in Image 2 above. This screen will allow you to generate a new secret access key and a new x.509 security certificate.

Your access key does not change and is included on all requests to AWS. You can think of your access key as your username. Think of you secret key as your password. You can change your secret key at any time but your access key stays the same all the time.

Amazon notes on the page that you must protect your secret key and never email it to anyone. You will need to give it away under certain conditions though. When you use a third party tool like ElasticFox or Cloud Studio, you need to enter your credentials. You will also need to give your secret key to a third party vendor like RightScale who will issue requests on your behalf.

AWS Access identifiers

You can see your secret key by clicking the “+ Show” link. You will come to this screen to get your access keys. When entering the values into other tools, use cut and paste to do so. When you paste, paste it first into notepad or a VI session. For some reason, when cutting this data, it usually has several extra spaces at the end that will prevent it from working when you paste.

To generate a new key, just press the Generate button. Because this key is like a password, you may want to regenerate the key on aregular basis. If you are sharing this key at all, you will need to make sure you update anyone who has it.

]]> http://clouddb.info/2009/05/17/using-and-managing-aws-part-3-aws-security/feed/ 2