Archive for May, 2009

Using and Managing AWS – Part 6: SSH Key Pairs

May 26th, 2009 Comments off

Generate Your Keys

Now that you have chosen your instance, but before starting you actually start your instance, you need to generate your key pairs. The keypairs are SSH keypairs. A later post will explain SSH in greater detail but the keys come in a pair because there is both public and private components.

SSH is a Secure SHell. This is a command prompt like a DOS box or a telnet connection. However, unlike DOS and Telnet, it is very secure. The private key is the local machine’s secret password. The public key is shared to any host that the local machine will connect to.

The host is able to create a query after seeing the public key that only someone with the private key could answer. The private key is never shared but the host is convinced that it is talking to the person (or machine) that is says it is.

This may sound confusing but it is actually very secure. It’s is much better than passwords that can be hacked or accidentally given away.

Amazon supports SSH and secure communications out of the box. If you choose to revert to simple protocols such as telnet and ftp and to password authentication, you may do so. However, your first connection to any instance started through AWS will have to be via SSH. Amazon makes it easy to be secure but gives you the option of making it less secure.

So at least one pair of keys needs to be generated. Each tool set that you choose will create the files in a different way. If you are running the command line tools, you will run the ec2-add-keypair program. If running ElasticFox or CloudStudio, you will have a button on the GUI. However you create the keypair, the end result is that you will end up with a file that tends in a .pem format.

When running SSH (and the tools) from a Windows client, you will need to convert the .pem file to a PuTTY formatted key file. PuTTY, like SSH will be documented in greater detail in a near future post. Review that post for tips on Converting SSH to PuTTY.

You choose an instance’s keypair when you start it and you cannot change that after it is running. Generate your key pair and getting working first.

Using and Managing AWS – Part 5: Choosing a Machine Image

May 21st, 2009 Comments off

Choose an AMI

Amazon, and Amazon clients, are providing a huge variation of machine images. The short story is that you can choose between MS-Windows, Linux and Sun Solaris for your OS. The real story is that it is a bit more complicated than that.

The real question is what applications do you plan to run and what expertise do you have on hand or plan to hire? A quick example is a database like MySQL. MySQL runs on various operating systems. If you have Windows expertise, you may want to stick with windows. On the other hand, you can run some Linux instances with MySQL pre-installed and configured.

This about the stack that you want to run. I generally run Linux instances. They are a few cents cheaper per CPU hour and I am good enough with Linux that it doesn’t cause me any issues. I can run Oracle, MySQL and Postgres side-by-side. I occasionally do run Windows instances though just to compare offerings.

If you run SQL Server, you will need to run Windows. Almost any other software stack offers an option of OS. If you do run Windows, you will be running Windows Server 2003, in either 32 or 64 bit. SQL Server can be the Express Edition or the full blown commercial edition (which costs extra for licensing).

If you want to run Solaris, you currently have to register with Sun to get access to the OpenSolaris instance. It’s free but it requires registration. With OpenSolaris you get DTrace and ZFS, two selling points for many people.

You get OpenSolaris 2008.05 or Solaris Community Edition and pricing is the same as a Linux install. You can AMIs with AMP preinstalled as well as stacks like Drupal and MySQL.

For Linux installs, the choices are almost limitless: Fedora, CentOS, Ubuntu, Oracle Unbreakable Linux, RedHat. You name it, it’s probably there. Many of these come with pre-installed software stacks. No download and configure, just run.

Plan to try many instance types. You may even end up with a RightScale instance.

Amazon Web Services Export/Import

May 21st, 2009 Comments off

Amazon is offering an exceptionally cool new feature called “AWS Import/Export”. Basically, you ship amazon your USB or eSata device and amazon will plug it into their hardware and load it.

With terabyte datasets becoming the norm, and petabyte on the way, I knew Amazon would eventually address this in some way. They did it faster than I thought they would.

You’ll pay per device and per load hour in addition to normal S3 storage and calls. You won’t pay any transfer fees.

This will be huge for people who want to make large data sets available (internally or externally for pay) and for CDN users.

The import feature requires some advance coordination with Amazon but BucketExplorer and S3Fox already support it.

Export is TBD but I imagine you’ll send them a device and manifest and they’ll offload the bucket for you.

Read the rest of the details on the amazon blog: AWS Import/Export: Ship Us That Disk!.


Categories: cloud computing Tags:

Using and Managing AWS – Part 4: Choosing a Tool

May 19th, 2009 1 comment

Choose Your Tool

When working with AWS, you have a choice of tools. You should try several tools and use the one that works best for your needs. Some tools are provided by Amazon and others are provided by third party developers. I cover seven tools in chapters that follow this one but that list is not a comprehensive list. It’s just the tools that I have used myself and that I know for a fact do work.

Some services are more programming tools that anything else. SQS is like that. It is a queuing service that you will plug into your applications. You can interface with SQS using PHP, C#, Ruby, Perl and many other languages. Actually, you can also write interfaces to S3 or EC2 using a language of your choice but how many of us really want to write an interface?

When choosing your tool, you need to think about how you will be using AWS. Will you primarily be an S3 user? In that case you will to choose a tool with robust S3 handling. If you don’t plan to use EC2 at all, you may want to skip the tools that provide EC2 functionality and stick with the S3 browsers like S3 Browser and S3 Organizer.

On the other hand, if your only use of S3 will be snapshot backups of you EC2 instances and EBS volumes, you will want to pick a tool that helps you choose an AMI, run it and monitor it. ElasticFox and Cloud Studio are ideal for that environment.

If you plan to use both EC2 and S3 fairly heavily, Cloud Studio provides nice S3 support while ElasticFox is lacks S3 support. If you are a Firefox user though, the combination of S3 Browser and ElasticFox will provide all of the functionality you’re likely to need.

For those individuals who like pain, Amazon provides the AWS Command Line Tool set. It’s not for everyone but for those who enjoy typing at a command prompt, more power to you. Actually, I am somewhat joking. Everyone should be a little bit familiar with the command line tools just in case something goes wrong with the GUI tools.

Time will add more tools, I’m sure. The point I am making here, is to explore the options and choose a tool that fits your needs.

If you are an S3 only user, your setup needs ends here. Once you have signed up and chosen a tool, you can get started working.

The remaining posts in this series are geared towards EC2 users. Before logging into your chosen tool, you need to put some thought into your instance security, storage and usage.

A quick overview of PuTTY and SSH for AWS Newbies

May 17th, 2009 10 comments

Linux Access with SSH & PuTTY

This post will (attempt) to explain what SSH and PuTTY are so that as a user you understand the terminology of AWS and so that you can be productive in the environment. This post will not attempt to make you an expert in SSH. For best practices in implementing SSH, I strongly recommend a book dedicated to hardening *nix (Linux, Unix, Solaris, etc).


In the early days, not that long ago really, of networking, very simple tools were used to work with remote computers: telnet as a console, ftp for file copying, rsh for remote command execution and others. These were easy to configure and use tools. They were client server in that a software component needed to run on both the local machine (client) and the remote machine (server).

While easy to use, they were very insecure. They made no pretense at verifying that the calling host really was the calling host. Everything was username/password based and both the username and the password were passed around the network in cleartext. If you intercepted the little data packages that were being routed around the network (with a sniffer for example), you would be able to extract the login credentials. Even if you encrypted all of your data, your credentials were still in the clear.

SSH is an attempt (quite successful) to fix those insecurities without making things anymore complex than they need to be. SSH stands for Secure SHell. However, SSH is not really a command shell, it is rather a protocol that encrypts communications. That means that programs that use SSH can work like telnet or ftp but will be more secure.

Note: Technically, SSH is also a tool. There is a client terminal program called SSH. It’s a non-graphical command line tool that provides a window which executes a command shell on the remote system.

SSH offers multiple modes of connecting but for the purposes of AWS, we will talk about key based access. To make things more secure, EC2 uses a key based authentication. Before starting an instance, you need to create a key pair.

Note: The below explanation of SSH is a gross over simplification. I am just trying to give you a feel for what is going on. If you really want to understand the technical details, I really do recommend that you purchase a book. My personal recommendation is SSH, The Secure Shell: The Definitive Guide from O’Reilly.

When an instance starts up for the first time, EC2 copies the ssh key that you created to the proper directory on the remote server. The remote server will be running the SSH Server software.

You will then use an SSH client to connect to the server. The client will ask for some information proving that the server really is who it says it is. The first time you connect to a server, the client won’t have that information available so it will prompt you to vertify that the server is legitimate.

You verify that information by comparing a thumbprint. Verifying a host is a bit beyond this book but do an internet search for for “ssh host thumbprint”. You’ll find a variety of articles explaining it in detail.

Once the client accepts the host, the client will send secret information to the host. This is your key data. If the host is able to make a match, it will authenticate you and let you login in. If the host then asks for a password, you key did not work and something is not configured properly. In my experience, it will probably be that your client key file is not in the place your client is expecting it to be.

What happens next depends on the tool you are using. If you are using a terminal program, ssh for example, you will now have a command prompt. If you are using sftp or scp, you will be able to copy files.

In addition to command line tools, there are GUI tools that use the SSH protocol. WinSCP is an excellent SCP client for Windows.

Regardless of the tools you use, SSH is busy encrypting everything you send over the wire. The SSH protocol has evolved over the years, and will probably evolve even more in the future, but it is currently running a very secure form of encryption.

If you are running Linux, you are pretty much finished at this point. SSH ships with every Linux distribution that I am aware of. If you are using Windows, however, you either need to install CyWin (a unix environment that runs in windows), or you’ll want to get PuTTY.


You can download all of the programs discussed in this section at:

I honestly have no idea why PuTTY is spelled PuTTY. I can figure the TTY part of it is from the Unix command that output a display. I’m not sure bout the Pu though.

I do know what PuTTY is though. PuTTY is a very simple implementation of an MS-Windows SSH terminal client. When I say it is simple, I mean that as a complement. This is a tool that does not get in the way.

You tell PuTTY to connect to a remote server and, as long as your keys are configured, it will connect you. If are not using keys, you can connect with passwords (if the host allows that). As a best practice, keys are recommends over passwords.

PuTTY is the terminal client but you can get a couple of other tools from the same author. PSFTP and PSCP offer secure file transfers. These tools are as easy to use as PuTTY and work pretty much the same way.

For command line syntax and configuration, take a look at the documentation at the link above.

A note about SSH keys and PuTTY, they are not compatible. This same web site offers a utility called PuTTYgen. When you create a key pair for EC2, you download that file to your local machine. PuTTYgen converts that file (a .pem file) to a private key file (a .ppk file).

PuTTY Key Generator

PuTTY Key Generator

The tool is named puttygen.exe. Run the executable and the above window pops up. To convert an amazon key to a PuTTY key, use the menu option Conversions ? Import Key. Load the .pem file that you downloaded and press the Save Private Key button.

It will warn you about leaving the passphrase blank. That’s ok.

Save the file to the location that PuTTY has been configured to look in for it’s keys.