The two primary cloud service providers available at this time are Amazon and Microsoft.
Amazon Web Services (AWS)
- Elastic Compute Cloud (EC2)
Amazon Elastic Compute Cloud (Amazon EC2) is
a web service that provides resizable compute capacity in the cloud. It
is designed to make web-scale computing easier for developers.
Amazon SimpleDB is a web service providing the core database functions
of data indexing and querying. This service works in close conjunction
with Amazon Simple Storage Service (Amazon S3) and Amazon Elastic
Compute Cloud (Amazon EC2), collectively
providing the ability to store, process and query data sets in the
cloud, making web-scale computing easier and more cost-effective for
developers.
- Simple Storage Service (S3)
Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers.
Amazon CloudFront is a web service for content delivery. It integrates
with other Amazon Web Services to give developers and businesses an
easy way to distribute content to end users with low latency, high data
transfer speeds, and no commitments.
- Simple Queue Service (SQS)
Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers. By using Amazon SQS,
developers can simply move data between distributed components of their
applications that perform different tasks, without losing messages or
requiring each component to be always available. Amazon SQS makes it easy to build an automated workflow, working in close conjunction with the Amazon Elastic Compute Cloud (Amazon EC2) and the other AWS infrastructure web services.
Microsoft Azure Services Platform
The Azure™ Services Platform is designed to help developers quickly and
easily create, deploy, manage, and distribute web services and
applications on the Internet. Windows® Azure is a cloud services
operating system that serves as the development, service hosting and
service management environment for the Azure Services Platform. Windows
Azure provides developers with on-demand compute and storage to host,
scale, and manage web applications on the internet through Microsoft
data centers.
Microsoft .NET Services are a set of Microsoft-hosted, highly scalable,
developer-oriented services that provide key building blocks required
by many cloud-based and cloud-aware applications. Much like the .NET
Framework provides higher-level class libraries that make developers
more productive, .NET Services enables a developer to focus on their
application logic rather than building and deploying their own
cloud-based infrastructure services.
The Microsoft .NET Access Control Service provides an easy way to
control web applications and services while integrating with
standards-based identity providers, including enterprise directories
and web identity systems such as Windows Live ID. Authorization
decisions can be pulled out of the application and into a set of
declarative rules that can transform incoming security claims into
claims that applications understand.
The Microsoft .NET Service Bus makes it easy to connect applications
together over the Internet. Services that register on the Bus can
easily be discovered and accessed, across any network topology. The
Service Bus provides the familiar Enterprise Service Bus application
pattern, while helping to solve some of the hard issues that arise when
implementing this pattern across network, security, and organizational
boundaries, at Internet-scale.
The Microsoft .NET Workflow Service is a high-scale host for running
workflows in the cloud. It provides a set of activities optimized for
sending, receiving, and manipulating HTTP and Service Bus messages; a
set of hosted tools to deploy, manage and track the execution of
workflow instances; and a set of management API’s. Workflows can be
constructed using the familiar Visual Studio 2008 Workflow Designer.
Microsoft® SQL Services delivers on the Microsoft Data Platform vision
of extending the SQL Server capabilities to the cloud as web-based
services, enabling you to store structured, semi-structured, and
unstructured data. SQL Services will deliver a rich set of integrated
services that you can use to perform relational queries, search,
reporting, analytics, integration and synchronize data with mobile
users, remote offices and business partners. Currently, SQL Services
offers relational database service called Microsoft® SQL Data Services.
Other services will be available in future.
Live Services is a set of building blocks within the Azure Services
Platform for handling user data and application resources. Live
Services provides developers with an easy on-ramp to build rich social
applications and experiences, across a range of digital devices that
can connect with one of the largest audiences on the Web.
In the future, developers will have access to functionality from
Microsoft SharePoint® Services in the Azure Services Platform. With the
flexibility to use familiar developer tools like Microsoft Visual
Studio, developers will be able to rapidly build applications that use
SharePoint capabilities as building blocks for their own applications.
Developers can expect a breadth of SharePoint capabilities across the
spectrum of on-premises and online applications using the Azure
Services Platform.
In the future, developers will have access to Microsoft Dynamics CRM
functionality in the Azure Services Platform. With the flexibility to
use familiar developer tools like Visual Studio, developers will be
able to rapidly build applications that use Microsoft Dynamics CRM
capabilities as building blocks for their own applications. Developers
can expect a breadth of Microsoft Dynamics CRM capabilities across the
spectrum of on-premises and online applications using the Azure
Services Platform.
4babac72-d996-4aaa-bdf1-a936e6b90738|0|.0
Amazon released the AWS Console about a week ago and I've been using it almost exclusively since instead of ElasticFox. For the most part I think it is better, but at the same time I notice things that are not quite as good. A plus is that it is accessible via any browser and any system, so you can always access your EC2 configurations no matter where or what browser you are running. A negative is that it appears to be slower than ElasticFox, and I have found myself annoyingly having to wait at various times.
One thing I like is that it opens into a dashboard. It's not a whole lot of useful information right now, but it is nice and will allow a nice overview once more features are added.
But if you go to, say the instances form, you'll see the following which isn't a whole lot different than in ElasticFox (and the same for the other forms):
Where this tool will become real interesting is when the offer the following features that they are teasing you with on the overview page:
Administrative interfaces to the other AWS services will be really nice, but I like the potential of tagging as well as the admin, load scaling and monitoring functions will be really nice.
The solution does still need a nice graphical user interface in WPF / Silverlight. I've got it in the works but it is still early. Maybe something there to talk about on the weekend.
c5e9fde6-d79a-4de1-9e01-065c06ad758b|0|.0
I've started the development of this library. It's been an interesting experience using the existing Amazon EC2 C# library as a base for this library. All I've got to say about that library is it really is coded weird; I'm not going to say its coded badly, lets just leave it at weird. Lets just say it really appears to be coded by a non .net person.
If you remember from an earlier post, I have a few requirements for this library:
- It should compile and run under both WPF and Silverlight with the same code base
- It should make asynchronous calls the norm (since that is the model for Silverlight)
- It should use the WebClient class, the norm for Silverlight
Yes, I'm kind of coding to the least common denominator of Silverlight (as versus to WPF), but it is a better model in my mind anyway.
One of the first issues I had was in determining how to issue REST requests for EC2 using the WebClient class. Believe it or not, I really did not find a lot of info on this while googling around. In the end, it turned out to be pretty easy. The code to do this is exemplified in this picture:
In short, this method takes a dictionary of key value pairs which are mapped to REST parameters. Each caller of this method (like the AttachVolume method) will set its parameters (at a minimum the Action key). It then adds in the required parameters in the call (access keys, signatures, ...), most of which are passed in on the constructor of this object and others calculated on the fly.
The WebClient object is then created, and the proxy added if in use, and then two headers that are required. A callback is setup for handling the async response, and then the data is passed up with a POST method using the UploadDataAsync method. I happen to pass a custom object as the UserData that encapsulates a callback passed from the calling method, and that can be seen in the explanations to come as well as by examining the async handler.
That's pretty much it to send the REST command to EC2. Now lets see how to construct a command that uses this technique.
I've been building this out with an immediate need which requires me to be able to attach and detach volumes from an EC2 instance, so I have focused on the Attach-Volume, Detach-Volume, as well as the Describe-Volumes commands (the last is needed to track the attach/detach status). As an example, lets look at the code to detach a volume:
The method first sets up the parameters required for this actual process, and then calls the invoke method to send the command to EC2. You can see here that I also pass a delegate to a callback that will handle the response (the invokes call back calls this method), and I also allow higher layers to pass their own user data (it helps this library to deal with multithreading).
The real trick now comes in the handling of the response in the callback, and where I think my approach differs greatly from the existing Amazon EC2 library. That library applies XSLT embedded in the resource to build a result object (and also does not do things async, as well as having all kinds of other overhead), where I just use a little Linq to XML to process the return XML into an actual object. Did I say I really like Linq to XML?
Matter of fact, let me show the code for the describe volumes command. This is nead as it has a link query that shows two levels of query in one statement, as the response is an array of volumes, with each volume having a potential of multiple attachement objects. The Linq to XML below does this easily and clarly in one statement:
Notice the nested from statement; Attachments is a list property of the DescribeVolumesResult class, and the "from" will be interpreted to be the currently selected volume entry in the XML as represented by the volume variable. I also really like the use of absolute XPath to select the outer level objects, and then the relative XPath that is automatically relative to the XElement selected from the outer "from".
BTW, did I say I love Linq to XML?
I have a few more layers on top of this, but this is the basic stuff to make this work. Those upper layers allow things like blocking until a detach/attach is complete, which I need for scenarios that I will also explain at another time. There is also some interesting things that I will have to say about multi-targeting this code for both WPF and Silverlight that I'll discuss at another time (but for now lets say its a pain in the arse).
67e3ce60-75bf-4bd5-9419-10003cc4f2be|0|.0
As part of all of this research into Amazon cloud computing I've started to program against the API's to see how they work. Eventually I'd like to build a client that does a lot more than what can be done with ElasticFox, which is a great tool, but falls short when you try to do things that require multiple steps. For example, one scenario that I'm already repeating all the time is the following:
1) Launch AMI
2) Mount 2 volumes to it (one step for each)
This isn't very difficult to do, but imaging if I'm building out managed services on behalf of other clients of mine, and the # of clients gets large, such that this will actually start to take a lot of time in the aggregate. I'd like to have a tool that can do this automatically, and it is possible to program this to be done through the web services.
Another scenario that I'm thinking will be useful is to be able to do automated backups of AMI's and / or volumes, and perhaps also roll back AMI's and / or volumes on a regular schedule. Again, not too hard with ElasticFox, but again a manual process that doesn't scale well.
So, I've started programming a WPF client using one of the C# api's available on the Amazon site, the
C# Library for EC2. It's been pretty easy to use so far, and I wanted something C# based instead of pure SOAP / REST. There are deficiencies in the library that I will address later, but I wanted to show what I've done so far even though it's really just proving to myself that this works.
The following is a control that I put together to manipulate S3 buckets. It's a listbox that is databound to the results from the 'listAllMyBuckets' method on the AWSAuthConnection class in the C# library:
Indeed, these are the two buckets that I have right now.
For reference, I include the XAML and the code behind for the control:
I've also coded this to show me my running AMI instances, but I'll save space here and not post that.
As I mentioned earlier I consider this library to be good but also to have a few deficiencies which I am going about fixing, and I'll probably post these changes back into the amazon code samples. My two big issues at this point with the library are:
- All the calls are synchronous, and
- The "business objects", like 'Bucket', do not utilize INotifyPropertyChanged
The first causes the UI to lag when the calls are waiting to complete. This could be addressed by running these calls in a background thread, but that only solves this for WPF. Inherently, this library will not work under Silverlight as all calls are asynchronous in Silverlight. So, all these calls need to be made to be asynchronous to support Silverlight, which I definitely want to use the library with.
The second point prevents a lot of the nice databinding capabililities, particularly two-way binding, in WPF and Silverlight. As an example, suppose you are monitoring an AMI initializing and want to change the UI when the state changes. The UI control will be bound to a 'RunningInstance' class object, and will show the current state of the instance. You'll then need to poll occasionally to see if the state changed, and when it does, you will currently need to change both the property value in the RunningInstance object, and then also code the change to the UI as the property will not advise any listeners of it's change.
Both of these aren't bad design issues in the library, as when it was built I'm sure it was not with the intention of being used in WPF and Silverlight. So, I'll fix them up and put them back in the Amazon code samples. I'll address the property update problem first as it is more important to me right now, and then handle the async calls afterwards.
Edit: I've since looked into this and to me it seems that it will be far less work to just rewrite the library. In addition to not taking advantage of either INotifyPropertyChanged, or using asynchronous calls, there are also great advantages to be had in using LINQ 2 XML to parse the results. Currently, this is all done with XSLT, and that's just way too complicated when you can use newer language constructs.
Another decision that I've come to is that since I'm going to work on rewriting this instead of extending it, I'm going to post the new code on Codeplex instead of back at Amazon. I've actually created the project and it is here
http://www.codeplex.com/AmazonAWSSharp. No code yet, but hopefully some basic S3 stuff in the next few days.
6e8e817e-feb2-41a2-bcd3-94dce739cd0b|0|.0
While doing all of this blogging on cloud computing lately, it came to my realization that I should probably put together a template/outline for a complete set of posts to cover everything that I'd like to discuss, and to perhaps serve as an outline for a book or at least a series of white papers. What I've come up with so far is below, and I'll modify this list as I go along refining the topics:
Working title: Building Cloud Applications with .NET, Azure, and Amazon
Services
I.
Introduction
1.
Overview
a.
What is cloud computing?
b.
What problems does cloud computing solve?
c.
Business Benefits of Cloud Computing
d.
Components of a cloud application
i.
Table based storage
ii.
File based storage
iii.
Service bus
iv.
Workflow
v.
Web front ends
vi.
Dynamic processing allocation
vii.
Queue based messaging
viii.
Web and worker roles
ix.
Mesh Services
x.
Data services
xi.
Geographic diversity
xii.
Elastic addresses
e.
Where are we going in this book?
2.
Cloud Computing Architectural Patterns
a.
Overview
b.
Example Architectures
i.
Processing Pipelines
ii.
Batch Processing Systems
iii.
Websites
c.
Characteristics of Cloud Systems
i.
Loosely Coupled
ii.
Self scalable
iii.
Highly Parallel
iv.
Must be reboot and relaunch resilient
v.
Distributed storage and computing supporting
failover
II.
Cloud Services: Azure and Amazon
3.
Windows Azure Services
a.
Overview
b.
Comparison to reference architecture
c.
Components of Azure
i.
Internet Service Bus (.NET Services)
ii.
Azure web applications
iii.
Mesh
iv.
SQL Data Services
4.
Amazon Services
a.
General overview
b.
S3
i.
Buckets,
ii.
…
c.
EC2
i.
AMI’s
ii.
Kernels
iii.
Security Groups
d.
SQS
e.
SimpleDB
5.
Comparison of Azure and Amazon Cloud Services
a.
Similarities and Differences
b.
Using the two together
III.
Programming the Cloud with .NET
6.
Web front ends
a.
ASP.Net applications for Azure
b.
ASP.NET on Amazon
7.
The Internet Service Bus: Connecting Desktops
and Servers to the Cloud and to each other
a.
Overview of ISB
b.
Connecting to the Cloud to request Services
c.
Integrating with Partners through the ISB
8.
Storing and Retrieving Data in the Cloud
a.
Blob storage: Azure blobs and S3 storage
b.
Table data storage: Azure tables and Amazon
SimpleDB
9.
Databases
a.
SQL Data Services
b.
Connecting to your SQL Server
c.
Amazon SimpleDB SQL
10.
Queuing: using Azure and Amazon Queues
11.
Workflow: Orchestrating the Cloud
a.
MS Workflow
b.
Self implemented workflow
12.
Mesh Services: Replicating Data
13.
IV.
Deploying, Operating and Optimizing Cloud
Applications
13.
Application Packaging:
a.
Azure packages
b.
Amazon AMI’s
14.
Deployment to the Cloud:
a.
Provisioning Azure applications
b.
Deploying AMI’s at Amazon
15.
Geodiversity
a.
Using Azure Geodiversity
b.
Amazon Availability Zones
c. Amazon Cloud Front
16.
Data partitioning
a.
Why Partition the Data?
b.
Partitioning Azure Table Data
c.
Similar for Amazon
17.
Monitoring
a.
Logging
b.
Self monitoring in the cloud
c.
Remote cloud monitoring
d.
Talking back to OSS systems
V.
Closing
18.
Best Practices for Cloud Computing
a.
Storage
b.
Queueing
c.
Table data manipulation
d.
Start multiple instances at once
e.
Plan for failure
19.
Cloud Computing Futures
a.
Automatic Demand Based Allocation
b.
Better Monitoring
c.
Windows Live Services
d.
Advanced Amazon Services
i.
Mechanical Turk
ii.
Payment Services
iii.
Search
32093343-62a8-49df-b11b-9af98eda9f99|0|.0