In this post I will describe how to install AppFabric Cache (AFC) and begin talking to it with a C# application. There are many useful things that can be done with AFC, but through I series of post I will discuss a scenario that I am using it for now which is to provide a scalable middle tier data service for routing events from back end systems to Silverlight clients.
But lets start with just getting this to work, which initially seems easy, but like everything else has its issues. First, we need to get a few installation packages.
Download the proper install for AFC from:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=8197ad8d-673f-4efb-b165-82710f2648c3
I'm installing this on a fresh Windows 2008 R2 box, so I also need to install .NET 4.0, which you can get here:
http://www.microsoft.com/downloads/details.aspx?FamilyID=9cfb2d51-5ff4-4491-b0e5-b386f32c0992&displaylang=en
Once you have those and .NET 4 installed, start the AppFabric Cache setup, sign your life away, and press next.
You then get the customer experience improvement dialog:
Right now I don’t want to participate in the improvement program so I just press ‘Next’, which takes us to the select services dialog:
I only need caching for this demonstration, so I select them and press next, and it takes you to the confirmation screen:
Press install. When done, you get the ‘installed ok’ screen:
At this point we need to configure the cache service, so leave the check box selected and press ‘Finish’. A few moments later you’ll be presented with the send feedback page for the configuration wizard:
I select ‘no’ and press next to get to the main configuration screen:
Now with this screen I’m not sure why it is possible to not check the Set Caching Service configuration box. I tried that, and could not get things to work (in retrospect, I think it was because of the yet to be described security issue, but it’s just best to do this with a cluster [even a one node cluster] anyway). So, check it and fill out the information as needed.
For my configuration I created an account named ‘velocity’ on the local machine and use that as the login account for the cache service; the setup program does not let you use a built in account.
I am also going to use the XML configuration technique as I want to keep this lightweight without using SQL Server. To do this you must have a network share accessible, so I created a folder on this system (VELOCITYA) and shared it as ‘VelocityConfig’.
Since this is the first system installed, I select ‘New cluster’, and I’m going to use a small cluster (1-5 machines).
Select ‘Next’ and you will be taken to the page to configure the ports that the cache will use.
Leave these as the defaults. Note, I have the windows firewall disabled so those options are also disabled.
Now press ‘Finish’, and accept the confirmation dialog. After a few seconds your configuration will be completed.
But, you are not done, yet.
There are a few powershell commands that first need to be executed. The install will have installed several menu items:
Select “Caching Administrator Windows PowerShell’. This will open a powershell console where you need to run several commands. First run this command: ‘Use-CacheCluster’.
The output is pretty basic – nothing. Now run ‘Start-CacheCluster’. A teal box will show some status of starting the cluster:
And then you’ll get some status of the nodes in the cluster.
Now that we have a one node cluster running, lets try to verify we can talk to it with some C# code.
First, you’ll need the caching client assemblies. I believe you can install these on your development system with the AppFabric Cache installer, but I grabbed mine from the cache server itself in the \windows\system32\appfabric folder:
The code I have here is the simplest code you can use to simply connect to the local system’s cache and retrieve the default cache (with the caveat of some security code needed to solve a security problem that is described soon).
When running this on the VELOCITYA server, the following is the result:
What’s the deal with this? Well, that was a lot of googling on my part to figure out, so I’ll save you the trouble.
The short of it is because the AppFabric cache is not running in a domain it is not authenticating the request, even from the local box. To fix this, we must modify our cache clusters configuration file. There are a few ways to do this, but I found it simplest to just open the XML file in the network share (\\VELOCITYA\VelocityConfig\ClusterConfig.XML"). This is the default content after the configuration that we previously performed:
We need to add the following lines (26-28):
Ideally not the best for security but it is good enough for our purposes of demonstration. Also, I needed to reboot the cache server after making this change. I tried to restart the service, but that didn’t seem to get the settings. Once successful, the program output will be the following:
Success!
Note that those lines added to the configuration file also required the addition of lines 32-34 in the source code. I’ll describe other security models in later posts, but this suffices at this point in time.
Also, please note that the install be default does not set the cache service to start automatically, so you’ll need to go and change that in the services dialog (either start it manually or configure it to auto start).
Next post... Adding items to the cache and getting notifications of new items in the cache.
a7505cbc-f77a-4fb6-87e2-72da90c172bc|0|.0
Categories: .Net, GPGPU
Posted by
mheydt on
4/4/2010 12:16 PM |
Comments (0)
I've been looking into some GPU work lately as I have a potential opportunity to potentially do a project like this for a client. I've looked at the stuff a while ago, but today a quick search seems to show the following as the libraries to use for .NET:
I'll probably give both of these over the next few weeks.
Some resources:
2bbdaa3c-f382-46df-ac25-1e7c56ac2b79|0|.0
Categories: .Net
Posted by
mheydt on
12/12/2009 8:59 PM |
Comments (0)
I've been catching up on podcasts the last few days, and there are a few cool tools and languages that I've listened about that I'd like to enumerate here.
- Solver Foundation (http://www.solverfoundation.com/). Provides a tool box and expression language in .NET for solving optimization problems. Great uses for risk management.
- Clojure (clojure.org). A .net functional language that is supposedly very similar to Lisp.
- Accelerator v2, a library from MSR for doing highly parallel computing on GPU's
562446aa-6a62-4789-b7ac-20b52d0dc095|0|.0
I'm building an application to front Twitter and provide some other services to the user around Twitter. I think it's kind of neat looking with what I've done so far, but I have a bunch of list boxes (and scrollviewers) that I think just look ugly:
I was figuring that I would want to make them look more like the scroll bars on the iPhone or in other apps I use like EventBox:
So I googled a little and came up with this, which looks to exactly fit the bill:
Mark .NET: Styling a ListBox With Silverlight 2 Beta 2 (Part 2) - Scrollbars
With some modifications, my app now looks like this....
Much Nicer!
One thing to note about the code on that site. Don't cut and paste it from the blog pages - it does not work. Get the actual
solution that is posted and use the XAML in that as it works great.
efd5cf8a-d4b3-49d5-bf69-8ee47469ff4d|0|.0
Well, I felt real good tonight when I got the EC2 library for WPF to cross compile to Silverlight, which in of itself was not as easy as it should have been, but only to get block by security violations errors which I can only assume that is because EC2 does not provide the proper policy files.
Point is, I know this, but was so lost in the "fun" of the cross compile effort that I spaced out that this would be a problem. So, its onto needing to write a back end web service to proxy the calls over to EC2 from the Silverlight client. I've done this before, and its relatively easy, and will actually allow me to use the WPF library that I wrote already, but it's a little dissappointing none the less.
5aff6e1b-5860-4dde-8186-d40f3a9c3796|0|.0
I've started the development of this library. It's been an interesting experience using the existing Amazon EC2 C# library as a base for this library. All I've got to say about that library is it really is coded weird; I'm not going to say its coded badly, lets just leave it at weird. Lets just say it really appears to be coded by a non .net person.
If you remember from an earlier post, I have a few requirements for this library:
- It should compile and run under both WPF and Silverlight with the same code base
- It should make asynchronous calls the norm (since that is the model for Silverlight)
- It should use the WebClient class, the norm for Silverlight
Yes, I'm kind of coding to the least common denominator of Silverlight (as versus to WPF), but it is a better model in my mind anyway.
One of the first issues I had was in determining how to issue REST requests for EC2 using the WebClient class. Believe it or not, I really did not find a lot of info on this while googling around. In the end, it turned out to be pretty easy. The code to do this is exemplified in this picture:
In short, this method takes a dictionary of key value pairs which are mapped to REST parameters. Each caller of this method (like the AttachVolume method) will set its parameters (at a minimum the Action key). It then adds in the required parameters in the call (access keys, signatures, ...), most of which are passed in on the constructor of this object and others calculated on the fly.
The WebClient object is then created, and the proxy added if in use, and then two headers that are required. A callback is setup for handling the async response, and then the data is passed up with a POST method using the UploadDataAsync method. I happen to pass a custom object as the UserData that encapsulates a callback passed from the calling method, and that can be seen in the explanations to come as well as by examining the async handler.
That's pretty much it to send the REST command to EC2. Now lets see how to construct a command that uses this technique.
I've been building this out with an immediate need which requires me to be able to attach and detach volumes from an EC2 instance, so I have focused on the Attach-Volume, Detach-Volume, as well as the Describe-Volumes commands (the last is needed to track the attach/detach status). As an example, lets look at the code to detach a volume:
The method first sets up the parameters required for this actual process, and then calls the invoke method to send the command to EC2. You can see here that I also pass a delegate to a callback that will handle the response (the invokes call back calls this method), and I also allow higher layers to pass their own user data (it helps this library to deal with multithreading).
The real trick now comes in the handling of the response in the callback, and where I think my approach differs greatly from the existing Amazon EC2 library. That library applies XSLT embedded in the resource to build a result object (and also does not do things async, as well as having all kinds of other overhead), where I just use a little Linq to XML to process the return XML into an actual object. Did I say I really like Linq to XML?
Matter of fact, let me show the code for the describe volumes command. This is nead as it has a link query that shows two levels of query in one statement, as the response is an array of volumes, with each volume having a potential of multiple attachement objects. The Linq to XML below does this easily and clarly in one statement:
Notice the nested from statement; Attachments is a list property of the DescribeVolumesResult class, and the "from" will be interpreted to be the currently selected volume entry in the XML as represented by the volume variable. I also really like the use of absolute XPath to select the outer level objects, and then the relative XPath that is automatically relative to the XElement selected from the outer "from".
BTW, did I say I love Linq to XML?
I have a few more layers on top of this, but this is the basic stuff to make this work. Those upper layers allow things like blocking until a detach/attach is complete, which I need for scenarios that I will also explain at another time. There is also some interesting things that I will have to say about multi-targeting this code for both WPF and Silverlight that I'll discuss at another time (but for now lets say its a pain in the arse).
67e3ce60-75bf-4bd5-9419-10003cc4f2be|0|.0
As part of all of this research into Amazon cloud computing I've started to program against the API's to see how they work. Eventually I'd like to build a client that does a lot more than what can be done with ElasticFox, which is a great tool, but falls short when you try to do things that require multiple steps. For example, one scenario that I'm already repeating all the time is the following:
1) Launch AMI
2) Mount 2 volumes to it (one step for each)
This isn't very difficult to do, but imaging if I'm building out managed services on behalf of other clients of mine, and the # of clients gets large, such that this will actually start to take a lot of time in the aggregate. I'd like to have a tool that can do this automatically, and it is possible to program this to be done through the web services.
Another scenario that I'm thinking will be useful is to be able to do automated backups of AMI's and / or volumes, and perhaps also roll back AMI's and / or volumes on a regular schedule. Again, not too hard with ElasticFox, but again a manual process that doesn't scale well.
So, I've started programming a WPF client using one of the C# api's available on the Amazon site, the
C# Library for EC2. It's been pretty easy to use so far, and I wanted something C# based instead of pure SOAP / REST. There are deficiencies in the library that I will address later, but I wanted to show what I've done so far even though it's really just proving to myself that this works.
The following is a control that I put together to manipulate S3 buckets. It's a listbox that is databound to the results from the 'listAllMyBuckets' method on the AWSAuthConnection class in the C# library:
Indeed, these are the two buckets that I have right now.
For reference, I include the XAML and the code behind for the control:
I've also coded this to show me my running AMI instances, but I'll save space here and not post that.
As I mentioned earlier I consider this library to be good but also to have a few deficiencies which I am going about fixing, and I'll probably post these changes back into the amazon code samples. My two big issues at this point with the library are:
- All the calls are synchronous, and
- The "business objects", like 'Bucket', do not utilize INotifyPropertyChanged
The first causes the UI to lag when the calls are waiting to complete. This could be addressed by running these calls in a background thread, but that only solves this for WPF. Inherently, this library will not work under Silverlight as all calls are asynchronous in Silverlight. So, all these calls need to be made to be asynchronous to support Silverlight, which I definitely want to use the library with.
The second point prevents a lot of the nice databinding capabililities, particularly two-way binding, in WPF and Silverlight. As an example, suppose you are monitoring an AMI initializing and want to change the UI when the state changes. The UI control will be bound to a 'RunningInstance' class object, and will show the current state of the instance. You'll then need to poll occasionally to see if the state changed, and when it does, you will currently need to change both the property value in the RunningInstance object, and then also code the change to the UI as the property will not advise any listeners of it's change.
Both of these aren't bad design issues in the library, as when it was built I'm sure it was not with the intention of being used in WPF and Silverlight. So, I'll fix them up and put them back in the Amazon code samples. I'll address the property update problem first as it is more important to me right now, and then handle the async calls afterwards.
Edit: I've since looked into this and to me it seems that it will be far less work to just rewrite the library. In addition to not taking advantage of either INotifyPropertyChanged, or using asynchronous calls, there are also great advantages to be had in using LINQ 2 XML to parse the results. Currently, this is all done with XSLT, and that's just way too complicated when you can use newer language constructs.
Another decision that I've come to is that since I'm going to work on rewriting this instead of extending it, I'm going to post the new code on Codeplex instead of back at Amazon. I've actually created the project and it is here
http://www.codeplex.com/AmazonAWSSharp. No code yet, but hopefully some basic S3 stuff in the next few days.
6e8e817e-feb2-41a2-bcd3-94dce739cd0b|0|.0
While doing all of this blogging on cloud computing lately, it came to my realization that I should probably put together a template/outline for a complete set of posts to cover everything that I'd like to discuss, and to perhaps serve as an outline for a book or at least a series of white papers. What I've come up with so far is below, and I'll modify this list as I go along refining the topics:
Working title: Building Cloud Applications with .NET, Azure, and Amazon
Services
I.
Introduction
1.
Overview
a.
What is cloud computing?
b.
What problems does cloud computing solve?
c.
Business Benefits of Cloud Computing
d.
Components of a cloud application
i.
Table based storage
ii.
File based storage
iii.
Service bus
iv.
Workflow
v.
Web front ends
vi.
Dynamic processing allocation
vii.
Queue based messaging
viii.
Web and worker roles
ix.
Mesh Services
x.
Data services
xi.
Geographic diversity
xii.
Elastic addresses
e.
Where are we going in this book?
2.
Cloud Computing Architectural Patterns
a.
Overview
b.
Example Architectures
i.
Processing Pipelines
ii.
Batch Processing Systems
iii.
Websites
c.
Characteristics of Cloud Systems
i.
Loosely Coupled
ii.
Self scalable
iii.
Highly Parallel
iv.
Must be reboot and relaunch resilient
v.
Distributed storage and computing supporting
failover
II.
Cloud Services: Azure and Amazon
3.
Windows Azure Services
a.
Overview
b.
Comparison to reference architecture
c.
Components of Azure
i.
Internet Service Bus (.NET Services)
ii.
Azure web applications
iii.
Mesh
iv.
SQL Data Services
4.
Amazon Services
a.
General overview
b.
S3
i.
Buckets,
ii.
…
c.
EC2
i.
AMI’s
ii.
Kernels
iii.
Security Groups
d.
SQS
e.
SimpleDB
5.
Comparison of Azure and Amazon Cloud Services
a.
Similarities and Differences
b.
Using the two together
III.
Programming the Cloud with .NET
6.
Web front ends
a.
ASP.Net applications for Azure
b.
ASP.NET on Amazon
7.
The Internet Service Bus: Connecting Desktops
and Servers to the Cloud and to each other
a.
Overview of ISB
b.
Connecting to the Cloud to request Services
c.
Integrating with Partners through the ISB
8.
Storing and Retrieving Data in the Cloud
a.
Blob storage: Azure blobs and S3 storage
b.
Table data storage: Azure tables and Amazon
SimpleDB
9.
Databases
a.
SQL Data Services
b.
Connecting to your SQL Server
c.
Amazon SimpleDB SQL
10.
Queuing: using Azure and Amazon Queues
11.
Workflow: Orchestrating the Cloud
a.
MS Workflow
b.
Self implemented workflow
12.
Mesh Services: Replicating Data
13.
IV.
Deploying, Operating and Optimizing Cloud
Applications
13.
Application Packaging:
a.
Azure packages
b.
Amazon AMI’s
14.
Deployment to the Cloud:
a.
Provisioning Azure applications
b.
Deploying AMI’s at Amazon
15.
Geodiversity
a.
Using Azure Geodiversity
b.
Amazon Availability Zones
c. Amazon Cloud Front
16.
Data partitioning
a.
Why Partition the Data?
b.
Partitioning Azure Table Data
c.
Similar for Amazon
17.
Monitoring
a.
Logging
b.
Self monitoring in the cloud
c.
Remote cloud monitoring
d.
Talking back to OSS systems
V.
Closing
18.
Best Practices for Cloud Computing
a.
Storage
b.
Queueing
c.
Table data manipulation
d.
Start multiple instances at once
e.
Plan for failure
19.
Cloud Computing Futures
a.
Automatic Demand Based Allocation
b.
Better Monitoring
c.
Windows Live Services
d.
Advanced Amazon Services
i.
Mechanical Turk
ii.
Payment Services
iii.
Search
32093343-62a8-49df-b11b-9af98eda9f99|0|.0
Categories: .Net
Posted by
mheydt on
12/17/2008 6:42 PM |
Comments (0)
Just came across this today:
VirtualBox. I'm going to give it a try. I'm a heavy vmware and parallels user, but a nice open source solution would also be really nice. I'll let everyone know what I think. Installing now...
Tags: virtual, cloud
01298964-bded-4b4c-b4c4-662dd58b9c9e|0|.0
- Cloud services have specific design considerations: Always on, distributed state, large scale, and failure handling.
- Azure is an OS for the cloud: scale out, dynamic and on demand
- Azure manages services not just servers: tell is what you want and it will help automate the details
- Frees developers from many platform issues: allows concentrating on logic instead of platform
This session will show us how this is done.
Characteristics of cloud computing
- Scale out not up
- Add and remove capacity on demand
- Pay for what you use as you go
- Automation is key to reducing costs
Design coniderations
- Failure of any node is expected: each node is a cache and must be replicated
- No one-time install step: apps need to reinitialize on restarts, don't assume previous local state is available
- Configuration changes to due to load or failures: handle dynamic configuration changes
- Services are always running: rolling up grades/downgrades; must handle data schema changes
- Services are build on multiple nodes / roles: document service architecture and communications paths
- Services can grow very large: careful state management at scale is needed
benefits of adhering to a windows Azure design point
- Azure manages services not just severs: tell it what you want and it automates; system manges nodes, services and network
- Automates sevice live-cycle management: MDA, allocation deployment snd SLA
- Turns pool of physical resources into a shared fabric: pay for what you use; platform insures isolation
Azure Service Lifecycls
- Coding and modeling
- Provisioning
- Deployment
- Maintain goal state
Service Model Guides Automation
- Describes service as distributed entities: authored by service developer, and configured by service developer
- Logical description of the services: same model used for testing and prod, mapped to hardware at deployment time
- Powerful declarative composition language
Azure Service Model Elements
- Service
- Role
- Group
- Endpoint
- Channel
- Interface
- Configuration settings
Fault Domains
- Purpose: avoid single points of failure
- Unit of failure: ex: compute node, rack of machines
- System considers fault domains when allocating service roles: ex: dont put all roles in the same rack
- Service owner assigns number required by each role: ex: 10 front-ends across 2 fault domains
Update Domains
- ensure service stays up while updating
- unit of software/configuration update: ex: set of nodes to update
- Used when rolling forward or backward
- Developer assigns # required by each role
Dynamic Configuration Settings
- Purpose is to communicate settings to service roles; there is no registry for services
- Application configuraiton settings; declared by the developer; set by deployer
- System configuration settings: pre-declared, sample kinds for all roles (instance it, fault domain id, update domain id); assigned by the system
- Both cases, available at run-time via callbacks when values change
Azure Automation
- Fabric controller: maps declarative service specs to available resources; manages service life cycle starting from the bare metal; maintains system helth and SLA
- What's special about it?: MDA
Azure Push-Button Deployment
- Allocate nodes, across fault domains, update domains
- Place OS and role images on nodes
- Configure settings
- Start Roles
- Configure load balancers
- Maintain desiered # of roles: failed roles automaticall restarted; node failure results in new nodes automatically allocated
Managing running services
- Adding capacity: push-button; steps repeated to the running service
- Removing: pb, steps reversed
- Rolling service upgrades: pb, iterative\
Rapid reliable software provisioning
- image based multicast deployment (scalable and reliable)
- seperate OS and service images: images copid, not installed; same images used for physical machines and vms
- multiple images are cached
Monitoring and Events
- Log collection
- Alerts
- Usage metering
- Data available through portal
Service Isolation and Security
- Yours are isolated from other services: model is boundary of isolation; local node resources is temp storage; network end-points
- Isoolation using multiple mechanisms
- Automatic application of windows security patches (rolling OS image upgrades)
Axure is Highly Available
- Netowrk has redudancy: switches, lb, access routers
- Services deployed across fault domains (lb's route to active nodes only)
- Fabric controller state checkpointed:can roll back to previous chekpoints
Azure automates
- Provisioning and monitoring of hardware
- Hardware life cycle mmt
- Capacity planning
- Internal security measures
Roadmap
- PDC: automated service dployment; subset of service model - simple set of service template; can change # of isntances; simple upgrades / downgrades; automated service failure and recovery; hardware mgmt; managed code / asp.net; run in fixed-size VM instances; external virtual IP address per service; service network isolation enforcement
- 2009: expose more of underlying service model, native code; multiple data centers
084540da-eb0a-4bdd-a217-7a5849684c5a|0|.0