Session: Essential Cloud Storage

by mheydt 28. October 2008 19:24 >
REST and ADO.NET interfaces
First need to create a storage account
  • gives you a 256-bit secret key
  • Will provide security for accessing the store
Abstractions
  • Blobs (names files and metadata)
  • Tables (structured set of entities)
  • Queues
Concepts
  • Storage account
  • Container
  • Blobs
Sharing policies is at the container level

Notes on Blob access:
  • 50GB total blob storage limit for now.
  • Put/Get/Delete API. 
  • 64MB limit per blob in CTP with PutBlob.  larger must use PutBlock. 
  • Block is up to 4MB.
  • Can associate metadata with the blob: name value pairs, 8KB limit per blob
  • Uploads can be resumed: blobs broken in to chunks.  PutBlock, PubBlockList API's. 
  • Blocks not use will be GC'd. 
  • GetBLockList can be retrieved for a blob. 
  • MD5 can be put in the REST PutBlock and the storage system will use it.
  • Not a real directory structure, but delimiters can be specified that simulate it.
  • Update, Append and Copy coming in the future.
Notes on Tables:
  • Massively scalable and durable (replicated at least 3 times)
  • Can be accessed by ado.net 3.5 sp1
  • Table is created in and scoped by account
  • Table is a set of entities
  • Entity is a set of properties
  • Partition Key is exposed in the programming model, and helps to assist with scalability. 
  • Partition gives entity locality (stored together - better query), and scalability (usage patterns monitored and used for load balancing)
  • partition key should be used in query to optimize, otherwise all partitions must be examined.
  • Entity can have up to 255 properties, including Partition Key, Row Key, and one system maintained version property
  • No fixed schema
  • Pertition Key and Row key string up to 64KB
  • Table Entities access view ADO.NET DS, LINQ, and REST
  • Insert/Update/Delete, query over tables
  • Tables represented as class objects, adorn with DataServiceKey attribute
  • Every account has a master table called "Tables"
  • Use a DataServiceContext object to get entry point to the storage account
  • LINQ works just like you expect
  • Optimistic concurrency checking
  • Not a relational database: no joins, foreign keys, etc
  • For now, results are sorted by partitionkey and rowkey.  seconday indexes in the future
  • Also, single entity transactions for now, future will be multiple entities
Notes on Queues:
  • Provides reliable delivery of messages between workers and roles
  • No limit on # of messages, but they can only live for a week
  • Message <= 8KB.  Larger store a blob and reference it in a message
  • Create/Clear/Delete, Inspect length of queue
  • Messages: enqueue, dequeue, delete
Notes on durability, avail, scale
  • Data replicated at least 3 times across different fault and upgrade domains in the same data center
  • Geo-diversity will be after the CTP
  • Efficient filover
  • Automatic load balance hot data
  • Monitor usage patterns fro optimization
  • Caching of hot blobs, entities and queues

Tags:

Azure | PDC08

blog comments powered by Disqus

about the author

I'm a .NET, XAML, and iOS polyglot that loves playing with new things and making cool and innovative stuff.  I am also a Mac junkie.

I am Principal Technologist for SunGard Global Services in NYC, in their Advanced Technologies practice, and I work extensively with SunGard's energy and financial customers.

Note the the posting on this blog are my own and do not represent the position, strategies or opinions of SGS.

twitter

I can't stop thinking big!
Sunday 1:08AM via WindowsLive
Just watched Moneyball. That's my pick for best movie this year.
Saturday 3:51PM via WindowsLive
@vincebelpiede: Report: Skype For Windows Phone Beta Imminent http://t.co/KYNjgg1L#mhtnd
Wednesday 8:39AM via Twitter for Mac
@mashable: Kinect Fusion Will Turn Gaming (and More) Into a 3D Fun House - http://t.co/Ihrq2fY2#mhtnd
Wednesday 8:39AM via Twitter for Mac
New Kinect SDK: http://t.co/57MvA5L5 #mhtnd
Wednesday 8:39AM via Twitter for Mac
Follow me on Twitter

recent comments

None

month list