Categories: iCity, p2pSB Posted by mheydt on 9/25/2007 3:54 PM | Comments (0)
I want the operation of trackers in iCity to be more robust than in normal bittorrent clients.  In a normal bittorrent network, a client opens a torrent file which has the address of a tracker for that particular torrent, and the client connects to that tracker.  The tracker is typically a dedicated server system that coordinates communications amongst peers and may not actually share the file itself, but simply manage a client finding other clients in the network.

This makes it such that the tracker system must be online all of the time, and at a particular location (address) in the network.  This is disadventagoues for several reasons, including that 1) if the tracker goes offline, all communications cease (single point of failure), and that a client cannot always itself host tracking services.  The latter point is important as without the capability of tracking items you share, you must upload the torrent file to the tracker system for hosting which involves additional steps and gets you into the tracker availability situation previoiusly mentioned.

What I propose as an implementation of tracking in iCity is the following:

1) Any node can spin up a tracker at any time and track their own shares of packages.

2) Any node can be configured with the identity of other tracker nodes in the iCity community. They can contact those trackers and download a torrent file to download the desired file using that tracker.

3) If not desirous of self hosting a tracker, the node can ask other nodes (in particular other dedicated trackers) to host tracking for a package.  Note that this can be multiple systems to have redundant trackers to provide for greater availability.

4) If not being able to connect to any trackers for a particular package, the node can query other nodes in the community for tracking services for the package and connect to any of them.

In all, this does not change the model of tracking or the protocols involved in bittorrent tracking.  It does however build an overlay p2p network to faciliate distribution and access of tracking services for content in the community.

Categories: iCity Posted by mheydt on 9/24/2007 3:53 PM | Comments (0)
I've decided that packages in iCity should be first class entities to allow them to be located and hosted at different places in the network.  This does several important things.  From one perspective, it allows anonymous posting of data to the community so that a person can not be identified as an originator of the information.  This will be good for those that want this (you know who you are).

More importantly it solves a hairy issue in distribution (when combined with versioning).  If data did not have its own identity, it would be needed to find a particular user who is hosting the data to be be able to retrieve it.  With it's own identity, any system hosting the package could respond to queries regardless of the users of the system.

it also allows for hosting multiple copies of data in multiple locations, which would be excellent for redundancy and scalability.  Anyone (or any system) hosting the content could be then able to automatically spin up torrent services to allow replication.

Categories: p2pSB Posted by mheydt on 9/24/2007 3:51 PM | Comments (0)
I was giving thought to publishing content in iCity (formally now called nInformationCity).  Specifically, I'm trying to work up a particular use case for this.  The ones I've come by at this point are:

Use a WPF app to select files on your system for publishing.  You can then associate them with a particular community, select a package name (something to identify the files by), and select publish.  The system would then mark them as owned by you and assign it a version id (1 for a new package), and mark it as published.  Additionally, you can assign a taxonomy to the package to facilitate searching and location, as well as various other metadata fields such as a description.

What this allows is then for other users of iCity to query your system for packages.  Upon receipt of a query for packages, the system would return to the peer a summary of packages, and a further request could retrieve detail of any or all of the packages.  From that, a peer can request a transfer of a package which would be accomplished by using nDistribute.  This would then spin up a bittorrent tracker on your system (or on a dedicated tracker system) other clients can connect to in order to
transfer the content.

Even more interestingly, other peers can request a subscription to a package.  Upon changes to the package by yourself, iCity would inform peers in that community of new packages being deployed.  Or, upon subscribing to the community, all others in the community could be automatically pushed update messages to any packages published to the community.

The concept also becomes important for versioning and notification of new content.  If the "owner" of some data changes that content, others that already have the data must be able to be able to know if they do not have the most recent version, and when publishing data versions can be incremented, others notified, and content distributed.  Also, it can be used to hosting multiple versions of related data in the network if that is so desired.

I also think that it is probably important to be able to identify in the system a new package id for each version, as well as an id / version tree as if multiple people make changes to a single package, it would be good to be able to identify the history of changes for reconciliation.