One Big Fluke

The highlight from SxSW (besides catching up with nice people) is food.

BBQ crash course (Snow's brisket in the center)

The third dimension of scaling.

Wondering about the next frontier for data systems is a recurring impulse for us programmers. Recently it's become more clear to me what the third dimension of scaling could be. The divisions blur and this is not meant to be rigorous, but I think an abbreviated timeline for each dimension helps me define progress and the trajectory. In the diagrams the green squares represent data, the outlines are machines, and the purple is for views.

1. Data
The first dimension was data. The military had their firing tables. People like Diebold envisioned data processing. All input and output data were private to the machine and user doing the computation. This was scaling up, mostly through dollars; it reminds me of the work on the moon landing.

• One machine
  
  • More bits (BCD, 12 bit, 32 bit, 64 bit)
  • More storage (memory, tape, disk)
  • Bigger chips (IBM360 had a polynomial divide instruction!)
• Big-iron super computers
  
  • Bigger memories, multi-level caches
  • Multi-processor concurrency, high-speed interconnect
  • Vast storage arrays
• Monolithic
  
  • Reliability through expensive, redundant hardware

2. Machines
The second dimension was machines. It was made possible by high-speed networking and dropping prices. Instead of scaling up this was scaling out-- as wide as possible. This has led to infrastructure as a service and the end of monolithic vendors. This effort is still underway, mostly solved, and the future holds efficiency gains more than raw performance improvements. Interestingly, in this dimension the input data is often public and the output data is also semi-public (probably the web's influence).

• Commodity hardware
  
  • Three-tier architecture (frontend, appserver, database)
  • Use a bigger DB machine and cheaper stateless nodes
  • Consistent caching for high-throughput reads
• Vertical partitions
  
  • Split MySQL into parallel DBs; overcome the limits of a single machine
  • Reliability through master/slave replication
  • Hot shards and failures handled manually
• Distributed data
  
  • Bigtable, Cassandra, and "NoSQL" systems; analysis through MapReduce
  • Hotspots handled by tablet-splitting and automatic rebalancing
  • Eventually consistent via multi-master replication, strongly consistent via Paxos

3. Views
The third dimension is views: unique perspectives on shared data where attributes vary based on who or what is viewing. Its necessity came in part from personalization and socialization on the web, but also multi-tenancy, hosted services, and growing datasets. For this dimension the input data is both public and private and the output data blends. Caching of shared, public views isn't nearly as effective. The scaling challenge is connecting disparate, multifaceted information.

• Personalization
  
  • Offline analysis of past history and behavior to derive new recommendations
  • Blend user's analyzed preferences with global data, common signals
  • Work done at query time, sometimes cacheable, often results are one-time-only
• Broadcast messaging
  
  • Fan-out of messages to per-user consumption feeds
  • Work done at write-time (Twitter), read-time (Facebook), or both
  • Message destinations are contacts, groups for access control
• Filtering and correlation
  
  • Search by keyword, geospatially across public and private data in real time
  • Cross-publisher integration with access control and immediacy
  • Relevance computed for new information at moment of collection/creation

~
I'm not sure if the "third dimension of scaling" is the perfect phrase, but I think it's good for describing where we've been and where we're going.

Tons o' Stats.

The reference PubSubHubbub hub has been growing briskly (more on that another time). We needed more fine-grained measurement of growth and the status of feed publishers and subscribers. To that end I implemented a reservoir sampler to give us a statistical estimate of the high volume traffic. This approach samples both the input key and value, so it can estimate latency and error rate (numerical metrics) easily.

Below are various screenshots of this new code in action (it's live). The subscriber diagnostics and publisher diagnostics pages on the reference hub will let you see these stats for your feed or subscriber callback's domain (if you supply the right secret token). The redacted boxes protect sensitive URLs and stats.

Own your content on the web with WebFinger

Now that all Gmail users have WebFinger enabled, its time for some real talk about why it's useful.

Tantek is passionate about his presence on the web. He's mentioned the importance of link shorteners but this has nothing to do with a 140 character limit; the concern here is attribution. When content is redistributed through services like Google Buzz and FriendFeed the URLs shared are owned by Google or Facebook. This exposes content creators to the risk of these services changing unexpectedly, but more importantly it disconnects creators from the means of distribution: links.

As a straw-man solution, let's say that Google Buzz let you redirect all of your content through your own domain (using a CNAME alias); that means your items would be shared on URLs like http://buzz.example.com/1234 instead of http://www.google.com/buzz/1234. At any time in the future you could change your links to redirect to another server (not Buzz) without losing any attribution. All of the URLs distributed throughout the web would continue to work, but you would make them point at a new place. For example, someone could export all of their Buzz data to WordPress and host it independently. Enabling URL ownership like this is an extension of the Data Liberation efforts, but for what we do on the web: share.

There are many problems with this straw man: it doesn't scale well because you would need to configure a new CNAME for every site you use on the web; it's error-prone because you could forget to set your redirection preference; and it's probably too complex for an average user to do.

Instead, what if sites could determine, just from your logged-in identity, how you want your URLs to be shared? The site would see that you're johndoe@example.com and magically determine the standard web service for generating short links on your behalf. You could configure this shortener a single time, associate it with your email address a single time, and then Google Buzz and hundreds of other sites could use your service to generate links to your content.

This is the promise of what you can do with WebFinger: associate global preferences with identity.

For the technical people, this works because the site would find a URL shortening service in the user's XRD file that looks like this:

<?xml version='1.0'?>
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
  <Subject>acct:johndoe@example.com</Subject>
  <Alias>http://johndoe.example.com</Alias>
  <Link rel='http://openshortener.org/spec' 
        href='http://johndoe.example.com/shortener'/>
</XRD>

Note that this shortening standard isn't established yet, but it should be. Defining services like this is part of what DeWitt and others meant by "work with everyone here on where we are going next". Now that we have WebFinger, we're calling on you and other developers out there to run with it: Define the services you want to use, how they will work, and provide sample implementations. This is how the ecosystem of WebFinger services will grow.