WebFinger, OAuth and Freebusy lookups

One of the more frustrating aspects of calendaring systems is that the freebusy lookups are all proprietary.  Meeting invitations can be sent from one system to another (assuming you know a time to meet).  However, it is not possible to lookup when someone from Lotuslive, someone from gmail and someone from Yahoo are all available to meet.  In the corporate space this type of scheduling is invaluable.

The format for looking up someone’s freebusy time is included in a standard that was completed in 1998, but they punted on all the hard stuff.  The hard bit, as I have mentioned before, is working out where someone’s freebusy is stored on the web and then authenticating with that store in a manner that can be verified. WebFinger and OAuth are now putting the complete round trip within spitting distance.

Below I’ll propose an approach to scheduling a meeting with my mom (who uses gmail) from LotusLive (which I use). I will be rob@robubu.com (but using LotusLive for my calendaring service) and my mom is mom@gmail.com.  We’ll also assume that my mom has told google that it can share her calendar free time data with any one in her contact list and that I am in her contact list.

  1. I head into my calendar service (on lotuslive), click on Add Event and type mom@gmail.com into the invitees list.
  2. LotusLive now uses WebFinger to lookup the different api services that google provides for access to my mom’s data along with the corresponding URL for the service.  The details on how this works are outlined here on Eran’s blog. At the end of this, LotusLive gets back a XRD document that looks something like the following.

    <?xml version='1.0' encoding='UTF-8'?>
    <XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
        <Subject>acct:mom@gmail.com</Subject>
        <Alias>http://www.google.com/profiles/mom</Alias>
        <Link rel='http://portablecontacts.net/spec/1.0'
              href='http://google.com/api/people/' />
        <Link rel='http://ietf.org/icalendar/freebusy'
              href='https://google.com/api/calendar/mom/freebusy/' />
    </XRD>

    From this LotusLive can now determine that my mom’s freebusy endpoint is at https://google.com/api/calendar/mom/freebusy/.  It concludes this by looking for the link with a rel attribute of http://ietf.org/icalendar/freebusy

  3. If my mom had made her freetime calendar data public then LotusLive can simply retrieve the data from the URL, but to add to the complexity let’s assume that it requires authentication i.e. LotusLive needs to prove to Google that it has rob@robubu.com at the browser and then Google checks that rob@robubu.com is in my mom’s contact list. We’ll do something here very similar to what signed fetches do in opensocial i.e. lotuslive will use OAuth to assert that it has rob@robubu.com at the browser. What we’ll end up with is a url that looks something like

    https://google.com/api/calendar/mom/freebusy/
    ?opensocial_viewer_id=rob@robyates.com
    &xoauth_public_key=https://lotuslive.com/keys/publickey.crt
    &ampoauth_signature_method=RSA-SHA1
    &oauth_signature=djr9rjt0jd78jf88%26jjd99%2524tj88uiths3

    LotusLive has here claimed that it has rob@robubu.com at the browser and using OAuth has signed the request with a private key.  It has also indicated where the public key is to validate the signature.

  4. Google receives the request, retrieves the public key and verifies the signature.  If it trusts signatures and keys from LotusLive (verifiable by retrieving certs from an https url with a lotuslive.com domain) then it is done at this point. However that is a fairly large amount of trust to place on LotusLive as LotusLive could assert on behalf of any identity. Google really needs to check that LotusLive can assert rob@robubu.com’s identity.  Here we’ll use webfinger again.
  5. Google now does a WebFinger lookup on rob@robubu.com and gets an XRD document such as the one below

    <?xml version='1.0' encoding='UTF-8'?>
    <XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
        <Subject>acct:rob@robubu.com</Subject>
        <Link rel='IDP' href='
    https://lotuslive.com' />
    </XRD>

    Google now sees that lotuslive.com is a valid Identity Provider for rob@robubu.com and so accepts the assertion.

  6. Google checks that rob@robubu.com is in my mom’s list of contacts and as I am returns her freebusy.
  7. Finally, LotusLive gets a response from Google outlining my mom’s free time and displays it in a nice calendar.  I can choose a time that she is free and send her an invite.

I know this is not perfect and I know there are probably a fair amount of changes that are needed, but I wanted to jot down something that, I think, is fairly close to a workable solution.  Am very interested in other’s thoughts.

p.s. WebFinger on email addresses does provide a means of discovering valid email addresses, but no where near as much as this does.  The fight against spam can’t center on not making email addresses discoverable.

Opensocial and OAuth specs

The REST api for opensocial makes its appearance in opensocial 0.8. The specification references some other specifications that are also worth a look.

OAuth Consumer Request – This is proposed as the means for server to server authentication between the Consumer site and the Service provider. It has the potential to replace basic auth over SSL which is the only real standards based approach for securely authenticating using a shared secret, given that digest was underspecified.

XRDS-Simple – This also looks promising and it is tackling the whole xri / yadis discovery mess that openid 2.0 seems burdened with.

Atom Publishing Protocol – not enough for blogs?

Roy Fielding: It is only when we talk about specific applications of AtomPP, such as an authoring interface to a corporate blog, that we can say anything about the anticipated state change on the server. SUCH INFORMATION DOES NOT NEED TO BE IN THE PROTOCOL SPECIFICATION.

Is that clear? Maybe someone needs to write an application guide for transparent authoring via Atom

Maybe someone should. I’ve asked.

The Atom Publishing Protocols flexibility is its real strength, but it seems like an omission if the working group declares victory without producing an additional specification / guide that actually allows blogging clients to be interoperable with blogging servers. Wasn’t that the point?

CalAtom – draft

So here is the first rough draft of the CalAtom spec.

We’ve made a few changes since the earlier posts on CalAtom, probably the biggest is the fact that CalAtom clients and servers MUST support xcal representations of calendar data.

I expect this to change a fair bit before formally posting the spec (I, at least need to spell check it), but this provides a general idea as to how it is all going to work. Feedback and anyone interested in implementing, helping out with the spec etc. are all very welcome. Please contact me or just respond to this blog post.

I know already that it is out of date as it is dependent on two specs that weren’t publically available when it was written. One now resides here and another one I believe is being published later today, I will update this post once that happens.

Abdera – Turbo

So James pretty much single-handedly wrote abdera (now apache’s atom parser).  The one contribution I can probably claim is a performance turbo boost. The turbo comes from the fact that it is often the case that only part of the atom feed is actually used by a program. If you need all of the feed to be placed in the tree structure produced by adbera then this turbo won’t help you.

One of the things we frequently do is just pull out the titles, the links and maybe the categories for the entries so we can display them in a list. We don’t need or even want the rest of the feed structure to be parsed or resident in memory. Abdera’s FOMParserOptions allows a program to declare the elements that they want parsed ahead of the parse by setting the ParseFilter (a List of QName’s).  This provides an opportunity to optimize a given parse.

To test the improvements I wrote a little program (see below) that extracts just the entries’ titles from a feed (the feed from robubu.com as of today). I then did performance comparisons against ROME, and abdera without the ParseFilter.  Here’s the results (allocated bytes and machine instructions)

ROME:            2.64 MB, 223 mil.
Abdera:            286 KB, 30 mil.
Abdera+turbo: 66 KB, 25 mil.

So the turbo gives a slight saving for cpu but as would be expected a significant saving in terms of allocated bytes, using 40 times less memory than the same parse in ROME.  Now, I do admit that the test is biased and not that realistic, but it does give you an idea of the kinds of savings that can achieved if there are large portions of a feed that you don’t want to parse.

The code to do this is fairly straighforward.

InputStream is = this.getClass().getResourceAsStream("robubu.atom"); 

//create the list to hold the qname's to parse
List elementsToParse = new ArrayList(); 
elementsToParse.add(new 
  QName("http://www.w3.org/2005/Atom","feed"));
elementsToParse.add(new 
  QName("http://www.w3.org/2005/Atom","entry")); 
elementsToParse.add(new 
  QName("http://www.w3.org/2005/Atom","title")); 

/*create a ParserOptions object and set the 
parse filter using the list just created */ 
FOMParserOptions options = new FOMParserOptions();
options.setParseFilter(elementsToParse); 

//parse using the options 
Document doc = Parser.INSTANCE.parse(is,"",options); 

/*parse as normal, but only elements 
in the list appear in the tree*/
Feed feed = (Feed)doc.getRoot(); 
List entries = feed.getEntries(); 
Iterator i = entries.iterator(); 
while(i.hasNext()){ 
  Entry entry = (Entry)i.next(); 
  System.out.println(entry.getTitle()); 
}
is.close();

I think that this is pretty self explanatory but please let me know if it isn’t.  It is also worth noting that all the parents of any element required by the program must also appear in the filter list. i.e. in the example above "entry" and "feed" must also be in the filter along with "title" so that the title of the entries in the feed can be retrieved from the dom. Also note that if no filter is specified then all elements are parsed and available in the resulting tree. Please let us know if this is an optimization that is useful to you.

Sorting and Filtering in Atom – CardAtom

So CalAtom can get kind of complicated when it comes to querying, so before tackling the querying in that properly (I think what we have proposed to date needs a lot more work) I wanted to first see if the querying / sorting required for CardAtom could be accomplished. CardAtom would be an attempt to remote an api for managing contacts in an address book. It would store, update and manipulate collections of vCards via the Atom Publishing Protocol. The CRUD operations would be identical to those described in CalAtom only with vCard payloads (see the early slides in this CalAtom presentation for details on how this can work). While this works, APP is presently limited to always returning the entire collection and doing so in last modified order.  So how should sorting and filtering be accomplished in APP. We’ll take these one at a time.

Sorting

The atom publishing protocol mandates the collection order to be by last modified date. This is not that useful to a CardAtom client that wants to display the collection by familyname or givenname. The client could download the entire collection and then do local sorting, but as the number of contacts increases this becomes less and less viable.

Servers can produce collections in any order and make these alternative sort orders available via a url, the only tricky bit is communicating their existence and location to the client. This can, however, be accomplished by placing the sort order’s url in a link element within the original feed. The links "rel" attribute is used to indicate the particular sort order available at the url. So the feed for the vCard collection now looks like this.


<feed xmlns="http://www.w3.org/2005/Atom">
<title type="text">rob's contacts</title>
<updated>2005-07-31T12:29:29Z</updated>
<id>tag:example.org,2003:3</id>
<link rel="self" type="application/atom+xml"
href="http://example.org/contacts" />
<link
rel="http://purl.org/CardAtom/sort/byFamilyName/asc"
type="application/atom+xml"
title="by Family Name"
href="http://example.org/contacts/byFamilyName"/>

<entry>
<title>Rob Yates</title>
<summary type="html">
&lt;p>&lt;strong>Tel:&lt;/strong>+1-234-567-8901&lt;/p>
</summary>
<link rel="edit-media" type="text/directory" href="http://example.org/contact/1"/>
<id>tag:example.org,2003:3.2397</id>
<updated>2005-07-31T12:29:29Z</updated>
<content src="http://example.org/contact/1" type="text/directory" />
</entry>
<entry> .....
</feed>

Note that this uses the fact that link/@rel (as defined by atom) can actually take any arbitrary url to define its meaning. A CardAtom specification could therefore define a set of link relationships that define the mandatory and optional sorts that a CardAtom collection supports. A client reading the feed can search for a particular sort order using the value of link/@rel and if it wants to render the collection in that order it can simply retrieve the corresponding urls contents, nice.

I can also imagine "standard" sort orders being defined by specific "rel" values, e.g. "by Title" or "by Author Name".

Filtering

Filtering is much trickier. How does the server communicate to the client the searches/filters that it supports. The server could allow for very flexible and complex queries to be written in which case something like XQuery or SPARQL should be used. While extremely flexible, the problem with those is that the server MUST allow any attribute to be searched and this dramatically increases both the cost of implementation and the subsequent optimizations. For CardAtom it seems that we really only need to support full text searches as well as filtering by FamilyName and GivenName. We just need a way to describe these options to the client, and so it was that James reminded me of A9’s opensearch. Opensearch contains a description document that describes a search supported by the site. For CardAtom we want to offer a full text search, a familyName search and a givenName search. First off here is one that describes the full text search.


<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
<ShortName>Search</ShortName>
<Description>Search the contact store</Description>
<Url type="application/atom+xml"
template="http://example.org/contacts?q={searchTerms}/>
</OpenSearchDescription>

note that {searchTerms} in the url above defines where a substitution should be made.  Opensearch also defines the meaning of "searchTerms".  Then here’s the familyName search.


<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/"
xmlns:xcard="http://www.ietf.org/internet-drafts/draft-dawson-vCard-xml-dtd-
04.txt">
<ShortName>by Family Name</ShortName>
<Description>Search the contact store for a given family name</Description>
<Url type="application/atom+xml"
template="http://example.org/contacts?familyName={xcard:family}/>
</OpenSearchDescription>

Note that this one uses the xCard namespace to indicate that the substitution variable should be of type <family> as described in the xCard specification.

And so, if we make these two description files available at an appropriate location we can now link to those in the feed as well (although I only show one for brevity), e.g.


<feed xmlns="http://www.w3.org/2005/Atom">
<title type="text">rob's contacts</title>
<updated>2005-07-31T12:29:29Z</updated>
<id>tag:example.org,2003:3</id>
<link rel="self" type="application/atom+xml"
href="http://example.org/contacts" />
<link
rel="http://purl.org/CardAtom/sort/byFamilyName/asc"
type="application/atom+xml"
title="by Family Name"
href="http://example.org/contacts/byFamilyName"/>

<link
rel="http://purl.org/CardAtom/search/byFamilyName"
type="application/opensearchdescription+xml"
href="http://example.org/contacts/search/byFamilyName"/>

<entry>
<title>Rob Yates</title>
<summary type="html">
&lt;p>&lt;strong>Tel:&lt;/strong>+1-234-567-8901&lt;/p>
</summary>
<link rel="edit-media" type="text/directory" href="http://example.org/contact/1"/>
<id>tag:example.org,2003:3.2397</id>
<updated>2005-07-31T12:29:29Z</updated>
<content src="http://example.org/contact/1" type="text/directory" />
</entry>
<entry> .....
</feed>

So this seems to all be working, although we haven’t coded it yet :).  The feed/collection describes its alternate sort orders and its possible filters and how to invoke them.

One thing that I think still needs further thought though is whether these sort or search/filter links are discoverable outside of the collection document.  It seems wrong that a client must first load a collection with the default sort order only to locate the sort order that it actually wants to use  Should there be an introspection document per collection that can somehow be retrieved from the collection url?  Maybe a GET against the collection url with an accept header of "application/atomserv+xml", not sure…

 I welcome suggestions for improvements or alternative approaches, these are features that as we use Atom for more things it seems like we need. Once the core spec is complete I hope these are considerations for the working group.

CalAtom

Updated May 17th: I posted some of the issues that I had to the ATOM working group.  Their advice, as ever, was very useful.  I have updated the post to contain their recommendations and so no longer require content negotiation.

Updated, May 12th: In recent discussions it has been pointed out to me that I don’t need to choose a specific format for the exchange format (I had previously limited CalAtom to xCal). I have made significant changes to the post to reflect these thoughts.

CalDav is close to final call. I have read the spec and it seems that the bar to implement is fairly high. The spec is 90 pages long, excluding appendices, and it builds upon many other specifications. Google’s Calendar Data api, on the other hand, seems like a much simpler approach but doesn’t reuse any existing calendaring specifications and in no way comes close to the feature set that is in CalDav. CalAtom would be an attempt to rework CalDav to have the underlying protocol be APP instead of WebDav. I think this has the potential to simplify clients and servers and can build on some recent advances in data storage.

The first decision that needs to be made is the data format that will be exchanged by clients and servers. The most natural is iCalendar and this was the one chosen by CalDav. The iCalendar format has also been mapped to xml (xCal, xCalendar) and to rdf (rdfCal). ATOM will actually let us support and mix all of them together and this is the path that I now want to explore. It is worth noting that 30 pages of the CalDav specification are devoted to querying. It would be nice if, instead, we could use a standard query language. There have already been some examples of this. If CalAtom could use a standard search api such as sql, xquery or even sparql then that may save a lot of work, especially if our data store natively supported the chosen query language. Given that ATOM is in xml and that oracle, db2 and sqlserver now support xml storage it would seem that one avenue of investigation should be XQuery or XPath.

So, how would CalAtom work? First up, there would need to be an ATOM collection that can accept calendar posts. The ATOM introspection document would declare a collection that “accepts” data of the appropriate media type e.g. “application/calendar+xml”., “text/calendar” or “application/rdf+xml” (I can’t find a mime type for rdfCal). So with James’s new magic, proposed for the next draft of the atom publishing protocol, the introspection document looks like this.

<?xml version="1.0" encoding='utf-8'?>
<service xmlns="http://purl.org/atom/app#">
 <workspace title="Main Site" >
  <collection title="My Calendar" href="http://example.org/calendar" >
   <accept>application/calendar+xml, text/calendar, application/rdf+xml</accept>
  </collection>
 </workspace>
</service>

A client can now determine that the “My Calendar” collection will accept xCal, iCalendar and rdfCal (hand waving for rdfCal, as the accept element isn’t limiting enough). So far, so good. Now to create a Calendar event. (For anyone paying real close attention I am going to assume that we follow the same rules for the xCal posts as outlined in section 4.1 of the current CalDav draft, we probably also need some additional extensions to the introspection document that govern the type of xCal entry i.e. vType, vEvent, vJournal or vFreebusy, we’ll ignore this for now). So to create a simple calendar event this is the post (if we are posting as xCal)

POST /calendar HTTP/1.1
Host: example.org
User- Agent: Thingio/1.0
Content- Type: application/calendar+xml
Content- Length: nnn

<?xml version="1.0" encoding="UTF-8"?>
<vcalendar version="2.0" 
  prodid="-//hacksw/handcal//NONSGML 1.0//EN">
 <vevent>
  <uid>19970901T130000Z-123401@host.com</uid>
  <dtstamp>19970901T130000Z</dtstamp>
  <dtstart>19970903T163000Z</dtstart>
  <dtend>19970903T190000Z</dtend>
  <summary>Annual Employee Review</summary>
  <class>PRIVATE</class>
  <categories>
   <item>Business</item>
   <item>Human Resources</item>
  </categories>
 </vevent>
</vcalendar>

with response

HTTP/1.1 201 Created
Date: Fri, 7 Oct 2005 17:17:11 GMT
Content- Length: nnn
Content- Type: application/atom+xml; charset="utf-8"
Content- Location: http://example.org/calendar/1.atom
Location: http://example.org/calendar/1.atom

<?xml version="1.0"?>
<entry xmlns="http://www.w3.org/2005/Atom">
 <title>Annual Employee Review</title>
 <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
 <updated>2003-12-13T18:30:02Z</updated>
 <author><name>John Doe</name></author>
 <summary type="text" />
 <link rel="alternate" type="text/calendar" 
   href="http://example.org/calendar/1.ics" />
 <link rel="alternate" type="application/rdf+xml" 
   href="http://example.org/calendar/1.rdf" />
 <content type="application/calendar+xml" 
   src=http://example.org/calendar/1.xml/>
 <link rel="edit" type="application/atom+xml" 
   href="http://example.org/calendar/1.atom" />
 <link rel="edit-media" type="application/calendar+xml" 
   href=”http://example.org/calendar/1.xml” />
 <link rel="edit-media" type="text/calendar" 
   href="http://example.org/calendar/1.ics" />
 <link rel="edit-media" type="application/rdf+xml" 
   href="http://example.org/calendar/1.rdf" />
</entry>

 

So this looks like it will work. Note that the response actually lists 3 “edit-media” urls. The server has accepted a post with an xCal body and made available editable representations of it in its original xcal and also in two additional formats. The server would also accept iCalendar or rdfCal posts. This seems really nice, clients can choose the representation that they want to work with and edit that one. Conventional APP can now be used to get paged access to the collection and to manipulate the collection. This gets us CRUD operations on calendaring resources and as a bonus we also have a feed to the calendar that gets updated when entries change or get added, however it is still a long long way from CalDav.

There are several things still needed, but they all look very doable. I am going to punt on Access Control, although there are some starts on this. Access control will eventually be needed in APP and so we will wait for that discussion to get started. It also appears that CalDav’s Calendar Collection properties (section 5.2) shouldn’t be too difficult to incorporate (they are simply extensions in the Introspection document). Then the biggest remaining item and admittedly the hardest part is querying the calendar, repeating events force things to get complicated pretty quickly and so I am going to leave that for another day, although it seems that the use of XPath or Xquery is going to be the way to go here. For the record Google has very limited query capability at the moment when compared to CalDav.

CalAtom seems to hold a lot of promise, it has (in Atom) a simpler underlying model than CalDav. ATOM and APP has already enticed google into giving it a go for Calendaring and so it seems worthy of further investigation. The challenge for APP is whether it is currently specified enough to start tackling problems outside of its blogging homeland. Only time will tell.

Customizing the Atom Publishing Protocol

So google has released the first mainstream implementation of the Atom Publishing Protocol (APP) for something other than blogging, Google’s new Calendar API is APP based. The question that everybody is now asking is “did they do it right?”.

Joe seems to think so, his main criticism seems to be about their security model, Elias is a little more critical and James seems to like it…….. so far. The one thing that seems very wrong to me is that they have produced yet another standard for calandering. I now have lots of questions and no real good answers, could this be done better?, can it reuse the existing calendaring standards? how does any of this relate to CalDav, is that DOA? and finally how should arbitrary applications extend ATOM for their api?

I am a big big fan of the Atom Publishing Protocol, it seems lightweight, easy to implement and “shines a light for best-practice REST“. However, its extensibility model is still getting worked out. Google’s extensibility approach uses the same extensibility model utilized by all the RSS extensions, e.g. Apple’s iTunes and Yahoo’s Media Extensions. The approach is to define new xml elements and namespaces and then sprinkle them throughout the feed. While this works it doesn’t seem to be leading to much interoperability. ATOM however has been defined to carry arbitrary payloads (RSS can only carry HTML inline) and so it should be able to play nicer with exiting standards. ATOM can, for example, carry iCalendar or xCal inline, and either of these payloads feel like a more interoperable solution for Google’s calendar api.

There seems to me to be a bigger issue lurking here, namely, what is the ATOM Publishing Protocol for? At the moment I see a spectrum of answers ranging from “it allows any blogging client to talk to any blogging server” to “it is a universal api that can be used for modeling anything, think of it like SOAP, is just an envelope”. Google is clearly leaning towards the latter, whereas the charter describes a means to edit “Weblogs, online journals, Wikis, and similar content”, is a calendar entry similar to a weblog?, is a purchase order?.

Practically speaking, this seems to boil down to whether there is a universal ATOM client? i.e. an ATOM client that can talk to any ATOM server. Google doesn’t think so (when posting Calendar entries the <gd:when> element is required), however, the ATOM working group has been running “interop testing”, so it should be possible????? Confused yet? As best I can tell ATOM clients best option is to discover the types of things that can be posted to a collection and decide whether or not it is equipped to post. In the current draft this is not really possible, but this is about to change. James has proposed an excellent set of changes that look as though they will be adopted in the next revision. They outline a means for servers to advertise the media types that they accept in collections and the ability to have ATOM entries automatically associated with the posted media. So a client could detect that a specific collection accepted “text/calendar” and then post an iCalendar file. This seems like a nice approach, that google et al could / should consider. It works very well as long as the objects being posted have defined media types.

There are still, however, a couple of interop problems. What instead would happen if we wanted to post purchase orders? or rob’s foo objects? These objects don’t have media types so how does a server advertise that it accepts them? Another problem is how a server advertises the extensions it supports in Atom Entries. A collection can only advertise that it accepts ATOM entries, but how does it communicate to the client that it accepts the “threading extension” or xCal in the entries <content> field? This seems fairly important.

The Atom Publishing Protocol is gaining momentum and google’s data api is only going to help that. The real test for ATOM now is understanding how to extend it and how to extend it in a fashion that builds on and reuses existing standards. That’ll be the key to true interoperability.