Category: atom

Roy Fielding: It is only when we talk about specific applications of AtomPP, such as an authoring interface to a corporate blog, that we can say anything about the anticipated state change on the server. SUCH INFORMATION DOES NOT NEED TO BE IN THE PROTOCOL SPECIFICATION.

Is that clear? Maybe someone needs to write an application guide for transparent authoring via Atom

Maybe someone should. I’ve asked.

The Atom Publishing Protocols flexibility is its real strength, but it seems like an omission if the working group declares victory without producing an additional specification / guide that actually allows blogging clients to be interoperable with blogging servers. Wasn’t that the point?

So James pretty much single-handedly wrote abdera (now apache’s atom parser).  The one contribution I can probably claim is a performance turbo boost. The turbo comes from the fact that it is often the case that only part of the atom feed is actually used by a program. If you need all of the feed to be placed in the tree structure produced by adbera then this turbo won’t help you.

One of the things we frequently do is just pull out the titles, the links and maybe the categories for the entries so we can display them in a list. We don’t need or even want the rest of the feed structure to be parsed or resident in memory. Abdera’s FOMParserOptions allows a program to declare the elements that they want parsed ahead of the parse by setting the ParseFilter (a List of QName’s).  This provides an opportunity to optimize a given parse.

To test the improvements I wrote a little program (see below) that extracts just the entries’ titles from a feed (the feed from robubu.com as of today). I then did performance comparisons against ROME, and abdera without the ParseFilter.  Here’s the results (allocated bytes and machine instructions)

ROME:            2.64 MB, 223 mil.
Abdera:            286 KB, 30 mil.
Abdera+turbo: 66 KB, 25 mil.

So the turbo gives a slight saving for cpu but as would be expected a significant saving in terms of allocated bytes, using 40 times less memory than the same parse in ROME.  Now, I do admit that the test is biased and not that realistic, but it does give you an idea of the kinds of savings that can achieved if there are large portions of a feed that you don’t want to parse.

The code to do this is fairly straighforward.

InputStream is = this.getClass().getResourceAsStream("robubu.atom"); 

//create the list to hold the qname's to parse
List elementsToParse = new ArrayList(); 
elementsToParse.add(new 
  QName("http://www.w3.org/2005/Atom","feed"));
elementsToParse.add(new 
  QName("http://www.w3.org/2005/Atom","entry")); 
elementsToParse.add(new 
  QName("http://www.w3.org/2005/Atom","title")); 

/*create a ParserOptions object and set the 
parse filter using the list just created */ 
FOMParserOptions options = new FOMParserOptions();
options.setParseFilter(elementsToParse); 

//parse using the options 
Document doc = Parser.INSTANCE.parse(is,"",options); 

/*parse as normal, but only elements 
in the list appear in the tree*/
Feed feed = (Feed)doc.getRoot(); 
List entries = feed.getEntries(); 
Iterator i = entries.iterator(); 
while(i.hasNext()){ 
  Entry entry = (Entry)i.next(); 
  System.out.println(entry.getTitle()); 
}
is.close();

I think that this is pretty self explanatory but please let me know if it isn’t.  It is also worth noting that all the parents of any element required by the program must also appear in the filter list. i.e. in the example above "entry" and "feed" must also be in the filter along with "title" so that the title of the entries in the feed can be retrieved from the dom. Also note that if no filter is specified then all elements are parsed and available in the resulting tree. Please let us know if this is an optimization that is useful to you.

So CalAtom can get kind of complicated when it comes to querying, so before tackling the querying in that properly (I think what we have proposed to date needs a lot more work) I wanted to first see if the querying / sorting required for CardAtom could be accomplished. CardAtom would be an attempt to remote an api for managing contacts in an address book. It would store, update and manipulate collections of vCards via the Atom Publishing Protocol. The CRUD operations would be identical to those described in CalAtom only with vCard payloads (see the early slides in this CalAtom presentation for details on how this can work). While this works, APP is presently limited to always returning the entire collection and doing so in last modified order.  So how should sorting and filtering be accomplished in APP. We’ll take these one at a time.

Sorting

The atom publishing protocol mandates the collection order to be by last modified date. This is not that useful to a CardAtom client that wants to display the collection by familyname or givenname. The client could download the entire collection and then do local sorting, but as the number of contacts increases this becomes less and less viable.

Servers can produce collections in any order and make these alternative sort orders available via a url, the only tricky bit is communicating their existence and location to the client. This can, however, be accomplished by placing the sort order’s url in a link element within the original feed. The links "rel" attribute is used to indicate the particular sort order available at the url. So the feed for the vCard collection now looks like this.


<feed xmlns="http://www.w3.org/2005/Atom">
<title type="text">rob's contacts</title>
<updated>2005-07-31T12:29:29Z</updated>
<id>tag:example.org,2003:3</id>
<link rel="self" type="application/atom+xml"
href="http://example.org/contacts" />
<link
rel="http://purl.org/CardAtom/sort/byFamilyName/asc"
type="application/atom+xml"
title="by Family Name"
href="http://example.org/contacts/byFamilyName"/>
<entry>
<title>Rob Yates</title>
<summary type="html">
&lt;p>&lt;strong>Tel:&lt;/strong>+1-234-567-8901&lt;/p>
</summary>
<link rel="edit-media" type="text/directory" href="http://example.org/contact/1"/>
<id>tag:example.org,2003:3.2397</id>
<updated>2005-07-31T12:29:29Z</updated>
<content src="http://example.org/contact/1" type="text/directory" />
</entry>
<entry> .....
</feed>


Note that this uses the fact that link/@rel (as defined by atom) can actually take any arbitrary url to define its meaning. A CardAtom specification could therefore define a set of link relationships that define the mandatory and optional sorts that a CardAtom collection supports. A client reading the feed can search for a particular sort order using the value of link/@rel and if it wants to render the collection in that order it can simply retrieve the corresponding urls contents, nice.

I can also imagine "standard" sort orders being defined by specific "rel" values, e.g. "by Title" or "by Author Name".

Filtering

Filtering is much trickier. How does the server communicate to the client the searches/filters that it supports. The server could allow for very flexible and complex queries to be written in which case something like XQuery or SPARQL should be used. While extremely flexible, the problem with those is that the server MUST allow any attribute to be searched and this dramatically increases both the cost of implementation and the subsequent optimizations. For CardAtom it seems that we really only need to support full text searches as well as filtering by FamilyName and GivenName. We just need a way to describe these options to the client, and so it was that James reminded me of A9’s opensearch. Opensearch contains a description document that describes a search supported by the site. For CardAtom we want to offer a full text search, a familyName search and a givenName search. First off here is one that describes the full text search.


<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
<ShortName>Search</ShortName>
<Description>Search the contact store</Description>
<Url type="application/atom+xml"
template="http://example.org/contacts?q={searchTerms}/>
</OpenSearchDescription>


note that {searchTerms} in the url above defines where a substitution should be made.  Opensearch also defines the meaning of "searchTerms".  Then here’s the familyName search.


<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/"
xmlns:xcard="http://www.ietf.org/internet-drafts/draft-dawson-vCard-xml-dtd-
04.txt">
<ShortName>by Family Name</ShortName>
<Description>Search the contact store for a given family name</Description>
<Url type="application/atom+xml"
template="http://example.org/contacts?familyName={xcard:family}/>
</OpenSearchDescription>


Note that this one uses the xCard namespace to indicate that the substitution variable should be of type <family> as described in the xCard specification.

And so, if we make these two description files available at an appropriate location we can now link to those in the feed as well (although I only show one for brevity), e.g.


<feed xmlns="http://www.w3.org/2005/Atom">
<title type="text">rob's contacts</title>
<updated>2005-07-31T12:29:29Z</updated>
<id>tag:example.org,2003:3</id>
<link rel="self" type="application/atom+xml"
href="http://example.org/contacts" />
<link
rel="http://purl.org/CardAtom/sort/byFamilyName/asc"
type="application/atom+xml"
title="by Family Name"
href="http://example.org/contacts/byFamilyName"/>
<link
rel="http://purl.org/CardAtom/search/byFamilyName"
type="application/opensearchdescription+xml"
href="http://example.org/contacts/search/byFamilyName"/>
<entry>
<title>Rob Yates</title>
<summary type="html">
&lt;p>&lt;strong>Tel:&lt;/strong>+1-234-567-8901&lt;/p>
</summary>
<link rel="edit-media" type="text/directory" href="http://example.org/contact/1"/>
<id>tag:example.org,2003:3.2397</id>
<updated>2005-07-31T12:29:29Z</updated>
<content src="http://example.org/contact/1" type="text/directory" />
</entry>
<entry> .....
</feed>


So this seems to all be working, although we haven’t coded it yet :).  The feed/collection describes its alternate sort orders and its possible filters and how to invoke them.

One thing that I think still needs further thought though is whether these sort or search/filter links are discoverable outside of the collection document.  It seems wrong that a client must first load a collection with the default sort order only to locate the sort order that it actually wants to use  Should there be an introspection document per collection that can somehow be retrieved from the collection url?  Maybe a GET against the collection url with an accept header of "application/atomserv+xml", not sure…

 I welcome suggestions for improvements or alternative approaches, these are features that as we use Atom for more things it seems like we need. Once the core spec is complete I hope these are considerations for the working group.

Updated May 17th: I posted some of the issues that I had to the ATOM working group.  Their advice, as ever, was very useful.  I have updated the post to contain their recommendations and so no longer require content negotiation.

Updated, May 12th: In recent discussions it has been pointed out to me that I don’t need to choose a specific format for the exchange format (I had previously limited CalAtom to xCal). I have made significant changes to the post to reflect these thoughts.

CalDav is close to final call. I have read the spec and it seems that the bar to implement is fairly high. The spec is 90 pages long, excluding appendices, and it builds upon many other specifications. Google’s Calendar Data api, on the other hand, seems like a much simpler approach but doesn’t reuse any existing calendaring specifications and in no way comes close to the feature set that is in CalDav. CalAtom would be an attempt to rework CalDav to have the underlying protocol be APP instead of WebDav. I think this has the potential to simplify clients and servers and can build on some recent advances in data storage.

The first decision that needs to be made is the data format that will be exchanged by clients and servers. The most natural is iCalendar and this was the one chosen by CalDav. The iCalendar format has also been mapped to xml (xCal, xCalendar) and to rdf (rdfCal). ATOM will actually let us support and mix all of them together and this is the path that I now want to explore. It is worth noting that 30 pages of the CalDav specification are devoted to querying. It would be nice if, instead, we could use a standard query language. There have already been some examples of this. If CalAtom could use a standard search api such as sql, xquery or even sparql then that may save a lot of work, especially if our data store natively supported the chosen query language. Given that ATOM is in xml and that oracle, db2 and sqlserver now support xml storage it would seem that one avenue of investigation should be XQuery or XPath.

So, how would CalAtom work? First up, there would need to be an ATOM collection that can accept calendar posts. The ATOM introspection document would declare a collection that “accepts” data of the appropriate media type e.g. “application/calendar+xml”., “text/calendar” or “application/rdf+xml” (I can’t find a mime type for rdfCal). So with James’s new magic, proposed for the next draft of the atom publishing protocol, the introspection document looks like this.

<?xml version="1.0" encoding='utf-8'?>
<service xmlns="http://purl.org/atom/app#">
 <workspace title="Main Site" >
  <collection title="My Calendar" href="http://example.org/calendar" >
   <accept>application/calendar+xml, text/calendar, application/rdf+xml</accept>
  </collection>
 </workspace>
</service>

A client can now determine that the “My Calendar” collection will accept xCal, iCalendar and rdfCal (hand waving for rdfCal, as the accept element isn’t limiting enough). So far, so good. Now to create a Calendar event. (For anyone paying real close attention I am going to assume that we follow the same rules for the xCal posts as outlined in section 4.1 of the current CalDav draft, we probably also need some additional extensions to the introspection document that govern the type of xCal entry i.e. vType, vEvent, vJournal or vFreebusy, we’ll ignore this for now). So to create a simple calendar event this is the post (if we are posting as xCal)

POST /calendar HTTP/1.1
Host: example.org
User- Agent: Thingio/1.0
Content- Type: application/calendar+xml
Content- Length: nnn

<?xml version="1.0" encoding="UTF-8"?>
<vcalendar version="2.0" 
  prodid="-//hacksw/handcal//NONSGML 1.0//EN">
 <vevent>
  <uid>19970901T130000Z-123401@host.com</uid>
  <dtstamp>19970901T130000Z</dtstamp>
  <dtstart>19970903T163000Z</dtstart>
  <dtend>19970903T190000Z</dtend>
  <summary>Annual Employee Review</summary>
  <class>PRIVATE</class>
  <categories>
   <item>Business</item>
   <item>Human Resources</item>
  </categories>
 </vevent>
</vcalendar>

with response

HTTP/1.1 201 Created
Date: Fri, 7 Oct 2005 17:17:11 GMT
Content- Length: nnn
Content- Type: application/atom+xml; charset="utf-8"
Content- Location: http://example.org/calendar/1.atom
Location: http://example.org/calendar/1.atom

<?xml version="1.0"?>
<entry xmlns="http://www.w3.org/2005/Atom">
 <title>Annual Employee Review</title>
 <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
 <updated>2003-12-13T18:30:02Z</updated>
 <author><name>John Doe</name></author>
 <summary type="text" />
 <link rel="alternate" type="text/calendar" 
   href="http://example.org/calendar/1.ics" />
 <link rel="alternate" type="application/rdf+xml" 
   href="http://example.org/calendar/1.rdf" />
 <content type="application/calendar+xml" 
   src=http://example.org/calendar/1.xml/>
 <link rel="edit" type="application/atom+xml" 
   href="http://example.org/calendar/1.atom" />
 <link rel="edit-media" type="application/calendar+xml" 
   href=”http://example.org/calendar/1.xml” />
 <link rel="edit-media" type="text/calendar" 
   href="http://example.org/calendar/1.ics" />
 <link rel="edit-media" type="application/rdf+xml" 
   href="http://example.org/calendar/1.rdf" />
</entry>

 

So this looks like it will work. Note that the response actually lists 3 “edit-media” urls. The server has accepted a post with an xCal body and made available editable representations of it in its original xcal and also in two additional formats. The server would also accept iCalendar or rdfCal posts. This seems really nice, clients can choose the representation that they want to work with and edit that one. Conventional APP can now be used to get paged access to the collection and to manipulate the collection. This gets us CRUD operations on calendaring resources and as a bonus we also have a feed to the calendar that gets updated when entries change or get added, however it is still a long long way from CalDav.

There are several things still needed, but they all look very doable. I am going to punt on Access Control, although there are some starts on this. Access control will eventually be needed in APP and so we will wait for that discussion to get started. It also appears that CalDav’s Calendar Collection properties (section 5.2) shouldn’t be too difficult to incorporate (they are simply extensions in the Introspection document). Then the biggest remaining item and admittedly the hardest part is querying the calendar, repeating events force things to get complicated pretty quickly and so I am going to leave that for another day, although it seems that the use of XPath or Xquery is going to be the way to go here. For the record Google has very limited query capability at the moment when compared to CalDav.

CalAtom seems to hold a lot of promise, it has (in Atom) a simpler underlying model than CalDav. ATOM and APP has already enticed google into giving it a go for Calendaring and so it seems worthy of further investigation. The challenge for APP is whether it is currently specified enough to start tackling problems outside of its blogging homeland. Only time will tell.

So google has released the first mainstream implementation of the Atom Publishing Protocol (APP) for something other than blogging, Google’s new Calendar API is APP based. The question that everybody is now asking is “did they do it right?”.

Joe seems to think so, his main criticism seems to be about their security model, Elias is a little more critical and James seems to like it…….. so far. The one thing that seems very wrong to me is that they have produced yet another standard for calandering. I now have lots of questions and no real good answers, could this be done better?, can it reuse the existing calendaring standards? how does any of this relate to CalDav, is that DOA? and finally how should arbitrary applications extend ATOM for their api?

I am a big big fan of the Atom Publishing Protocol, it seems lightweight, easy to implement and “shines a light for best-practice REST“. However, its extensibility model is still getting worked out. Google’s extensibility approach uses the same extensibility model utilized by all the RSS extensions, e.g. Apple’s iTunes and Yahoo’s Media Extensions. The approach is to define new xml elements and namespaces and then sprinkle them throughout the feed. While this works it doesn’t seem to be leading to much interoperability. ATOM however has been defined to carry arbitrary payloads (RSS can only carry HTML inline) and so it should be able to play nicer with exiting standards. ATOM can, for example, carry iCalendar or xCal inline, and either of these payloads feel like a more interoperable solution for Google’s calendar api.

There seems to me to be a bigger issue lurking here, namely, what is the ATOM Publishing Protocol for? At the moment I see a spectrum of answers ranging from “it allows any blogging client to talk to any blogging server” to “it is a universal api that can be used for modeling anything, think of it like SOAP, is just an envelope”. Google is clearly leaning towards the latter, whereas the charter describes a means to edit “Weblogs, online journals, Wikis, and similar content”, is a calendar entry similar to a weblog?, is a purchase order?.

Practically speaking, this seems to boil down to whether there is a universal ATOM client? i.e. an ATOM client that can talk to any ATOM server. Google doesn’t think so (when posting Calendar entries the <gd:when> element is required), however, the ATOM working group has been running “interop testing”, so it should be possible????? Confused yet? As best I can tell ATOM clients best option is to discover the types of things that can be posted to a collection and decide whether or not it is equipped to post. In the current draft this is not really possible, but this is about to change. James has proposed an excellent set of changes that look as though they will be adopted in the next revision. They outline a means for servers to advertise the media types that they accept in collections and the ability to have ATOM entries automatically associated with the posted media. So a client could detect that a specific collection accepted “text/calendar” and then post an iCalendar file. This seems like a nice approach, that google et al could / should consider. It works very well as long as the objects being posted have defined media types.

There are still, however, a couple of interop problems. What instead would happen if we wanted to post purchase orders? or rob’s foo objects? These objects don’t have media types so how does a server advertise that it accepts them? Another problem is how a server advertises the extensions it supports in Atom Entries. A collection can only advertise that it accepts ATOM entries, but how does it communicate to the client that it accepts the “threading extension” or xCal in the entries <content> field? This seems fairly important.

The Atom Publishing Protocol is gaining momentum and google’s data api is only going to help that. The real test for ATOM now is understanding how to extend it and how to extend it in a fashion that builds on and reuses existing standards. That’ll be the key to true interoperability.