Design Rationale

Why are you using Ajax/DHTML and not a Firefox plugin or a server-side approach?

Mainly for ease of installation.

Writing a Firefox plugin isn't necessary to do what we want to do. By using Ajax/DHTML, installing HyperScope boils down to putting some static files on your server. Using Ajax/DHTML also allows us to support other browsers, such as Internet Explorer.

The HyperScope does require a web server -- see below for details. However, it does not require a server-side application... yet. One reason we avoided implementing some of HyperScope's capabilities on the server-side was that many of its features require a fast interactive response time. One of the original insights of the Augment system was that if you speed up the amount of interaction a user can do with a system, then you can drastically improve their ability to do knowledge work.

In the future, HyperScope will need a server-side component to implement some additional capabilities, such as editing.

Why OPML and not HTML, XOXO, semantic HTML, or your own XML file format?

Rolling our own would have been silly. There are already good formats that meet our needs. Our goal is to bootstrap.

The reasons we did not choose HTML, XOXO, or semantic HTML are subtle. Brad did a lot of prototyping before settling on a design and discovered some issues when experimenting with an an HTML version of the file-format on the client:

Doing HyperScope's addressing, jumping, and viewing by walking the DOM with JavaScript is very slow, especially with complex addresses
XPath doesn't work on an HTML DOM on Internet Explorer.
XPath will work fine with XHTML and XOXO (both XML-based), but the XPath and design becomes much more complex, because both XHTML and XOXO are verbose and do not represent the underlying hierarchy in a simple, natural way.
Finally, applying successive viewspecs to a rendered version of a document is error prone, because you have to remember to reset parts of the DOM everytime an address, viewspec, content filter, transclusion, etc. changes. This is tricky without a more robust XML file format that is independent of the rendered view.

OPML exists and has a large community of users and tools, including editors, converters, web services etc. Although we lightly sprinkle our own namespaced attributes into the OPML document to support extra addressing types, HyperScope will work with existing, non-HyperScope OPML documents.

Why do the full pipeline everytime?

Brad (our lead developer) chose correctness over performance. Every relative address is converted into a fully qualified address, which has enough information to fully render it using the current state. For example, if you were to enter #.n:zA for Jump to Item after successively applying the following viewspecs on seperate occasions: y, m, B, P, g, and after jumping to node ID 023, where the current document is http://bootstrap.org/hyarch.opml, this would get expanded to be the following by hs.address.Address.resolve():

  http://bootstrap.org/hyarch.opml#023.n:ymBPgzA

We expanded the file portion from # to:

  http://bootstrap.org/hyarch.opml#

then expanded the context node to start at 023, then prepended all of our existing viewspecs before our new ones to ensure that old views are maintained even with relative links.

The pipeline internally encapsulates this complexity with the resolve() method. Since everything follows the same process, we reduce the number of code paths to get higher reliability, and we create a single place to optimize the performance which will benefit all address resolution. The unit tests work hand in hand with this to make sure we don't get regressions and handle all cases correctly.

Why don't you support cross-domain document loading?

The browser has a security policy that does not normally allow a document from one domain to interact with documents from another domain. In other words, you cannot open an XMLHttpRequest to a different domain than where you came from.

There is an edge condition related to transclusion that on its surface seems like it needs a traditional server-side component to get past this browser restriction. Suppose you have a document that does some transclusion on a document on another domain:

  INCLUDE "http://someotherdomain.org/foobar.opml#055"

If the HyperScope document was served from http://foobar.com, then I would get a security exception when I tried to open an XMLHttpRequest to that other domain to grab the transcluded fragment and include it at rendering time.

There are two solutions for this. The easiest/most obvious solution would be to have a simple PHP script that simply proxies the OPML file request. This has to be written carefully so you don't have a completely open proxy, but if you restrict it to just fetching XML MIME types then you are fine. HyperScope simply calls this PHP script, which lives on the same host as it was retrieved from, and the proxy contacts the external domain. This is a mature and traditional way to do things.

Another possibility would be to run several web services from some centralized server. One of them could take a third-party URL and fetch it's contents, then return it. It would basically be the PHP script described above, and would only return XML/OPML:

  http://bootstrap.org/hyperscope/proxy.php?url=http://foobar.com/arch.opml

The other web service would be a converter that would translate different file formats into HyperScope OPML documents. It would take a URL to some external content that is in some format, such as Microsoft Word, XOXO, arbitrary XML, etc., and simply convert it into HyperScope OPML on the fly and return it:

  http://bootstrap.org/hyperscope/convertor.php?url=http://w3c.org/xpath_spec.html

This PHP script would simply delegate conversion locally using tools such as a headless version of OpenOffice running locally. HyperScope would simply use these webservices to get its job done, including transclusion, document conversion, etc. Those who want to install HyperScope now don't have to install anything on the server-side except for uploading a few static files.

How can the browser call these web services with the browser security policy described above if it is served from a different host? We use something called FlashXmlHttpRequest, which Julien Couvreur created and which uses the dojo.flash library Brad created for another project. This uses a feature of flash which can call external webservices if they configure themselves correctly, which basically just means putting a simple file called crossdomain.xml at their root with some info:

  http://bootstrap.org/crossdomain.xml

Now our HyperScope client can call our webservices using FlashXmlHttpRequest, getting back normal XML/OPML that we can work with.