Snowl: an experiment with messaging in the browser

Last August, after Mitchell blogged about the future of email and penned an email call to action, I posted some thoughts about what a next-generation conversations app might look like.  I've kept thinking about it since then, and several months ago I proposed a labs experiment to build a Firefox extension that tries out some of those ideas.

Lots of users use desktop applications like Thunderbird and Apple Mail to converse with each other online, while many others use web apps, including not only webmail but also social networks, web discussion forums, and other site/service-specific tools.  Is there a role for Firefox in this mix?

That's the key question I'm trying to answer.

I've been hacking on a prototype on and off since then.  It's not quite usable yet, but it's getting there.  It currently has basic support for subscribing to, searching, and browsing messages from feeds via two views: a list view similar to a three-pane email application's interface and a variant of the "river of news" view described by Dave Winer.

I plan to have a 0.1 version of the prototype ready by the Mozilla Summit in two weeks, where I've proposed a session to present and discuss the experiment.  In the meantime, adventurous readers are welcome to check out the source and send me feedback via comments on this blog post, email, IRC, Facebook, your own blog post that references this one, etcetera, etcetera, and so forth.

Why I Host Projects on Mozdev

I've been using Mozdev for almost eight years, ever since it started hosting Forumzilla, my project to build a Mozilla-based web forum reader (which eventually became a Thunderbird extension and then Thunderbird's built-in feed reading feature).

Mozdev plays the critical role in the Mozilla community of hosting Mozilla-related projects that aren't considered core enough to the Mozilla mission to host on official community infrastructure.  It's a great adjunct to AMO, which distributes many of the projects that Mozdev hosts.

Over the years, I've been happy to host a number of such projects there, including the aforementioned Forumzilla plus Bugxula (defunct), Tinderstatus, and revision control/bug tracking for the Labs project Personas.

Although there have always been alternatives, they have been unpalatable: SourceForge has a cumbersome and unfamiliar interface, and hosting on my own server would have meant doing all my own systems administration.  I use Mozdev because I want to focus on my projects, not on the infrastructure for developing them.

Over the last couple of years, Google Code has become an option.  Although it provides much less functionality than Mozdev (and limits how many projects you can create), it does have a simpler interface and a more modern revision control system.

The folks who run Mozdev (which includes me) want simplicity and modern services (without sacrificing power) for Mozdev too, and we've identified three priorities in that regard: adding Mercurial for revision control, implementing WYSIWYG content editing, and automating project creation and management.

So I'm thrilled to read Mozdev developer Doug Warner's announcement today, via a post to the project owners mailing list, that he has landed Mercurial support.  Now you can get all the goodness of Hg hosted on your behalf for any of your Mozilla-related projects.

Eight years later, I think Mozdev remains the best place to host my projects, and it's just getting better.  I can't wait to see what's next (and I have some ideas about what simple WYSIWYG editing could look like, about which more soon).

Merging with Mercurial and Meld

When confronted with a merge conflict, Mercurial 0.9.5 (the version that comes with Ubuntu 8.04) uses its hgmerge script to call meld in a way that displays your version of the file on the left, the other version on the right, and hg's attempt to merge them in the middle, with conflict markers where it was unable to do so.

Here's meld handling the Merging conflicting changes example from the hg book:

You're supposed to fix the one in the middle, resolving the conflicts between the ones on either side, and then save the middle one (which will get saved to your working directory, from where you can commit your merge).

The following patch hacks hgmerge to display the base version (the one from which your version and the other version originated) on the left, the other version on the right, and your version in the middle.

--- hgmerge.orig        2008-06-01 13:51:10.000000000 -0700
+++ hgmerge     2008-06-01 14:35:46.000000000 -0700
@@ -158,13 +158,12 @@
         conflicts_or_success
     fi
 
     if [ -n "$MELD" ]; then
+        cp "$BACKUP" "$LOCAL"
         cp "$BACKUP" "$CHGTEST"
         # protect our feet - meld allows us to save to the left file
-        cp "$BACKUP" "$LOCAL.tmp.$RAND"
-        # Meld doesn't have automatic merging, so to reduce intervention
-        # use the file with conflicts
+        cp "$BASE" "$LOCAL.tmp.$RAND"
         $MELD "$LOCAL.tmp.$RAND" "$LOCAL" "$OTHER" || failure
         # Also it doesn't return good error code
         $TEST "$LOCAL" -nt "$CHGTEST" && conflicts_or_success || ask_if_merged
     fi

Here again you're supposed to fix and save the file in the middle, which gets saved to your working directory.

Here's what the example looks like using the hacked version of hgmerge:

After trying both versions of the script on several test and real merges (a few of which I screwed up), it's still not clear to me which approach is better (or if there's something better still). David Baron pointed out in a conversation that it can be useful to see the base version, but it's also useful to distinguish between your version and the version that ultimately results from the merge (which is presumably why kdiff3, which is shown in the book, displays all four: base, yours, other, and final).

Thoughts?

(Note: Devmo's Mercurial basics says to use Mercurial version 1.0 or later, although I hear that doesn't even include hgmerge, leaving configuration of a merge tool entirely up to each user, so the question is even more significant there.)

JS 1.8 syntax checking in Komodo

I've been using JavaScript 1.8 expression closures lately, but ActiveState Komodo's JS syntax checker, which is just SpiderMonkey 1.7, doesn't recognize them, so it treats them as a syntax error and stops checking the file:



Fortunately, the fix turns out to be pretty simple. Komodo ships with SpiderMonkey 1.7 as an executable, so I first built SpiderMonkey 1.8 (on Linux) using the SpiderMonkey build instructions:

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -l mozilla/js/src mozilla/js/src/config mozilla/js/src/editline mozilla/js/src/fdlibm
cd mozilla/js/src
BUILD_OPT=1 make -f Makefile.ref

Then I backed up Komodo's version and replaced it with the one I built:

cp Linux_All_OPT.OBJ/js ~/Applications/Komodo-IDE-4/lib/mozilla/js18
cd ~/Applications/Komodo-IDE-4/lib/mozilla
mv js js17
ln -s js18 js

Voila! Syntax checking works for expression closures:



In theory I might have just horked some part of Komodo that depends on JS 1.7, although the New in JavaScript 1.8 doc doesn't mention breaking changes between the two versions, and so far Komodo seems to be working fine.

If it does create problems, however, then a more contained fix, as ActiveState's Todd Whiteman points out on the Komodo Discuss mailing list, would be to update koXPCShellLinter.py's jsInterp variable to point to the 1.8 executable.

XPCOMless Preferences API

I've been working on yet another JavaScript API for accessing preferences. My goals for it are simplicity, intuitiveness, power, and perhaps performance. I'm also interested in learning whether freeing it from the restrictions of XPCOM can make it better than existing APIs.

The Basics

It's a JavaScript module, so you start by importing it from somewhere:

Components.utils.import("resource://somewhere/Preferences.js");

Getting and setting prefs is easy:

let foo = Preferences.get("extensions.test.foo");
Preferences.set("extensions.test.foo", foo);

As with FUEL's preferences API, datatypes are auto-detected, and you can pass a default value that the API will return if the pref doesn't have a value:

let foo = Preferences.get("extensions.test.nonexistent", "default value");
// foo == "default value"

Unlike FUEL, which returns null in the same situation, the module doesn't return a value when you get a nonexistent pref without specifying a default value:

let foo = Preferences.get("extensions.test.nonexistent");
// typeof foo == "undefined"

(Although the preferences service doesn't currently store null values, other interfaces like nsIVariant and nsIContentPrefService and embedded storage engines like SQLite distinguish between the null value and "doesn't have a value," as does JavaScript, so it seems more consistent and robust to do so here as well.)

Look Ma, No XPCOM

Because we aren't using XPCOM, we can include some interesting API features. First, as you may have noticed already, the interface doesn't require you to create a branch just to get a pref, but you can create one if you want to via an intuitive syntax:

let testBranch = new Preferences("extensions.test.");
// Preferences.get("extensions.test.foo") == testBranch.get("foo")

The get method uses polymorphism to enable you to retrieve multiple values in a single call, and, with JS 1.7's destructuring assignment, you can assign those values to individual variables:

let [foo, bar, baz] = testBranch.get(["foo", "bar", "baz"]);

And set lets you update multiple prefs in one call (although they still get updated individually on the backend, so each change results in a separate notification):

testBranch.set({ foo: 1, bar: "awesome", baz: true });

Performance?

Getting prefs via the module is several times slower than getting them directly from the preferences service, but it's much faster than using FUEL, and we can make the module just as fast as the direct approach by making it cache values (at some unknown set and memory cost):



Nevertheless I wonder if it's worth the added complexity and other iatrogenic costs of caching, given that preferences generally don't get accessed very frequently, and all of these methods are fast enough for small numbers of accesses.

Everything Else

I haven't yet built the rest of the API (has, reset/clear, locking, adding and removing observers, etc.). Is it worth doing so? Is this API better enough than FUEL's or simply direct access to the XPCOM preferences service? And are there other improvements we can make given that we aren't limited to the language features XPCOM supports?

(To try it out, download the Preferences and/or CachingPreferences modules.)

a trademark to distinguish network neutral providers

While on vacation last week, I saw a sign in front of a Burger King in New York advertising free wifi.  The sign listed six things you can do with it (downloading songs, chatting, etc.), which left me wondering whether they provide the whole whopping internet or just the selected portions they highlight.

And that got me wondering whether it would make sense to trademark a logo and phrase for distinguishing internet providers that provide unobstructed access to make it easier for users to pick network neutral wifi hotspots at restaurants and hotels, DSL/cable providers, and mobile plans.

Providers licensing the logo could still rate limit, as long as they didn't do so on the basis of site or protocol.  And the trademarked content would be designed to be simple and understandable by non-technical users, like Creative Commons' license logos, TRUSTe's privacy seals, and the feed icon.

I'm not sure what organization would be most suitable for running such a program, but it should be not-for-profit, as Creative Commons, TRUSTe, and Mozilla all are.

Properly implemented, it seems like a program like this would make it a whole lot easier for users to pick providers that don't have frustrating problems caused by access obstructions while putting pressure on providers to provide the whole internet so they can qualify for the program.

Thoughts?

me three

myk@myk:~$ uname -a
Linux myk 2.6.22-14-generic #1 SMP Tue Feb 12 07:42:25 UTC 2008 i686 GNU/Linux
myk@myk:~$ history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
115 cvs
61 cd
50 ls
33 safari
31 komodo
28 firefox
23 cp
16 ll
12 zip
12 scp
11 mv

proposed changes to Mozdev roadmap

I'm on the board of the Mozdev organization, and we're in the process of reviewing our roadmap for development.

We're a small organization with limited resources, so we can't do everything that would be useful, and it makes sense to leverage the tooling taking place elsewhere in the Mozilla community.

At the same time, I want Mozdev to blaze a trail on the usability and functionality of key services like website development/deployment tools and discussion forums.  And it should be super-easy to set up and manage a project on Mozdev.

So I am proposing that we revise the roadmap to make the following three items the top priorities for the organization:

1. Add one additional revision control system, and deprecate CVS.
   
   The two systems in the running are Subversion and Mercurial, and they each have their advantages.  Subversion's chief advantages are its similarity to CVS and familiarity to some existing project owners, while Mercurial's are its modern design (including support for distributed development) and its momentum in the Mozilla community.
   
   Overall, I think we're better off adding Mercurial, which has the full weight of Mozilla tooling efforts behind it and which is on track to become the most popular system in the Mozilla community (until the next one comes along, anyway).
   
   So I think we should add Mercurial to the site.  But this would not preclude adding Subversion as well at some point if resources became available to deploy and maintain it.
   
   
2. Implement a new website hosting service with simple WYSIWYG wiki publishing and scp/sftp/ftps file upload.
   
   Here we're tackling two constituencies at once: those who just want a simple hosting option where it is easy to create and edit pages, and those who want complete control over the files that make up their site.
   
   The file upload option also addresses the needs of users who generally want the simple approach but occasionally need to upload a file or two (f.e. an image to display on a page in the wiki or a presentation in PDF format).
   
   
3. Automate project creation and management so that project creation requests can be addressed in minutes or hours and users can self-manage their projects.
   
   This item combines the automate project creation process and centralized account management system items from the second and third priority groups, respectively, on the existing roadmap.
   
   I think we should continue to require approval of new project requests from unknown people but no longer require approval of those from known, trusted people like existing project owners, so that trusted people can have a new project up and running in seconds.

There are certainly many other changes that would be beneficial, like integrating forums, feeds, mailing lists, and newsgroups, so users can pick their preferred delivery format and get the same communications as on every other channel.

But these three changes would make a big difference in the usability and functionality of the site, and I think we should make them our top priorities.

Thoughts?

Dynamic Personas - How They Work

The recently released update to the Personas extension includes support for dynamic personas, which are personas that change over time.  Here's a technical overview of the history and present condition of the feature (for a non-technical overview, see the labs blog).

Take One

Original discussions for making personas more dynamic started with the idea of building an API for them to specify a series of background images and when to switch between them.  But the more ideas we had about what personas might want to do, the more complicated this API became.

I wanted something both simpler to scale to more complex functionality and more powerful right off the bat, so I suggested we simply stick iframes behind the browser chrome at the top and bottom of the browser window, let personas load any web content (HTML, SVG, etc.) into them, and let them update themselves as needed ajaxically.

That seemed promising, so I prototyped it by XBL binding the top and bottom chrome into XUL stacks, making their backgrounds transparent, and sticking iframes underneath them.

That worked great until I locked down the iframes with type="content" for security, at which point they were hoisted to the tops of the stacks and covered up the browser chrome.

I asked about this on IRC, and roc pointed me to bug 130078, which won't be fixed in Gecko 1.9.  So I had to find a different solution.

Take Two

The one I hit upon, which is in the latest release, preserves as much of the web content magic of the original solution while still working (safely).  And it still enables personas to change over time, albeit not as rapidly.

The extension creates two iframes in the hidden window, loads the persona content into them (which can still be any web content), takes a snapshot of them using the canvas 2D context's drawWindow method, converts the snapshots to data: URL-encapsulated PNG images using canvas's toDataURL method, and then makes those images the background images for the top and bottom chrome.

Rinse and Repeat

The extension then leaves the persona loaded in the iframes and periodically (once per minute by default) updates the browser chrome with new snapshots.  And occasionally (once per hour by default) it reloads the persona from scratch, although personas are of course free to update themselves more frequently.

Once per minute is obviously not fast enough for animation (like an aquarium with fish swimming around in your toolbar), but it's fast enough for gradual changes, like a panoramic landscape that darkens as the sun sets or a pictorial depiction of the weather report.  And there are plenty of interesting personas for which this update frequency is fast enough.

(Incidentally, one can jack up the frequency with a hidden preference, but doing so is not recommended, since it could impact performance).

Bits and Pieces

When dynamic personas change the background, the optimal foreground color might change too, so the extension sets the foreground (text) color to the one specified on the root element of an HTML dynamic persona.

For example, Heldenhaft's Paderborn, Germany panorama persona is dark at night and light in the daytime, so I adapted some code from an NOAA Sunrise/Sunset Calculator to enable it to determine the status of the sun at its location and set its foreground color appropriately.

And if you want to test out this "any web content" claim, just select Preferences... from the personas menu, press the Custom Persona Editor... button to open the custom persona editor, and enter any URL (f.e. http://www.mozilla.com/) into the Header or Footer fields.  It might not be pretty, but it'll show you a chunk of the page behind the chrome.

Locking it Down

The code is a actually bit more complicated than described above, because the hidden window is an HTML document on Windows and Linux, and HTML iframes can't be locked down with type="content".

So instead of creating those iframes in the hidden window, it creates another iframe in the hidden window that loads a XUL document that contains the two iframes that load the persona content.

Tests like this one (and a version that tests the personas code directly) demonstrate that the content iframes are indeed locked down, so while personas can do anything web content can do in a browser tab, they can't break out of the content jail and access chrome UI or capabilities.

The only thing injected into chrome is static PNG snapshots of web content.

Of course, if you can think of a way around that or another security issue, I and other Personas hackers would be very interested in your thoughts.  To confirm your suspicions, install and test the extension or peruse the code online.

a trio of extensions

Last week while on vacation I spent a bit of time hacking, and I cooked up three extensions to improve the Firefox experience.

The first two are pretty trivial:

Bookmark Toolbar Icons unhides those icons on the Mac using Vlad's user chrome code (with the enhancement Abdulkadir Topal suggested in his comment on Vlad's post):

Extensions are easier to install and remove than chunks of user chrome code, so this makes that enhancement accessible to greater numbers of users.

---

And Command-PageUp/PageDown lets you use those shortcuts for switching between tabs on the Mac, which is handy for selectively closing tabs via the keyboard, since otherwise you have to switch accelerators between moving to a tab (Ctrl-PageUp/PageDown) and closing it (Cmd-W).

Besides making that use case easier, the Command accelerator feels a bit more Mactuitive in general, although there's no standard shortcut for tab-switching in particular. Apple apps like Safari and Terminal use the clunky Cmd-Shift-[/], while Colloquy uses Cmd-LeftArrow/RightArrow, and ActiveState Komodo uses Cmd-PageUp/PageDown (i.e. the same shortcut this extension enables in Firefox).

---

The third extension, Bookmarks UI Consolidator, is more involved. It consolidates the Bookmarks menu in the menubar and the Smart Bookmarks folder in the Bookmarks Toolbar into a single Bookmarks folder in the toolbar:

Besides simplifying the UI, the extension makes the toolbar keyboard-accessible. Just press Alt-B (Ctrl-B on Mac) to focus and open the folder, Esc to close it again (but leave it focused), and then the LeftArrow/RightArrow keys (or Tab/Shift-Tab) to move between items on the toolbar. Space/Return/Enter on an item in the toolbar loads a focused bookmark and opens a focused folder.

(That second step in the process, hitting Esc, shouldn't be necessary, but I haven't yet figured out how to make the toolbar behave like the menubar, where hitting RightArrow while a menu is open focuses the next menu to the right if the current menuitem doesn't have a submenu.)

Note: focusing outside the toolbar turns off the focusability of its items, so you don't have to tab through all items on the toolbar every time you want to go from the Search bar to the tab strip.

And while you can't drop off the toolbar with the arrow keys, you can drop off it with Tab/Shift-Tab (as with the tab strip, although there it drops you off immediately; maybe the Bookmarks Toolbar should behave the same).

Adding this keyboard access method allowed me to simplify further by removing the Bookmarks Toolbar menuitem, which was added primarily for accessibility over in bug 408938.

And replacing Smart Bookmarks with Bookmarks saves a "smart" amount of precious horizontal toolbar real estate for user bookmarks.

I wonder how far we could take this kind of change. Could we consolidate the History menu into the Back/Forward buttons dropdown menu? Could we get rid of the menubar entirely, integrating all menu-accessible functions into other UI (perhaps a command toolbar)?

Maybe the four menu commands (Bookmark This Page, Subscribe to This Page, Bookmark All Tabs, and Organize Bookmarks) could be buttons on the Bookmarks Toolbar for better discoverability and immediate accessibility (although it would take up more of that precious toolbar space).