Building Firefox in the Post-Browser Age
Posted: October 25, 2011 Filed under: firefox, mozilla | Tags: firefox, jetpack, mozilla 12 CommentsOk, so maybe we aren’t in the post-browser age *yet*. But we’re getting there, and quickly. Most of the “apps” I use on my phone are useless without an always-on data connection, and they communicate with their respective motherships via HTTP. We’re staring down a near-future with multiple Web-stack operating systems for both desktop and portable devices. We have server-side application platforms that look startlingly like pieces of a traditional Web client.
All of those places are the Web, so that’s where Mozilla has got to be, when and if it’s possible to do so. And between desktop Firefox, mobile Firefox, Chromeless, B2G, Pancake, Open Web Apps and the various Firefox features developed by the Labs and the Services groups, we’ve got a lot of application logic that needs to exist in various forms across those disparate environments.
Up until recently, even including most of the mobile efforts to date, we’ve had a pretty narrow idea of what constitutes Firefox: Mozilla’s browser application with a front-end built in XUL, and rendering content using Gecko, stored entirely in a single source repository.
This narrow view is insufficient given the needs of internet users and the plans we have to serve those needs in the immediate future. This has been starkly illustrated by the recent move to a native UI for mobile Firefox, projects like Pancake, and the expansion of the development of Firefox features by groups outside of the core team.
A few months ago I dumped a couple of thoughts into a thread on the mozilla.dev.planning newsgroup about these things. More than anything, that thread showed me that the broad spectrum of activity in Mozilla today makes our narrow view of Firefox a huge barrier to future success. Some people didn’t agree that there was a problem at all. Some people railed against Jetpack or Github, while admitting they’d never used either. Some people agreed that developing Firefox is slow and fragile, and pointed at the relative historical success of that approach. Disturbingly, I got a bunch of private emails thanking me for starting the conversation… what does *that* mean?! Overall though, there was a lot of agreement on this point: We need more people to be able to work on Firefox faster, and in a more heterogeneous environment.
There’s a bunch of work towards that end going on right now, both in Firefox team itself and in Mozilla generally, around lowering the barriers to contribution. Specific to Firefox core development though, one experiment in alternate approaches is the attempt to ship the BrowserID feature as a Jetpack-based add-on that is developed on Github and bundled with Firefox. There are a lot of moving parts, but the exercise is helping us figure out the up- and downsides to building features as add-ons, as well as providing performance data about the Add-on SDK. Maybe it’ll work, maybe we’ll have to re-route and patch it against the core. Maybe we’ll land somewhere in-between.
Regardless of that experiment’s outcome, I think we need to be experimenting hard with how we develop Firefox, and asking questions about the longer-term development landscape:
- Code changes currently have non-deterministic effects in the Firefox ecosystem. We have a jumble of services that stagger into existence at startup, and then race for the exit at shutdown, beating up the file-system at both ends of the application lifecycle. “Async” is a pattern, not a system – without a system, making a bunch of things asynchronous means that the application’s behavior as a whole is generally less predictable. Is there a more systematic way that we can manage the loading and unloading of core browser services?
- Calcification: Check out the “cleanup page”. There are long-despised-and-untouched pieces of our core infrastructure, such as URL classifier, importers, autocomplete, and parts of Places. Why is it so hard to change these? What are the barriers to making them better?
- Modularity: Cu.import is great in that it provides some of the benefits that we used XPCOM JS services for, but without the XPCOM. But are we using it enough? Jetpack development puts much more emphasis on modularity via a core built on CommonJS, and I’ve found it to make browser features written in Jetpack far easier to follow, debug, and contribute to. Maybe we should be putting code into modules where we’d normally add it to browser.js, or XBL widgets moreso than we are now? This could reduce our dependence on the XUL window mega-scope that we get in browser.js, which I’d argue leads to code that is easier to developer, debug, test and maintain.
- Abstracting the application logic away from XUL/XPCOM where possible, allowing for more portable code. This doesn’t make sense in a lot of places in the front-end, but in others such as sync or expiration policies or tab grouping algorithms or frecency generation, it might. These are things which could be useful across a number of different application contexts.
So where from here? There’s general agreement that the Add-on SDK needs to ship in the browser. This might help address some of the questions above. However, it won’t immediately help us share code with other Firefoxes or Mozilla projects, or make core development inherently less-fragile or our application behavior any more deterministic. And there are tools like Cu.import, which we have now, and Harmony modules, which we might have soon (can we use those in chrome?!) that could help with the modularity part.
But only some of this is about the technology – other parts are social. As I said above, some people do not agree that developing Firefox is slow and fraught with peril. Is that plain ol’ resistance to change, or just the lack of a clear alternative? And maybe we code reviewers should be more forward-looking, demanding larger refactorings instead of non-invasive surgeries. But that’s challenging when you’re constrained for time, or the regression cost of refactoring is so high that you become risk averse.
I’d love to hear your thoughts on the future of Firefox application development – especially the core Firefox team, and the people working on Firefox features in other groups or via add-ons. Myk Melez has been corralling a group to talk about feature development with the Add-on SDK specifically, but it quickly spreads into these broader issues. He’s starting a list for it, but until then there are regular meetings, details available in his dev.planning post.
dietrich.ayala 
Strangely, I just bought this up in a blog post recently and so will summarise. The issue Firefox faces is that it’s built to be a product where as it needs to essentially be a platform. It needs to be dissected into services and those services need be standalone. Those services need to only be allowed to communicate with each other via API. You should be able to drop in and out services to achieve different things. Until there’s a modular platform in place, Mozilla will suffer and things like new features or new platforms will suffer.
Wow, you really liked Steve Yegge’s Google/Amazon post, huh?
I loved it, it subscribes to my idealogical approach for Firefox and many other software projects. As did Schmidt’s Salesforce interview before it and Joe Walker’s comment on this very post. Even if you look at Robert Kaiser’s comment on this very blog, you can see that a more modular approach should be taken as there’s questions of what constitutes Firefox and fears as though the identity is being lost. Is Firefox a self-contained browser free or native code or is Firefox the best browser out there that’s goal is the freedom of everyone on the internet? Does something stop being Firefox because it has a different GUI? Personally I’d say not, but clearly this isn’t an opinion everyone holds. With modularisation such a question needn’t be asked. There is a platform there already and there’s a product there, it’s natural evolution to grow the former in order strengthen the latter.
For one thing, a lot of the things you call “Firefox” in here aren’t Firefox. They might be part of Mozilla, but Firefox isn’t all of that, it’s the Gecko-based browser with a Gecko-rendered UI (and no, that “native UI” mobile thing doesn’t deserve the name “Firefox”, probably not even “Mozilla”, it’s more a Netscape thing, a reminder of the non-innovative non-open-web past that Mozilla should have left behind years ago – but even B2G, which is clearly innovative and open web is not Firefox). We are overusing the brand “Firefox” until it’s so diluted that nobody knows any more what it stands for and the brand loses all its value. I don’t want that to happen.
For the other thing, I don’t agree that developing Firefox is slow, though it might be true for parts of it, esp. those that have been done in times when the goals were more important than the technology, i.e. when Firefox was first created. Code from those times still haunts us. There are surely ways to improve how we are doing things, and we need to follow those. But burning down all bridges behind us is not the way to do it. There is a lot of good stuff in our code. Let’s not burn down all of it but still innovate for the future. We surely need evolution, but revolution is not the way – it always creates more trouble and dangers than it helps, and usually turns out with those being winners that weren’t involved in steering it up. Which just might mean someone other than Mozilla and the Open Web.
Let’s move forward, but do it carefully. If we evolve rather than re-invent Mozilla, we will stay successful.
You might have misunderstood the post. I didn’t call for a revolution, or say to throw away all the Firefox code.
I think developing Firefox is slow, difficult to learn, and terribly regression prone. I think there are some architectural approaches we can take that will improve those things, and I’m encouraging people to think about that.
And whatever we call those other projects, almost all of them need to play with Firefox in one way or another. I’ll leave the naming stuff for the bikeshedders 🙂
Would it be fair to sum up the post in the question “What do we do to scale better?”
I think the core answer to that question comes down to “Decrease the dependancies between the parts”.
So things like having clearly defined APIs are a good thing, separate repos are a good thing, etc, etc, etc. Anything that allows me to fail without affecting you is a good thing. Anything that allows me to be re-written from scratch without affecting you is a good thing.
Yes! That’s a fantastic summary of the problem. One scaling bit is the code, which is what this post mostly focuses on.
Another is enabling contribution. The Stewards group is talking a lot about reducing the barriers to contribution, and retaining contributors – work that will definitely help us scale.
Great post, Dietrich. We’ve been noodling around on some of these issues in the devtools group without any great answers so far. As more of our code lands, we’re starting to see things like experimental Jetpacks popping up that tap into the modules we’re creating. Some people are working almost entirely in github, while others work in user-repos and still others bridging the two.
It’s been an interesting evolution over the past couple of years, and like a lot of Mozilla, it seems to be happening organically, whether we want it to or not. I think this is Just Fine.
Trying to coerce, codify and enforce however is probably not going to have a great deal of success as there are just too many entrenched stakeholders. Just this weekend, a thread on dev.planning about adding more commit hooks to our HG repository sits in stark contrast to the gist of this post and, in the end, I think the thread and the idea of more commit hooks were quashed due to unforeseen but well-practiced usage patterns.
Quashed? I must have been reading a different thread to you.
And do you really think having a commit log message convention is a big deal?
no, having a convention is a good thing, we have one. Enforcing it via commit hooks is going to be problematic.
Quashed wasn’t really the right word there anyway, since I think the decision was to “delay it past Monday”.
[…] Dietrich Ayala recently raised in a blog post a few ideas about Firefox and how to bring it forward in the current climate of this browser cold war. Dietrich has gone on to evolve his ideas for the modularisation and thus the progression of Firefox by suggesting that we bundle core features as add-ons. […]
[…] my initial dev.planning post, Joe Walker’s post ‘How to Eat an Elephant’, my blog post, dev.planning add-on bundling […]