Firefox Startup Performance Weekly Summary

This week brings a boost in visibility of results, not just for startup, but for all the performance testing we’re doing on all branches and platforms. As I mentioned last week, I was working on an automated method of generating the cross-branch startup results. Luckily Johnath and Chris Atlee had done the hard work when making the performance dashboard. It has a JSON file that contains a roll-up of the previous 7 days performance data, which is updated every 5 minutes with new Talos results. Using that as the datasource, I wrote  a script summarizes the results for each test+branch+OS combination across all the boxes that returned results, as well as stable branch and previous week differences. The final product gives a snapshot view of how each branch compares to the stable branch.

This is useful for a few different reasons. First, we (and the press and our users and our managers and everyone really) are able to know at-a-glance how any branch compares to the stable release branch. An example of another use is that before the JS team does a Tracemonkey merge, they can quickly see if any major performance effects can be expected.

The table shows red or green for any differences that are outside of a 2% threshold, to take test noise into account. This is quite liberal, as I’ve already calculated the numbers to take into account whether the difference is within the standard deviation. Reducing the noise in the tests would be a big help – perhaps some researcher will take up Roc’s challenge. A second point of trust is my math 🙂 I’m not a statistician, so please view source and let me know where I’ve miscalculated.

An example of the full report is here. I’m going to file a bug and work with release-engineering to get it pushed out to the graph server, where the dashboard lives. The data from this week’s startup table is copied below (though sans the color-coding of the real thing. Actually, now WordPress keeps stripping out my styles, so you get an ugly table this week).

Ts

	Firefox3.5	Firefox3.6	Firefox	TraceMonkey
Leopard	median: 1469deviation: 110mean: 1481 from last week: 2%	median: 1115 deviation: 59 mean: 1096 from last week: 3% from 3.5: 26%	median: 1036 deviation: 48 mean: 1006 from last week: 1% from 3.5: 32%	median: 1014 deviation: 38 mean: 1000 from last week: 0% from 3.5: 32%
Linux	median: 625 deviation: 7 mean: 626 from last week: 0%	median: 632 deviation: 7 mean: 633 from last week: 0% from 3.5: -1%	median: 619 deviation: 10 mean: 623 from last week: 1%	median: 628 deviation: 7 mean: 628 from last week: 0%
Vista	median: 538 deviation: 8 mean: 540 from last week: 0%	median: 533 deviation: 13 mean: 537 from last week: 1% from 3.5: 1%	median: 503 deviation: 23 mean: 509 from last week: 0% from 3.5: 6%	median: 511 deviation: 41 mean: 531 from last week: 11% from 3.5: 2%
XP	median: 461 deviation: 6 mean: 461 from last week: 0%	median: 464 deviation: 6 mean: 464 from last week: 1% from 3.5: -1%	median: 448 deviation: 7 mean: 448 from last week: 0% from 3.5: 3%	median: 530 deviation: 38 mean: 501 from last week: 10% from 3.5: -9%

This week’s activity:

• Taras and Joel are still working on bug 524202, tracking down exactly how and when dynamic library code is loaded. They’re past diagnostics, and are now into implementation, coaxing the linker into ordering functions in the optimal sequence. See this comment for a good summary of the issue. Joel put up a very detailed blog post about the work they’ve been doing.
• Ryan Flint posted an update on his startup bug activity this week.
• Have a patch enabling Windows cold-startup testing for Talos for bug 522807, but it’s causing the whole OS to freeze, only recoverable via reboot. Fun! I also added some details and links about how Prefetch/SuperFetch work on Windows to the wiki.
• Rob Strong pushed bug 311965 to mozilla-central, comm-central and 1.9.2 while also ensuring not to break all the toolkit apps that depend on this code. Truly a gentleman of the Mozilla ecosystem.
• Taras put a new patch up for service caching in bug 516085.
• Everything is about ready for re-enabling rebasing on Windows in bug 484799, just needs landing.
• John Dagget posted some test times in bug 519445 for yet further reductions in Mac startup time spent in font system initialization, just needs review.
• Bug 512645, removing the setTimeout 10ms wait in chrome JS, is ready to land. I’ll try to land this weekend if the bug owner doesn’t get around to it first (hint hint).

Projects in a holding pattern:

• Drew has a patch up for bug 506814, getting rid of Change GetPersistentDescriptor/SetPersistentDescriptor on Mac, just needs review from Josh.
• Ben Hsieh has been prototyping a whole Fastload cache replacement in bug 520309.
• JARification: David abandoned moving JS modules into a JAR file, since those files are fastloaded. However, since we want things like post-extension-install restarts to be fast, and those cause fastload cache invalidation, we might want to do things like this anyways. I filed a bug for the same treatment for components. These are lower priority, since they’re not the normal startup case. Follow along with all JAR-ification via the tracker bug.
• Startup Timeline: No updates, still not landed. Add [ft] in the whiteboard of your bug w/ the function names you want timed and David will generate it and update the bug.
• Static Analysis: No progress on bug 506128. David needs to file a bug with the final log of named-yet-uncalled functions.
• Dirty Profile Testing: No progress. Need to list scenarios, file bugs for each, generate Talos config patches and profile data, and then move it into Rel-Eng territory. Also, need to get a separate Tinderbox tree, since it’s going to cause a bazillion new columns.
• Joel Reymont noted in bug 513076 that there are serious drawbacks to getting our libraries in the dyld shared cache on Mac, so has deprioritized that work.
• No updates on Zack’s CSS parser changes in bug 513149.

As usual, more details and links are available on the project wiki, and we’re available to answer questions in #startup on irc.mozilla.org.

UPDATE: Fixed link to full report.

---