Firefox Startup Performance – Feb 5, 2010

Nothing major to report. Some of the big projects are in the final stretch, which is great to see.

• Static build: Joel’s in cleanup phase, making sure the mobile tinderboxes build with the changes. Core patch is waiting on first review from Ted.
• Ben’s fastload cache replacement is waiting on first-review from Ben Smedberg.
• Zach has more data on the effect of the CSS parser optimizations he’s been working on.
• Asaf put up an experimental patch for making the bookmarks toolbar all JS, no XBL.

Related work I did this week:

• Started updating the Performance Snapshot. Fixing a couple of bugs and making the percentages relative to the 3.6 branch.
• Spent some time poking at the graph server. It’s got serious performance problems, and is lacking a few features that’d make it immensely useful, instead of only somewhat useful. It just needs a little love, that’s all.
• Landed bug 506471, moving FUEL out of the startup path.

Firefox Startup Performance – January 29

No big changes from last week, so I’m not going to knock you over the head with the big table, but here are some notes on the progress of some of the projects and areas of research:

• Joel has patch in Ted’s review queue for the static build changes. He’s also working on getting Windows builds un-broken, with help from Ben Smedberg and Brad Lassey.
• Rob Campbell enabled PGO on the Mac, and found no detectable difference in performance. I built PGO on Linux, and found the same. It could be that there’s no improvement to be had. It could be that our profile-generation approach doesn’t target the codepaths that our performance tests measure. Not sure yet!
• I talked with Rob Stong about the fast-startup component, and he brought up a bunch of problems with enabling it in multi-user environments. I’m pretty convinced that there’s no way we can ship with it enabled. And I’m still convinced that it should be exposed as an option to users. I did get it actually working on Windows 7, and as expected, first startup post boot was 60% faster. The scary part is that it took 40% of startup to do the rest…
• I researched code locality options for Linux, and haven’t yet found a way to specify a function order like you’re able to on Darwin. Joel’s been spending some time looking into this on Mac, but gprof is broken on Snow Leopard – instrumented builds just hang. Boo.
• Joel talked with some Apple developers on the Darwin list and got some more information about compressing with HFS+. They thought it was pretty hacky, recommended improving code locality instead (See previous comment about gprof being broken. *sigh*). Compressing locally does break our code-signed builds, so might not be feasible anyways. Next is to look into compressing on the release side.

Firefox Startup Performance – January 22

A good week: Firefox 3.6 was released, and I’ve gotten a lot of feedback from friends and colleagues that startup has improved, and that it’s snappier than ever. Yesterday I blogged a roundup of the performance improvements in Firefox 3.6.

Top Startup Bugs

Estimated win	Bug #	Summary	Owner	Status	Notes
>10%	512584	Super fast paths for Components.classes and Components.interfaces	Taras Glek	in progress	Taras is traveling, says he’ll close it out when he returns.
>10%	525013	Investigate a more static build configuration of Firefox	Joel Reymont	in progress	More great data from Zack and Joel. See PDF charts of the different build configurations tested on different hard-drive speeds for Mac and for Linux.
up to 25%	514083	Per-file HFS+ compression on Mac OSX 10.6	Joel Reymont	in progress	Conclusion is that we need to make this happen via the installer, as well as the updater, since modifying files results in decompression. Need to find an owner for this still.
TBD	520309	Startup cache: replacement for fastload cache	Ben Hsieh	in progress	Considering moving back to the simpler caching approach.
TBD	503483	Turn on –enable-faststart for Firefox by default	Dietrich	needs testing	Started talking with Rob Strong about making the changes to the NSIS installer to make this happen.
TBD	513149	Speed up CSS parsing by using a machine generated lexer	Zack Weinberg	Zack’s blocked on other work	No update.Taras says about 6% of startup spent parsing CSS.

Other activity this week:

• Of the bugs I filed as a result of the JS profiling data (ranked log, stack view), we closed out bug 539991 and bug 539870, and have owners and patches on bug 539869 and bug 539997.
• I figured out the leak on bug 506471, to move FUEL out of the startup path, and will have review soon.
• Taras posted a comparison of the startup time on Linux with a new profile for Chromium and Minefield. Good data… for Linux users, but the lesson here is that we urgently need a canonical dirty profile to test with. I’m going to get with the Test Pilot folks to see about gathering the data to do this.
• Also, I found another post from Taras that I missed while I was out for the holidays, detailing the problems with measuring cold startup on Windows.

Projects in a holding pattern:

• More investigation into Hunspell changes in bug 468799, possibly incorporating some changes Chrome made.
• Ryan Flint has a WIP patch to minify JS on bug 524858 that significantly reduces the size of shipped JavaScript files.
• JARification: David abandoned moving JS modules into a JAR file, since those files are fastloaded. However, since we want things like post-extension-install restarts to be fast, and those cause fastload cache invalidation, we might want to do things like this anyways. I filed a bug for the same treatment for components. These are lower priority, since they’re not the normal startup case. Follow along with all JAR-ification via the tracker bug.
• Startup Timeline: No updates, still not landed. Add [ft] in the whiteboard of your bug w/ the function names you want timed and David will generate it and update the bug.
• Static Analysis: No progress on bug 506128. David needs to file a bug with the final log of named-yet-uncalled functions.
• Dirty Profile Testing: No progress. We have a list of test scenarios, still need to file bugs for each, generate Talos config patches and profile data, and then move it into Rel-Eng territory. Also, need to get a separate Tinderbox tree, since it’s going to cause a bazillion new columns.
• Joel Reymont noted in bug 513076 that there are serious drawbacks to getting our libraries in the dyld shared cache on Mac, so has deprioritized that work.

As usual, more details and links are available on the project wiki, and we’re available to answer questions in #startup on irc.mozilla.org.

Firefox 3.6 Performance: Startup and Snappiness Improvements

Firefox 3.6 was released today! For me, the most important changes in this release are the performance and stability improvements. On Mac especially, there are some big improvements in startup time, yielding up to 30% faster startup generally. And on both Mac and Windows, we fixed some pathologically bad startup scenarios.

The list of bugs fixed that specifically affect startup time is here, however a few should be called out individually:

• Ted Mielczarek and Drew Willcoxon combined all .xpt files into a single one, reducing the number of files read from the disk at startup.
• Rob Strong removed the need for services to load at startup for timer registration, reducing the number of files read from disk at startup, as well as reducing the amount of JavaScript executed. There are still services that need to be ported to this new scheme, so expect more improvements here.
• Vlad Vukicevic found that on Mac it could take up to a whole second to initialize Growl, if present. His fix initializes Growl on demand.
• Joel Reymont found that up to 10% of startup time on Mac was in the initialization of fonts. John Daggett and Jonathan Kew did some great work to reduce the problem in Firefox 3.6. They subsequently modified font loading on Mac to occur on-demand, so the next release will be even faster.
• Joel also found that up to 12% of startup on Mac was spent in some APIs that were only needed for debugging. The problem was quickly fixed by Masayuki Nakano.
• Taras Glek and Alfred Kayser spent a bunch of time improving the performance of how JARs are loaded and read from. The infrastructure improvements made are paying off as we move more groups of small files into these archives, reducing the number of individual file operations that occur at startup time.
• On Windows, it was found that a large portion of startup time was spent in the security libraries, building up entropy by trawling through IE’s cache folder and the Windows temp directory. Some Windows users reported upwards of 25k files in their temp directories, so you can imagine the delays that this behavior was causing for some users. The behavior is now fixed.

The full list of bugs fixed in Firefox 3.6 with the “perf” keyword is here. These range from improvements to web page loading times, general UI responsiveness, and improvements to specific UI actions such as searching History and Bookmarks:

• One of the biggest improvements is that Shawn Wilsher ported all of the Awesomebar code to JavaScript and moved the queries that build up the results to a background thread. This resulted in a massive improvement in UI responsiveness when typing in the location bar.
• Another infrastructural improvement that will affect general responsiveness is Marco Bonardo’s work to clean up the bookmarks and history database periodically.
• Per-tab-network-prioritization is the long way of saying that Firefox will prioritize the network traffic of pages you are currently interacting with. Paul O’Shannessy explains the feature here, and was able to finish it up in time to make 3.6.
• Finally, there are a TON of performance improvements to the JavaScript engine, which benefit just about every area of the browser, from startup time to page loading to extensions. Dave Mandelin put up a detailed post on Hacks.Mozilla.org last week that explains it all.

And all of this is just a taste of what’s to come: Over 60% of the bugs fixed as part of our focus on startup performance have landed on trunk, but didn’t make it in time for the 3.6 release. You can follow along on the wiki page for the startup project, and here on my blog, where I post status updates every Friday.

Firefox Startup Performance – January 15

Happy 2010 everyone!

Top Startup Bugs

Estimated win	Bug #	Summary	Owner	Status	Notes
>10%	512584	Super fast paths for Components.classes and Components.interfaces	Taras Glek	in progress	Taras is traveling, says he’ll close it out when he returns.
>10%	525013	Investigate a more static build configuration of Firefox	Joel Reymont	in progress	Lots of progress! Joel has a patch for Mac and Linux working on the tryserver, and numbers to confirm a partial static build gets ~8% improvement on Mac, with no compat problems. Zach added a bunch of number crunching to validate Joel’s numbers. On the Windows front, Taras has a patch in progress now.
up to 25%	514083	Per-file HFS+ compression on Mac OSX 10.6	Joel Reymont	in progress	Conclusion is that we need to make this happen via the installer, as well as the updater, since modifying files results in decompression. Need to find an owner for this still.
TBD	520309	Startup cache: replacement for fastload cache	Ben Hsieh	in progress	Ben is looking into a perf regression that arose after some major changes.
TBD	503483	Turn on –enable-faststart for Firefox by default	Dietrich	needs testing	No update, need to test still. Maybe just push on Windows here.
TBD	513149	Speed up CSS parsing by using a machine generated lexer	Zack Weinberg	Zack’s blocked on other work	No update.Taras says about 6% of startup spent parsing CSS.

Other activity this week:

• I updated Mark’s patch in bug 507012 to trunk and ran it on Windows. Here is the ranked log and the stack view. From perusing this data, I filed bug 539997, bug 539991, bug 539869 and bug 539870.
• I’ve been working on bug 506471, to move FUEL out of the startup path, and also making a bunch of minor improvements and cleanup in that code.
• While I was gone, Rob Strong put up a great post about what JS components are in the startup path and how much time they take up.

Projects in a holding pattern:

• More investigation into Hunspell changes in bug 468799, possibly incorporating some changes Chrome made.
• Ryan Flint has a WIP patch to minify JS on bug 524858 that significantly reduces the size of shipped JavaScript files.
• JARification: David abandoned moving JS modules into a JAR file, since those files are fastloaded. However, since we want things like post-extension-install restarts to be fast, and those cause fastload cache invalidation, we might want to do things like this anyways. I filed a bug for the same treatment for components. These are lower priority, since they’re not the normal startup case. Follow along with all JAR-ification via the tracker bug.
• Startup Timeline: No updates, still not landed. Add [ft] in the whiteboard of your bug w/ the function names you want timed and David will generate it and update the bug.
• Static Analysis: No progress on bug 506128. David needs to file a bug with the final log of named-yet-uncalled functions.
• Dirty Profile Testing: No progress. We have a list of test scenarios, still need to file bugs for each, generate Talos config patches and profile data, and then move it into Rel-Eng territory. Also, need to get a separate Tinderbox tree, since it’s going to cause a bazillion new columns.
• Joel Reymont noted in bug 513076 that there are serious drawbacks to getting our libraries in the dyld shared cache on Mac, so has deprioritized that work.

As usual, more details and links are available on the project wiki, and we’re available to answer questions in #startup on irc.mozilla.org.

Firefox Startup Performance – Dec 18

Not much change from my late post on Monday, but a few updates below. I’ll be out of the office as of this Monday, and back on 1/11/2010. If you’re starving for startup info during that time, hit up Taras, Joel and Planet Firefox for the goods. The current numbers for startup and all other tests are available on the Performance Snapshot Page.

Top Startup Bugs

Estimated win	Bug #	Summary	Owner	Status	Notes
>10%	512584	Super fast paths for Components.classes and Components.interfaces	Taras Glek	in progress	Taras says mostly there.
>10%	525013	Investigate a more static build configuration of Firefox	Joel Reymont	Mac and Linux in progress	Joel’s working on getting load of XPCOM components working now. Next step is completing runs of the performance tests on the tryserver.
up to 25%	514083	Per-file HFS+ compression on Mac OSX 10.6	Joel Reymont	in progress	Snow Leopard only. More testing needed to see if it’s feasible to compress at install-time, or if we’ll have to ship SL builds.
TBD	520309	Startup cache: replacement for fastload cache	Ben Hsieh	in progress	Brendan suggested some significant changes, in-progress.
TBD	503483	Turn on –enable-faststart for Firefox by default	Dietrich	needs testing	No update this week. Loads Firefox core libraries at boot time. Need to test on all OSes, publish the numbers, and get discussion going.
TBD	513149	Speed up CSS parsing by using a machine generated lexer	Zack Weinberg	Zack’s blocked on other work	Taras says about 6% of startup spent parsing CSS.

Other activity this week:

• More investigation into Hunspell changes in bug 468799, possibly incorporating some changes Chrome made.

Projects in a holding pattern:

• Ryan Flint has a WIP patch to minify JS on bug 524858 that significantly reduces the size of shipped JavaScript files.
• JARification: David abandoned moving JS modules into a JAR file, since those files are fastloaded. However, since we want things like post-extension-install restarts to be fast, and those cause fastload cache invalidation, we might want to do things like this anyways. I filed a bug for the same treatment for components. These are lower priority, since they’re not the normal startup case. Follow along with all JAR-ification via the tracker bug.
• Startup Timeline: No updates, still not landed. Add [ft] in the whiteboard of your bug w/ the function names you want timed and David will generate it and update the bug.
• Static Analysis: No progress on bug 506128. David needs to file a bug with the final log of named-yet-uncalled functions.
• Dirty Profile Testing: No progress. We have a list of test scenarios, still need to file bugs for each, generate Talos config patches and profile data, and then move it into Rel-Eng territory. Also, need to get a separate Tinderbox tree, since it’s going to cause a bazillion new columns.
• Joel Reymont noted in bug 513076 that there are serious drawbacks to getting our libraries in the dyld shared cache on Mac, so has deprioritized that work.

As usual, more details and links are available on the project wiki, and we’re available to answer questions in #startup on irc.mozilla.org.

Firefox Startup Performance – Dec 14

I’m going to be changing the format of these posts a bit, to put focus on the most important issues currently blocking a super-fast startup of Firefox. Hopefully it’ll bring attention on the longest-running, hardest-to-fix, but highest-impact bugs. Right now I’m defining high-impact as changes that bring a 10% or greater improvement. Some of these bugs we don’t yet have solid data on how much of an impact, but expect it to be filled in soon as we narrow the focus onto these. As always, the current numbers for startup and all other tests are available on the Performance Snapshot Page.

Top Startup Bugs

Estimated win	Bug #	Summary	Owner	Status	Notes
>10%	512584	Super fast paths for Components.classes and Components.interfaces	Taras Glek	in progress	Got consultation from mrbkap, needs new patch.
>10%	525013	Investigate a more static build configuration of Firefox	Joel Reymont	Mac and Linux in progress	Only measured on Mac so far, need Linux numbers and someone to figure out the Windows story.
up to 25%	514083	Per-file HFS+ compression on Mac OSX 10.6	Joel Reymont	in progress	Snow Leopard only.
TBD	520309	Startup cache: replacement for fastload cache	Ben Hsieh	review	More efficient than current fastload, and key to enabling fastloading of XBL, CSS, manifests and various other data. Need to push to Places branch to figure out base perf difference from current fastload.
TBD	503483	Turn on –enable-faststart for Firefox by default	Dietrich	needs testing	Loads Firefox core libraries at boot time. Need to test on all OSes, publish the numbers, and get discussion going.
TBD	513149	Speed up CSS parsing by using a machine generated lexer	Zack Weinberg	Zack’s blocked on other work	Taras says it’s high, but no figures, so need better data here.

Other activity this week:

• Jonathan Kew landed bug 519445 on trunk, which improves font loading on Mac. The patch didn’t dent the Ts graph, as the test loads a basically empty page. This patch has the largest effect when pages load various fonts, and have a large font collection. For instance, here John Dagget comments that on a system with 2000+ fonts, the loading time of the font system went from 6 seconds down to 0.33 seconds.
• Rob Strong blogged about time spent executing JS in the front-end, and put up a table of the worst offenders.
• The measurements Rob made above are from an instrumentation effort that’s happening in bug 507012, that will give us great visibility into where time is spent in JS. You can do similar profiling using DTrace, but that’s Mac only, and requires separate tools and scripts to be installed. This instrumentation will eventually be available across platforms in release builds of Firefox. It’ll default to off, but be togglable via an environmental variable or some other method. Providing tools like this makes it *easy* for anyone working on Firefox to diagnose performance problems.

Projects in a holding pattern:

• Ryan Flint has a WIP patch to minify JS on bug 524858 that significantly reduces the size of shipped JavaScript files.
• JARification: David abandoned moving JS modules into a JAR file, since those files are fastloaded. However, since we want things like post-extension-install restarts to be fast, and those cause fastload cache invalidation, we might want to do things like this anyways. I filed a bug for the same treatment for components. These are lower priority, since they’re not the normal startup case. Follow along with all JAR-ification via the tracker bug.
• Startup Timeline: No updates, still not landed. Add [ft] in the whiteboard of your bug w/ the function names you want timed and David will generate it and update the bug.
• Static Analysis: No progress on bug 506128. David needs to file a bug with the final log of named-yet-uncalled functions.
• Dirty Profile Testing: No progress. We have a list of test scenarios, still need to file bugs for each, generate Talos config patches and profile data, and then move it into Rel-Eng territory. Also, need to get a separate Tinderbox tree, since it’s going to cause a bazillion new columns.
• Joel Reymont noted in bug 513076 that there are serious drawbacks to getting our libraries in the dyld shared cache on Mac, so has deprioritized that work.

As usual, more details and links are available on the project wiki, and we’re available to answer questions in #startup on irc.mozilla.org.

Firefox Startup Performance Weekly Summary

Current numbers are available on the Performance Snapshot page for all tests.

Summary current startup performance, relative to Firefox 3.5:

• Warm startup: For Mac, 34% better on 3.6 and 32% better on 3.7. Flat on Windows for 3.6, and since Windows 3.6 was ~5% improved over 3.5 last post, so there appears to be a regression here. However, the graph data is not showing a 5% regression, so something might be incorrect in the snapshot code.
• Cold startup: For Mac, I’m discarding past numbers since we were not measuring it correctly (bug 532965). For Windows, still not measuring yet. For Linux, the ~9% regression discrepancy from the last post is now gone,  putting 3.6 at 3% improved over 3.5 and trunk at a 3% regression. However, there are regular multi-run spikes in the graph, so I’m not sure we’re getting the best data out of this measurement scheme either.

This week’s activity:

• Rob Strong landed bug 532552, which removes the blocklist service from the startup path. This log of WinCE startup, generated from Taras’ patch here, made it apparent that the service was in the startup path and taking a significant amount of time. Poke around that log, there’s more goods to be had.
• Taras landed a bunch of JAR related bugs on branch: bug 510844, bug 511754, bug 525755, bug 523065, bug 521227.
• Rob Strong put up a detailed post about his startup improvements to the update service, the effect on mobile, and pointed out some other front-end services that could benefit from the same treatment.
• Joel’s static Firefox is working (bug 525013), and is showing ~10% improvement in startup time on Mac. He’s moved on to improving code locality (bug 531406). However, time-ordering of symbols doesn’t seem to improve the sequentiality of the page-ins by much, so maybe more improvements there. More data on the bug.
• Ben is making progress on the fastload replacement in bug 520309.
• No updates on Windows cold-startup testing for Talos on bug 522807. I need to test on Vista, and turn off Pre/Superfetch.
• Taras has a new patch up for service caching (bug 516085).
• Rebasing on Windows in bug 484799 messed up mobile builds, was backed out.
• Jonathan Kew’s patch on bug 519445 for further reductions in Mac startup time spent in font system initialization, is ready to land when the tree re-opens.

Projects in a holding pattern:

• Super-fast-path-ing of Components.* (bug 512584) is on hold, blocking on getting help from JS team or someone else who knows that code.
• Ryan Flint has a WIP patch to minify JS on bug 524858 that significantly reduces the size of shipped JavaScript files.
• JARification: David abandoned moving JS modules into a JAR file, since those files are fastloaded. However, since we want things like post-extension-install restarts to be fast, and those cause fastload cache invalidation, we might want to do things like this anyways. I filed a bug for the same treatment for components. These are lower priority, since they’re not the normal startup case. Follow along with all JAR-ification via the tracker bug.
• Startup Timeline: No updates, still not landed. Add [ft] in the whiteboard of your bug w/ the function names you want timed and David will generate it and update the bug.
• Static Analysis: No progress on bug 506128. David needs to file a bug with the final log of named-yet-uncalled functions.
• Dirty Profile Testing: No progress. Need to list scenarios, file bugs for each, generate Talos config patches and profile data, and then move it into Rel-Eng territory. Also, need to get a separate Tinderbox tree, since it’s going to cause a bazillion new columns.
• Joel Reymont noted in bug 513076 that there are serious drawbacks to getting our libraries in the dyld shared cache on Mac, so has deprioritized that work.
• No updates on Zack’s CSS parser changes in bug 513149.

As usual, more details and links are available on the project wiki, and we’re available to answer questions in #startup on irc.mozilla.org.

Firefox Startup Performance Weekly Summary

Current numbers are available on the Performance Snapshot page.

Summary, relative to Firefox 3.5:

• Warm startup: For Mac, 36% better on 3.6 and 35% better on 3.7. For Windows, 5% and 5%. Flat on Linux. Also, Warm startup for Mac on 3.6 is a whopping 13% better than last week, due to the landing of bug 517804.
• Cold startup:  For Mac, 20% better on both 3.6 and 3.7. For Windows, not measuring yet. For Linux, we’re seeing a regression of ~9% across branch and trunk in the snapshot but not on the graphs, so I need to figure out where the discrepancy is.

This week’s activity:

• Dirty-cold-Ts went live this week, thanks to Alice and Lukas. Example: cold startup with a large places.sqlite on Mac.
• Joel is making progress on making a super-static Firefox in bug 525013.
• Ben is making progress on the fastload replacement in bug 520309.
• No updates on Windows cold-startup testing for Talos on bug 522807. I need to test on Vista, and turn off Pre/Superfetch.
• Taras has patches up for service caching (bug 516085) and super-fast-path-ing of Components.* (bug 512584), however the latter he’s hit a wall, passing on to Blake or someone else who knows that code.
• Ted landed rebasing on Windows in bug 484799.
• Jonathan Kew has a new patch in bug 519445 for further reductions in Mac startup time spent in font system initialization, just about there…
• Ryan Flint put a patch to minify JS on bug 524858, not working yet, but significantly reduced the size of shipped JavaScript files.

Projects in a holding pattern:

• JARification: David abandoned moving JS modules into a JAR file, since those files are fastloaded. However, since we want things like post-extension-install restarts to be fast, and those cause fastload cache invalidation, we might want to do things like this anyways. I filed a bug for the same treatment for components. These are lower priority, since they’re not the normal startup case. Follow along with all JAR-ification via the tracker bug.
• Startup Timeline: No updates, still not landed. Add [ft] in the whiteboard of your bug w/ the function names you want timed and David will generate it and update the bug.
• Static Analysis: No progress on bug 506128. David needs to file a bug with the final log of named-yet-uncalled functions.
• Dirty Profile Testing: No progress. Need to list scenarios, file bugs for each, generate Talos config patches and profile data, and then move it into Rel-Eng territory. Also, need to get a separate Tinderbox tree, since it’s going to cause a bazillion new columns.
• Joel Reymont noted in bug 513076 that there are serious drawbacks to getting our libraries in the dyld shared cache on Mac, so has deprioritized that work.
• No updates on Zack’s CSS parser changes in bug 513149.

As usual, more details and links are available on the project wiki, and we’re available to answer questions in #startup on irc.mozilla.org.

Easing Orange: Jetpack for Correlating Tinderbox Test Failures with Bugs

This is a Jetpack feature for finding out if a bug is already filed for a test failure on Tinderbox. When viewing log files for failed test runs, the Jetpack will add a link next to the test failure summary at the top of the log, that looks like “(maybe bug XXXXXX?)”. This allows sheriff’s and other awesome community members to easily mark known-oranges, and update the bug with log’s URL.

Install the Jetpack feature.

Notes:

• So far it only pulls file names out of error text, so won’t match leaks, crashes or oranges without a filename in the summary.
• Only searches the bug summary, and only searches known orange bugs (ie: has “[orange]” in the bug whiteboard).
• Scrapes bugzilla.mozilla.org, since the new REST api is very very slow, so might break with bugzilla upgrades.
• It doesn’t run until the log has completed loading, which sometimes can be a while. Load that shit in a background tab and be patient yo.

Let me know if this helps you out, or any bugs or improvements you’d like to see!