Inside ArenaNet: Chromium Embedded Framework in Guild Wars 2
I’m Ben Dunkin, a Senior Engine Programmer on the Guild Wars 2 team. I joined ArenaNet last year to round out the newly reformed engine team. This is the same team that’s been working on the DirectX 11 upgrade. That project is going quite well, but I’m here to talk about something else we’ve been working on: replacing CoherentUI with Chromium Embedded Framework!
What Are We Replacing?
We are replacing CoherentUI, a library that lets us run a web browser inside the game. We use CoherentUI in a bunch of places, including in-game books, the launcher, and the Trading Post. We will be replacing it with Chromium Embedded Framework (henceforth referred to as CEF). CEF is used to create the UIs of many other programs like Steam, the Battle.net launcher, and the Epic Games launcher.
Why Are We Replacing It?
One of the engine team’s areas of focus is modernizing the codebase. We have a lot of code that hasn’t changed in a long time. This has become an issue for us, as we are pushing old systems past their limits. A significant number of issues arise from our third-party libraries (pieces of code that we get from someone else). Some examples are DirectX 9 (for drawing to the screen), Havok (for physics), Umbra (for occlusion culling), and CoherentUI (for some of our UI).
Some libraries are difficult to update, requiring careful planning and cost-benefit analysis before upgrading. We made the decision to update DirectX 9 to DirectX 11 (see our previous blog post here). It has been a huge effort, but we expect the result to be worth it. On the other hand, we decided not to upgrade Havok. It is even more tightly integrated into the game than DirectX 9 was, and there is an unacceptable risk of introducing gameplay-breaking physics bugs.
CoherentUI is a simpler library to update, and we chose it next for a couple of reasons:
CoherentUI doesn’t support emerging security standards. Our third-party payment processing partners have indicated this will be an issue in the future. This is a serious risk impacting our ability to fund future development of the game.
CoherentUI is no longer supported by its vendor. We can’t improve its performance or fix its bugs ourselves. This ultimately manifests as unresolvable Trading Post bugs and performance issues.
CoherentUI is old. It has limited support for modern programming, formatting, and language standards, and it is not supported by modern tools that would help us develop for it faster.
CEF addresses these points and more. Payment processors are happy with it, its performance is much better, and it supports the latest tools and standards. It also gives us many more options to improve the performance further if we need to. We plan to keep it updated on a reasonable cadence so we don’t fall behind on performance improvements and bug fixes.
Some of you may have noticed that we upgraded CoherentUI last year, so why are we making another upgrade? We did the upgrade last year to temporarily address the first point above, giving us time to replace CoherentUI completely. Payment processors were satisfied with the newer version of CoherentUI, but we knew they would eventually say it, too, didn’t meet their standards.
What Will You See After This Change Is Made?
You might notice the Trading Post running a little more smoothly, but not much else. Our goal is to replace the underlying system as transparently as possible. We want to ensure the upgrade is working properly before working on performance or functionality improvements.
There are, however, significant differences between CoherentUI and CEF that you won’t see but which do help us develop faster. For example, here is the Guild Wars 2 website displayed in both.
CoherentUI displaying www.guildwars2.com:
CEF displaying www.guildwars2.com:
Note how CoherentUI doesn’t display the top bar, dropdown menu, news items, or carousel navigation correctly. With CEF we won’t have to work around these kinds of limitations, meaning we can develop the UI faster.
Here is a performance profile of loading the Trading Post.
CoherentUI loading the Trading Post:
CEF loading the Trading Post:
CoherentUI takes 19.241 milliseconds, while CEF takes 6.99 milliseconds. For reference, a game running at 60 frames per second renders a whole frame in 16.66 milliseconds. As you can see, CEF takes 36% of the time CoherentUI does on this task. This means we can spend less time optimizing web-based UI because it is already fast.
Note: There is a difference between UI performance and network performance. Sometimes the Trading Post appears slow because network requests are taking a long time, not because the UI is slow. Upgrading to CEF will not do anything to address slow network requests.
Release Strategy
After our previous attempt to release—and subsequently rolling it back—I saw some people ask a reasonable question: Why can’t we roll this out slowly, or by allowing people to opt in like we did with DirectX 11?
It would be our first choice to roll this change out slowly, but there are some complications that prevent us from doing so. We don’t currently have the tools necessary to release it slowly, and the same complicating factors that make it difficult to release slowly also apply to adding new tools.
Rolling things out slowly requires the game to contact our servers with an account ID so we can tell who to use the new system and who not to. The launcher runs before a player has logged in, which means we have a “chicken and egg” problem. We need an ID to determine which system to use, but we need the system to get an ID. Opting in has a similar problem. Settings are per account, but we don’t have the account yet when the launcher starts. Adding a new configuration system that didn’t use account IDs would require modifying the launcher as well, which means we would have all the same issues in rolling out the new configuration system as we do for the CEF system. We decided to minimize our risk with one change rather than two, although we may add such a configuration system in the future.
With DirectX 11, failing to initialize was a recoverable error. That means the game gets told initialization failed and can respond to that error. We responded by changing the setting back to DirectX 9 so that on the next launch, the game could start again. Unfortunately for us, the issues we see with CEF cause the launcher to crash immediately, which means we don’t have the ability to automatically switch back to CoherentUI. In other words, when things go bad, the only option players have would be to modify command line flags. Command line flags are a last resort to resolve player’s problems, so we treat even a relatively small number of launcher crashes as players being unable to play the game.
Being unable to play the game is one of the worst situations a player can be in, which is why the previous release was rolled back so quickly. It is also the reason we don’t gain anything by making it opt-in. For those who opt in and have issues, our only reasonable response is to roll back the change. That’s the same response as if there was no opt-in, except we get less information because only a small percentage of players would opt in.
A balance must be struck between minimizing player disruption and maximizing the information gained. The errors we are seeing are quite obscure. Only a tiny fraction of players encounters them, and we are unable to reproduce them internally. This means we need a large number of players running the new system just to get a few reports of errors. Since this is a critical system and a necessary upgrade, we have chosen to lean more toward maximizing information. We will limit how long players are potentially disrupted from playing the game by being quick to roll back if we receive enough crash reports.
Current Status
Our release of the CEF on March 14 had enough issues that we rolled it back. We fixed all the issues that we could reproduce and put in mitigations that we expect resolved the rest.
Both the launcher and Trading Post are mission-critical systems and are heavily tested internally before we attempt these releases. Our QA team is amazing and they have tested this upgrade on a variety of hardware and software configurations, but we can’t test all of the thousands of configurations our players use. The number of times the launcher is run during weeks of internal testing is small compared to even one day of our players starting up the game. We released a second time in the April 18 update, and if we see more serious issues, we will promptly roll back the release again until we can address them.
I can’t wait to see what improvements this upgrade unlocks for our players!
—Ben Dunkin
GuildWars2.com Read More
