Inside ArenaNet: Chromium Embedded Framework in Guild Wars 2

Inside ArenaNet: Chromium Embedded Framework in Guild Wars 2

I’m Ben Dunkin, a Senior Engine Programmer on the Guild Wars 2 team. I joined ArenaNet last year to round out the newly reformed engine team. This is the same team that’s been working on the DirectX 11 upgrade. That project is going quite well, but I’m here to talk about something else we’ve been working on: replacing CoherentUI with Chromium Embedded Framework!

What Are We Replacing?

We are replacing CoherentUI, a library that lets us run a web browser inside the game. We use CoherentUI in a bunch of places, including in-game books, the launcher, and the Trading Post. We will be replacing it with Chromium Embedded Framework (henceforth referred to as CEF). CEF is used to create the UIs of many other programs like Steam, the Battle.net launcher, and the Epic Games launcher.

Why Are We Replacing It?

One of the engine team’s areas of focus is modernizing the codebase. We have a lot of code that hasn’t changed in a long time. This has become an issue for us, as we are pushing old systems past their limits. A significant number of issues arise from our third-party libraries (pieces of code that we get from someone else). Some examples are DirectX 9 (for drawing to the screen), Havok (for physics), Umbra (for occlusion culling), and CoherentUI (for some of our UI).

Some libraries are difficult to update, requiring careful planning and cost-benefit analysis before upgrading. We made the decision to update DirectX 9 to DirectX 11 (see our previous blog post here). It has been a huge effort, but we expect the result to be worth it. On the other hand, we decided not to upgrade Havok. It is even more tightly integrated into the game than DirectX 9 was, and there is an unacceptable risk of introducing gameplay-breaking physics bugs.

CoherentUI is a simpler library to update, and we chose it next for a couple of reasons:

CoherentUI doesn’t support emerging security standards. Our third-party payment processing partners have indicated this will be an issue in the future. This is a serious risk impacting our ability to fund future development of the game.
CoherentUI is no longer supported by its vendor. We can’t improve its performance or fix its bugs ourselves. This ultimately manifests as unresolvable Trading Post bugs and performance issues.
CoherentUI is old. It has limited support for modern programming, formatting, and language standards, and it is not supported by modern tools that would help us develop for it faster.

CEF addresses these points and more. Payment processors are happy with it, its performance is much better, and it supports the latest tools and standards. It also gives us many more options to improve the performance further if we need to. We plan to keep it updated on a reasonable cadence so we don’t fall behind on performance improvements and bug fixes.

Some of you may have noticed that we upgraded CoherentUI last year, so why are we making another upgrade? We did the upgrade last year to temporarily address the first point above, giving us time to replace CoherentUI completely. Payment processors were satisfied with the newer version of CoherentUI, but we knew they would eventually say it, too, didn’t meet their standards.

What Will You See After This Change Is Made?

You might notice the Trading Post running a little more smoothly, but not much else. Our goal is to replace the underlying system as transparently as possible. We want to ensure the upgrade is working properly before working on performance or functionality improvements.

There are, however, significant differences between CoherentUI and CEF that you won’t see but which do help us develop faster. For example, here is the Guild Wars 2 website displayed in both.

CoherentUI displaying www.guildwars2.com:

CEF displaying www.guildwars2.com:

Note how CoherentUI doesn’t display the top bar, dropdown menu, news items, or carousel navigation correctly. With CEF we won’t have to work around these kinds of limitations, meaning we can develop the UI faster.

Here is a performance profile of loading the Trading Post.

CoherentUI loading the Trading Post:

CEF loading the Trading Post:

CoherentUI takes 19.241 milliseconds, while CEF takes 6.99 milliseconds. For reference, a game running at 60 frames per second renders a whole frame in 16.66 milliseconds. As you can see, CEF takes 36% of the time CoherentUI does on this task. This means we can spend less time optimizing web-based UI because it is already fast.

Note: There is a difference between UI performance and network performance. Sometimes the Trading Post appears slow because network requests are taking a long time, not because the UI is slow. Upgrading to CEF will not do anything to address slow network requests.

Release Strategy

After our previous attempt to release—and subsequently rolling it back—I saw some people ask a reasonable question: Why can’t we roll this out slowly, or by allowing people to opt in like we did with DirectX 11?

It would be our first choice to roll this change out slowly, but there are some complications that prevent us from doing so. We don’t currently have the tools necessary to release it slowly, and the same complicating factors that make it difficult to release slowly also apply to adding new tools.

Rolling things out slowly requires the game to contact our servers with an account ID so we can tell who to use the new system and who not to. The launcher runs before a player has logged in, which means we have a “chicken and egg” problem. We need an ID to determine which system to use, but we need the system to get an ID. Opting in has a similar problem. Settings are per account, but we don’t have the account yet when the launcher starts. Adding a new configuration system that didn’t use account IDs would require modifying the launcher as well, which means we would have all the same issues in rolling out the new configuration system as we do for the CEF system. We decided to minimize our risk with one change rather than two, although we may add such a configuration system in the future.

With DirectX 11, failing to initialize was a recoverable error. That means the game gets told initialization failed and can respond to that error. We responded by changing the setting back to DirectX 9 so that on the next launch, the game could start again. Unfortunately for us, the issues we see with CEF cause the launcher to crash immediately, which means we don’t have the ability to automatically switch back to CoherentUI. In other words, when things go bad, the only option players have would be to modify command line flags. Command line flags are a last resort to resolve player’s problems, so we treat even a relatively small number of launcher crashes as players being unable to play the game.

Being unable to play the game is one of the worst situations a player can be in, which is why the previous release was rolled back so quickly. It is also the reason we don’t gain anything by making it opt-in. For those who opt in and have issues, our only reasonable response is to roll back the change. That’s the same response as if there was no opt-in, except we get less information because only a small percentage of players would opt in.

A balance must be struck between minimizing player disruption and maximizing the information gained. The errors we are seeing are quite obscure. Only a tiny fraction of players encounters them, and we are unable to reproduce them internally. This means we need a large number of players running the new system just to get a few reports of errors. Since this is a critical system and a necessary upgrade, we have chosen to lean more toward maximizing information. We will limit how long players are potentially disrupted from playing the game by being quick to roll back if we receive enough crash reports.

Current Status

Our release of the CEF on March 14 had enough issues that we rolled it back. We fixed all the issues that we could reproduce and put in mitigations that we expect resolved the rest.

Both the launcher and Trading Post are mission-critical systems and are heavily tested internally before we attempt these releases. Our QA team is amazing and they have tested this upgrade on a variety of hardware and software configurations, but we can’t test all of the thousands of configurations our players use. The number of times the launcher is run during weeks of internal testing is small compared to even one day of our players starting up the game. We released a second time in the April 18 update, and if we see more serious issues, we will promptly roll back the release again until we can address them.

I can’t wait to see what improvements this upgrade unlocks for our players!

—Ben Dunkin

GuildWars2.com Read More

Welcome the Spirit of Spring with the Quiet Woods Chair Skin

Welcome the Spirit of Spring with the Quiet Woods Chair Skin

Quiet Woods Chair Skin

Sit beneath the shade of a gentle tree and spend your afternoon relaxing! A remnant of Melandru’s blessings will sooth your nap.

What’s in Stock

We’re refreshing our seasonal selection of unique weapon skins this Friday with a 20% discount. Some of the amazing discounted items include the Dreamwalker Scythe Staff Skin, Dreamwalker Dagger Skin, Dreamwalker Longbow Skin, Dreamwalker Sword Skin, Ritualist Staff Skin, Nightspeaker Greatsword Skin, Tea Time Focus Skin, Ravensoul Dagger Skin, Belinda’s Greatsword Skin, Kurzick Dual Axe Skin, and much more!

Returning This Week

Bag Slot Expansions are 20% off until May 8. Don’t miss out!

Available Now in the Gem Store!

Log into Guild Wars 2 and press ‘O’ to access the Black Lion Trading Company for these great offers and more!

GuildWars2.com Read More

Primastic Light Guides You with the Infinite Herald of Aurene Gathering Tools

Primastic Light Guides You with the Infinite Herald of Aurene Gathering Tools

Infinite Herald of Aurene Gathering Tools

Channel the Prismatic Elder Dragon power within your cosmic soul to gather the materials you need. This pack comes with a Herald of Aurene Mining Tool, Herald of Aurene Harvesting Tool, and Herald of Aurene Logging Tool for unlimited gathering. The tools are equipped with a Glyph of Bounty, Glyph of the Forester, and Glyph of the Prospector. Shine away!

Gem Store Packaging Update

We are updating selected gliders and novelties to match the current Gem Store combo format. They will be sold exclusively as combo items that contain both the glider and the matching novelty. If you already bought one of these items individually, you can claim the matching glider or novelty for free from Black Lion Exchange Specialists. This change will be permanent, and you can exchange your glider and novelty items at any time!

Exchangable Gliders and Novelties

Electromagnetic Ascender
Electromagnetic-Descender Glider
Inquest Overseer Chair
Inquest Overseer Glider
Magic Carpet
Magic Carpet Glider
Riding Broom
Riding Broom Glider
Bloodstone Ascender
Shattered Bloodstone Glider

Black Lion Chest Update: Eternal Knowledge Chest

Inside each chest, you’re guaranteed to find a redeemable Black Lion Statuette, an upgradeable Mini Blue Skimmer Pup, and two common items. You also have a chance to find something rarer in the fifth slot, including special items, glyphs, and skins from the Steel Lotus Weapon Collection and Lorekeeper Weapon Collection.

Exclusive Item: Scrivener’s Knowledge Shoulders Skin

Prophecies are written as the world hangs heavy on your shoulders. Embrace destiny.

What’s in Stock

We’re refreshing our seasonal selection of back items this Friday with a 20% discount. Some of the discounted back items include the Celestial Ministry Cape, Jora’s Cape, Forest Archer Pack, Butterfly Wings Backpack, Plush Quaggan Backpack Cover, and more. Look as stylish as you want to be!

Returning This Week

Character Slot Expansions are 20% off until May 1. Expand your character collection!

Available Now in the Gem Store!

Log into Guild Wars 2 and press ‘O’ to access the Black Lion Trading Company for these great offers and more!

GuildWars2.com Read More

Set Off to Radical Adventures with the Dynamics Racing Skiff Skin

Set Off to Radical Adventures with the Dynamics Racing Skiff Skin

Dynamics Racing Skiff Skin

Surf the ocean with the latest asura skiff tech. You’ll be the most radical captain to ever cast a line.

What’s in Stock

We’re refreshing our seasonal selection of infinite gathering tools and fishing rod skins this Friday with a 20% discount. Some of the discounted items include the Shrine Guardian Fishing Rod, Thunderstrike Harvesting Tool, Earthshaker Mining Tool, Firestorm Logging Tool, Infinite Unbound Magic Gathering Tools, and more. Make sure to get your favorite looks!

Returning This Week

The Copper-Fed Salvage-o-Matic, Silver-Fed Salvage-o-Matic, Runecrafter’s Salvage-o-Matic, and Equipment Template Expansions are 20% off and Build Template Expansions are 40% off until April 24.

Available Now in the Gem Store!

Log into Guild Wars 2 and press ‘O’ to access the Black Lion Trading Company for these great offers and more!

GuildWars2.com Read More

Defy Authority with the Rebel Long and Short Boots Package

Defy Authority with the Rebel Long and Short Boots Package

Rebel Long and Short Boots Package

Bat around the streets with a rebellious new look. Even the Ministry of Security will know to give you a wide berth.

What’s in Stock

We’re refreshing our seasonal selection of armor sets this Friday with a 20% discount. Some of the discounted sets include the Incarnate Light Armor Skin, Rampart Heavy Armor Skin, Strider’s Medium Armor Skin, and Aetherblade armor skins. Make sure to get your favorite looks!

Returning This Week

Bank Tab Expansions, permanent lounge passes, and the Infinite Volatile Magic Gathering Tools are 20% off until April 17.

Available Now in the Gem Store!

Log into Guild Wars 2 and press ‘O’ to access the Black Lion Trading Company for these great offers and more!

GuildWars2.com Read More