It’s no secret that there are performance issues on Badlands, I know it’s frustrating as a player and it’s especially frustrating for us as developers. We have pride in our product and want you all to be able to enjoy it, so it’s been especially stressful on us trying to address the issues on Badlands and not seeing the results we want. I know ultimately what you all want is a solution, and we promise to continue working on it until it’s resolved. I want to give you all some insight into what we’ve done so far to address the issues, some of the success we’ve had and what we’ve got coming up to slay the demons plaguing WAR, and in particular Badlands.
We started recognizing and receiving issue reports from Badlands shortly after the reclamation of our European customers. There was such a breadth of reports it wasn’t immediately clear what was to blame so we begun to dig in. Quickly a few major culprits arose out of the reports: items getting lost in mail and from live events, some new client crashes we hadn’t seen before and the most serious and disruptive of all, getting kicked to character or server select when moving between areas in the game.
There were a lot of leads to follow, some issues crossed paths with other issues, some issues once analyzed were found to be a combination of two or three smaller issues, and some were dead ends. With each maintenance patch you’re seeing the results of our investigations. Honestly the maintenance patches of the past few months have been some of our most extensive maintenance patches that we’ve ever done for WAR in an effort to resolve the issues you all have been suffering as soon as possible. Right now I want to concentrate on what’s giving Badlands players and WAR developers heartburn lately.
It’s all about the data. We’ve now been operating WAR for over two years and during that time many characters and guilds have been created. We hold onto all that data and at this point we have in the MILLIONS of characters and thousands of guilds on Badlands alone. That’s a lot of data and it’s very important for us to keep it safe. It’s beyond just data, it’s the collective experiences of all of our players, their adventures, their heart pounding victories and their heroic defeats. It’s undoubtedly our most precious asset and we treat it as such.
Every day we perform backups on this precious data, usually in the early morning when server populations are at their lowest. On a server like Badlands this normally takes a few hours to run. However all this changed with the European reclamation, these days Badlands activity is almost always high at all times of the day. This is awesome in and of itself; however this means player activity is competing a lot more for disk time with our backups than it has in the past. Combined with how much data we have to backup on Badlands and the amount of action going on this is causing our backups to run for extended periods of time, it especially gets hairy when our Backups run into primetime which has been the case the past few weekends. Sometimes this is leading to backups that take longer than a day to finish.
What does all of this have to do with you getting kicked to character or server select when zoning? Zoning behind the scenes is a process which passes the player from one physical server to another physical server, as part of that process we save the character to disk before moving the player between servers and then load the character up on the other end when they arrive. When we are stuck in this state with backups and player activity competing for disk time queues begin to form and build up faster than the system can clear them. If your save doesn’t get processed within a certain amount of time the game believes something is wrong and removes the character from the game world as a precaution.
This has put us in a dilemma, we could simply turn off backups or modify them in some way and many of these issues would go away or be lessened significantly. However in doing that we put characters, guilds, campaign data and more at risk of being rolled back for multiple days or worse, being lost all together. Turning off backups also has a negative impact on the quality of service our Customer Service Department can deliver to our users which is very important to us. Turning off backups has not been an option for us, your characters are too important to us and to you to risk them in such a fashion.
So what have we been doing to try and fix this issue? Optimizing what data we save, how we save it and when we save it, removing unused code and data, adding diagnostics and logging, trying new hardware, trying new hardware configurations, limiting trial access to Badlands and that’s just the highlights on a long list. With each change that hasn’t solved the issue our next attempt at solving it has had an increased amount of risk and increase potential of return -- making changes that impact data and how we manage it, especially character data, is normally something we are extremely cautious about and try if at all possible to reserve for new versions instead of maintenance patches in order to get plenty of testing on it. For the most part all of our attempts at solving the issue have made improvements, but it hasn’t been enough.
We have more changes to address the issues that have been in testing and additional changes in production that we really believe will solve this issue once and for all. Some have had enough testing that we hope to provide them to you within the next few maintenance patches. As well we already have plans in motion in the case that they don’t solve the issues being experienced on Badlands.
I pulled some data to help put some perspective on just how much effort we’ve put behind resolving issues on the engineering side alone. We’ve made approximately 70 code changes directly addressing the various issues and forms of lag players have been reporting. To put that into perspective version 1.3.6 involved 156 code changes, and this is just the WAR engineering staff! Our server administrators, quality assurance, customer service and community teams have put countless hours into helping identify and address the issues on Badlands as well as everyone else working on WAR.
I apologize; we apologize for the fact that the issues haven’t been resolved at this point. It’s been our top priority for weeks at this point and will continue to be so until it is resolved.
Along this road there have been a number of successes as well two of which I want to call out in particular:
Client stability – Prior to version 1.3.6 going live we were okay with the stability of the client, we’d love for it to never crash and will not be happy until that goal is met. However we made a breakthrough a few weeks ago which has reduced by approximately half the number of client crashes we are seeing.
Server stability – Flash crowd region crashes are largely a thing of the past. We’ve seen crowds more than double what we could support just a few versions ago battling it out in Open RvR. This has been through an extensive process of optimizing the server code as well as client and server messaging. Today we’re seeing larger crowds than ever battle it out, all the while the server is hardly breaking a sweat. These larger crowds bring with them new issues unique to having so many players in an area and we acknowledge that even though the server doesn’t crash the play experience can suffer. During version 1.3.6 maintenance patches we’ve continued to improve the play experience in these situations and there’s more to come.
I know these successes pale in comparison to the issues Badlands players in particular have been experiencing lately, but it’s also a sign of what’s right around the corner once we get this sorted out – some of the best play experiences and biggest RvR battles you’ve ever seen.
We will continue to work on this issue and any other issue that severely impacts the WAR play experience and do so while working to be more open regarding communication going forward. As always, thank you for your support and we’ll see you on the battlefield.
John Cox
Development Manager
BioWare Mythic
» Edited on: 2010-10-01 16:12:00