Yesterday we branched our code for GU54 which means we're safe to start checking things into our code repository for GU55 (Sentinel's Fate).
I've been working on the new Quest Helper window for awhile and today I was able to finally get this work checked in. I'm pretty happy with the results so far. We've wanted to make this change for a long time but just hadn't found the time to do it. Since we're planning on a big marketing push in February, we're trying to get in as many features as we can that will improve the experience for new players.
One of the biggest things that bothered me when making a new character was the fact that we could only track 1 quest at a time. This just doesn't work in the current environment where it's typical to grab 3-5 quests from a "quest hub" and go do them all at once.
I'm sure this window will be familiar looking if you've played other MMO's that have a quest tracker for multiple quests. There just aren't too many ways to innovate in this area. The goal is to present a list of to-do items in a concise and organized manner. I think this window delivers on that and I hope everyone will like it when it comes out in February.
Thursday, November 05, 2009
Saturday, October 24, 2009
PvP Merge Recap
It's been awhile since I've had a chance to blog and I really need to make a concerted effort to do it more frequently. It's certainly not for lack of things to blog about, but more about making the time to do it. I think if I stop worrying about writing long blog posts and try to write smaller updates, it'll be easier for me to keep up with it.
We've been preparing to merge the PvP server Venekor into Nagafen for several months now. We announced it around the time of Fan Faire and set a date for some time in September. As we approached the date we started experiencing performance problems on Nagafen's database that gave us some real concern about whether or not it would be able to handle the additional load from Venekor. Nagafen's performance has always been a little sluggish due to the higher population but a few weeks before the merge, the problem got much worse.
We had several meetings with our DBA's to discuss methods for improving the performance. Most of the options were band-aids or had no guarantee that they would solve the problem. Much to our surprise the DBA's were able to offer a complete upgrade to the database hardware and software. This was something we had been wanting to do for awhile but hadn't had the opportunity due to conflicting projects for the DBA Team. While this was certainly the most desirable option, there just wasn't enough time to do it before the merge that was only 2 weeks away. Although we knew that the decision would be met with some disappointment, we decided to delay the merge by another 4 weeks to give us time to upgrade the database hardware.
We scheduled the migration of the Nagafen data to the new database a week before the merge was to happen. The migration day came, the servers were brought down and the data was moved to the new database. Everything appeared to be perfect even after we brought up the servers on the new hardware. Unfortunately, there was a small subset of characters who were not able to log in. After some research we found that the migration software had a problem moving certain large BLOB fields. At this point we had to rollback to the old database and come up with a new plan. After more meetings we decide to try the migration again but resort to a different method of moving the data that was slower but more reliable. Several days later we tried again and everything worked great.
The DB migration happened on a Tuesday. Ideally we would have waited until the next Tuesday to perform the merge, but this would mean delaying it another whole week and we just didn't want to do that. So we decided to kick off the merge process Wednesday night at midnight. We had a 28 hour maintenance window to complete the merge.
On the evening of the merge we kicked off the script to export the data from Venekor. The script was running much slower than during our tests and looked as if it would take 26 hours just to export the data which was only half of the process. At this point we didn't have a lot of options so we let it go. By 7am Thursday morning the script had been running for 7 hours when it decided to crash with a segmentation fault. You can imagine how we felt knowing that the script was already taking twice as long as it should have been and on top of that we just lost 7 hours. We decided to run the script from another machine to rule out hardware problems and started it up again. That morning we discussed worst-case scenarios and pretty much decided that if the script crashed again we would have no other choice but to abort and troubleshoot the problem. However, not only did the script continue to run, it actually started running faster. The export script completed after about 15 hours. The next step of the process was to bring down Nagafen for a final backup and then start the import of the data.
The import went a good deal faster since it was running against the newer database hardware. About 12 hours later the import script finished and all of the Venekor data was now in Nagafen's database. There were still many things to do and fortunately the timing worked out so that the import finished during the day and not in the middle of the night. This allowed us to tackle the remaining tasks from the office during normal work hours. We continued to execute scripts that copied shared banks, guild halls and more. These scripts had to resolve name conflicts and keep up with ID changes so we could notify the chat and mail servers of changing player data. Chat servers were brought down to update their data which unfortunately had to impact all game servers. (Sorry about that)
As soon as all of the scripts were done we were ready to bring up Nagafen and have QA run through the standard sanity checks to make sure everything looks OK. Our smoketest takes several people an hour or two to complete and covers most of the gaming functionality. Unfortunately its just not possible to test every detail and every code-path. The smoketest passed and the server was unlocked.
For the most part, everything looks good. We haven't seen any problems with character data other than some issues with missing shared bank coin. We also had an issue with guild hall supply depot contents not transferring over correctly. These can be corrected with some additional time and overall I'd say the merge was a success. The server has only been up for about 5-6 hours at the time of this post, so I hope we don't discover any other issues. *Keeps fingers crossed*
Some of us will be busy fixing up these final issues over the next couple of days, but overall I'm very happy with how well things turned out. Everyone who had a part in the merge and DB migration did an excellent job and I'm proud to be working with a group of such intelligent and hard-working people.
We understand that the extended downtime was an inconvenience to those of you who played on Venekor and Nagafen. We apologize for that but hope this process will bring you much more enjoyment in the long run.
Thanks!
We've been preparing to merge the PvP server Venekor into Nagafen for several months now. We announced it around the time of Fan Faire and set a date for some time in September. As we approached the date we started experiencing performance problems on Nagafen's database that gave us some real concern about whether or not it would be able to handle the additional load from Venekor. Nagafen's performance has always been a little sluggish due to the higher population but a few weeks before the merge, the problem got much worse.
We had several meetings with our DBA's to discuss methods for improving the performance. Most of the options were band-aids or had no guarantee that they would solve the problem. Much to our surprise the DBA's were able to offer a complete upgrade to the database hardware and software. This was something we had been wanting to do for awhile but hadn't had the opportunity due to conflicting projects for the DBA Team. While this was certainly the most desirable option, there just wasn't enough time to do it before the merge that was only 2 weeks away. Although we knew that the decision would be met with some disappointment, we decided to delay the merge by another 4 weeks to give us time to upgrade the database hardware.
We scheduled the migration of the Nagafen data to the new database a week before the merge was to happen. The migration day came, the servers were brought down and the data was moved to the new database. Everything appeared to be perfect even after we brought up the servers on the new hardware. Unfortunately, there was a small subset of characters who were not able to log in. After some research we found that the migration software had a problem moving certain large BLOB fields. At this point we had to rollback to the old database and come up with a new plan. After more meetings we decide to try the migration again but resort to a different method of moving the data that was slower but more reliable. Several days later we tried again and everything worked great.
The DB migration happened on a Tuesday. Ideally we would have waited until the next Tuesday to perform the merge, but this would mean delaying it another whole week and we just didn't want to do that. So we decided to kick off the merge process Wednesday night at midnight. We had a 28 hour maintenance window to complete the merge.
On the evening of the merge we kicked off the script to export the data from Venekor. The script was running much slower than during our tests and looked as if it would take 26 hours just to export the data which was only half of the process. At this point we didn't have a lot of options so we let it go. By 7am Thursday morning the script had been running for 7 hours when it decided to crash with a segmentation fault. You can imagine how we felt knowing that the script was already taking twice as long as it should have been and on top of that we just lost 7 hours. We decided to run the script from another machine to rule out hardware problems and started it up again. That morning we discussed worst-case scenarios and pretty much decided that if the script crashed again we would have no other choice but to abort and troubleshoot the problem. However, not only did the script continue to run, it actually started running faster. The export script completed after about 15 hours. The next step of the process was to bring down Nagafen for a final backup and then start the import of the data.
The import went a good deal faster since it was running against the newer database hardware. About 12 hours later the import script finished and all of the Venekor data was now in Nagafen's database. There were still many things to do and fortunately the timing worked out so that the import finished during the day and not in the middle of the night. This allowed us to tackle the remaining tasks from the office during normal work hours. We continued to execute scripts that copied shared banks, guild halls and more. These scripts had to resolve name conflicts and keep up with ID changes so we could notify the chat and mail servers of changing player data. Chat servers were brought down to update their data which unfortunately had to impact all game servers. (Sorry about that)
As soon as all of the scripts were done we were ready to bring up Nagafen and have QA run through the standard sanity checks to make sure everything looks OK. Our smoketest takes several people an hour or two to complete and covers most of the gaming functionality. Unfortunately its just not possible to test every detail and every code-path. The smoketest passed and the server was unlocked.
For the most part, everything looks good. We haven't seen any problems with character data other than some issues with missing shared bank coin. We also had an issue with guild hall supply depot contents not transferring over correctly. These can be corrected with some additional time and overall I'd say the merge was a success. The server has only been up for about 5-6 hours at the time of this post, so I hope we don't discover any other issues. *Keeps fingers crossed*
Some of us will be busy fixing up these final issues over the next couple of days, but overall I'm very happy with how well things turned out. Everyone who had a part in the merge and DB migration did an excellent job and I'm proud to be working with a group of such intelligent and hard-working people.
We understand that the extended downtime was an inconvenience to those of you who played on Venekor and Nagafen. We apologize for that but hope this process will bring you much more enjoyment in the long run.
Thanks!
Saturday, June 13, 2009
Fan Faire iPhone Web App
I've been working on an iPhone Web App for our upcoming SOE Fan Faire in Las Vegas. The goal of the app is to give iPhone users an easy-to-access reference for all information they might need while attending Fan Faire.
The app will have the following information:
- Full schedule of all events and panels browsable by game.
- A map of Bally's convention center
- Twitter feed from our @soefanfaire twitter account for up-to-the minute news
- Extensive list of restaurants around Bally's and Paris hotels.
- Travel information for getting around Vegas
I threw together the basics of the web app within a couple of evenings of work. Now I'm working on the polish features that make it really useful.
Being able to filter the schedule by game is a cool feature and will help attendees find panels without scrolling through the entire list of events.
Originally the list of restaurants was going to be very basic but my wife encouraged me to go the extra step and provide more info about the restaurants. So each restaurant now has a detail page that shows their location, hours, type of cuisine, cost and description. Clicking the phone number will initiate dialing on your iPhone. There's even a button to take you to the restaurants website if you're interested in browsing their menu. (Thanks Tiffany for typing all that data in!)
The news section fetches the RSS feed from twitter and displays the last several entries in a list. If the twitter entry contains a URL, clicking the row will open the browser to the web page.
There are still some things I need to finish before I can call it done but its coming together nicely. If you'd like to see it while its in development, you can check it out at http://www.gregsplace.com/fanfaire. Keep in mind it will only work in the Safari browser because it makes heavy use of their iPhone extensions.
Let me know what you think!
Tuesday, April 14, 2009
EverQuest II Welcome Screen
Today we launched one of the features I've been working on over the past couple of weeks, the EQII Welcome screen. Already this feature was met with a little friction on the official forums and I can certainly understand why some players may not be interested in seeing this new window. Something that might not be apparent at first is that you can dismiss it by just hitting your escape key.
However, I wanted to point out some positive aspects of the window. I've been an EQII player for a lot longer than I've been an SOE employee and I think the welcome screen is a great addition to the game. Here are some reasons why I like it.
For awhile now, the server message of the day (MOTD) has started to lose a lot of its impact. The MOTD often contains information that some people aren't interested in reading. This leads to people ignoring it altogether and possibly missing something important. One of the biggest reasons we created the welcome screen was to give each area of the game that often has its own announcements, a specific place for those messages. By segregating the MOTD's like this, it should be easier for a player to glance at the one they like and ignore the rest. If you're in a guild, your guild's MOTD can also get lost in all the text that is written to your chat window when you log in.
I don't play much Legends of Norrath and haven't purchased anything in the Marketplace, but I still want to see what my guild is up to. I also want to be made aware of any other server-related messages like unscheduled downtime when I might be playing on the weekend. With the welcome screen, I can glance at those specific MOTDs and then hit escape when I'm done looking at them.
I also really like having the friends list on the welcome screen. For some reason I never really used the Community window in the past, possibly because there's nothing else on this window that interests me. However, when I log in, one of the first things I want to do is see who is online that I might be able to play with. So having my friends list visible on the welcome screen is really nice. Kindricke made a great suggestion on the forums about showing only your online friends. So I've made a change this morning so that the welcome screen will share the same setting as the Community window. So, to see only you online friends, just open the Community window (Y-key) and check the box to ignore offline friends. This setting will save between your play sessions.
Whether you like the Marketplace or not, sometimes there is an item that might be of interest to you. The recently created gender and name change potions were big hits. I'm sure many people who have never purchased a Marketplace item took advantage of this feature that people have been asking for. Similar to our character transfer service, we've talked about adding a name-change service for a long time. Now with the addition of the Marketplace, we were able to bring that feature into the game and not require you to use an external website. Surely everyone can see this as a big win. The marketplace item is much more reliable and less prone to failure than a completely separate process. So back to my original point. Even if you don't like the Marketplace, there might occasionally be an item that you would consider purchasing. Having the list of new items on the welcome screen makes it easy for you to see when we add a new item without opening the entire Marketplace window and scrolling through all the items that are already there.
The other main section of the welcome screen is the community box. Our community department publishes on average 1-3 articles a day on the website. For many players, these articles provide valuable information about upcoming features, events and more. The welcome screen grabs the RSS feed for these articles and displays the titles right on the page. So now you don't have to open a browser to keep up with these articles. If you see an article that looks interesting, you can click the link and read it right from within the game.
So to wrap up this rather long blog post, the welcome screen is more about convenience and getting the information that's important to you quickly and easily. Hopefully like me, you'll find this feature helpful and a nice addition to the game.
--- Optional Reading ---
As a side note, our original design idea for the welcome screen included an interface similar to iGoogle where you could move the widgets around and even decide which widgets were visible and which were hidden. We eventually opted for taking a simpler approach so the feature wouldn't take as long to implement. At some point in the future we might still look into a more customizable interface. For now, it'll be interesting to see what our modding community comes up with.
However, I wanted to point out some positive aspects of the window. I've been an EQII player for a lot longer than I've been an SOE employee and I think the welcome screen is a great addition to the game. Here are some reasons why I like it.
For awhile now, the server message of the day (MOTD) has started to lose a lot of its impact. The MOTD often contains information that some people aren't interested in reading. This leads to people ignoring it altogether and possibly missing something important. One of the biggest reasons we created the welcome screen was to give each area of the game that often has its own announcements, a specific place for those messages. By segregating the MOTD's like this, it should be easier for a player to glance at the one they like and ignore the rest. If you're in a guild, your guild's MOTD can also get lost in all the text that is written to your chat window when you log in.
I don't play much Legends of Norrath and haven't purchased anything in the Marketplace, but I still want to see what my guild is up to. I also want to be made aware of any other server-related messages like unscheduled downtime when I might be playing on the weekend. With the welcome screen, I can glance at those specific MOTDs and then hit escape when I'm done looking at them.
I also really like having the friends list on the welcome screen. For some reason I never really used the Community window in the past, possibly because there's nothing else on this window that interests me. However, when I log in, one of the first things I want to do is see who is online that I might be able to play with. So having my friends list visible on the welcome screen is really nice. Kindricke made a great suggestion on the forums about showing only your online friends. So I've made a change this morning so that the welcome screen will share the same setting as the Community window. So, to see only you online friends, just open the Community window (Y-key) and check the box to ignore offline friends. This setting will save between your play sessions.
Whether you like the Marketplace or not, sometimes there is an item that might be of interest to you. The recently created gender and name change potions were big hits. I'm sure many people who have never purchased a Marketplace item took advantage of this feature that people have been asking for. Similar to our character transfer service, we've talked about adding a name-change service for a long time. Now with the addition of the Marketplace, we were able to bring that feature into the game and not require you to use an external website. Surely everyone can see this as a big win. The marketplace item is much more reliable and less prone to failure than a completely separate process. So back to my original point. Even if you don't like the Marketplace, there might occasionally be an item that you would consider purchasing. Having the list of new items on the welcome screen makes it easy for you to see when we add a new item without opening the entire Marketplace window and scrolling through all the items that are already there.
The other main section of the welcome screen is the community box. Our community department publishes on average 1-3 articles a day on the website. For many players, these articles provide valuable information about upcoming features, events and more. The welcome screen grabs the RSS feed for these articles and displays the titles right on the page. So now you don't have to open a browser to keep up with these articles. If you see an article that looks interesting, you can click the link and read it right from within the game.
So to wrap up this rather long blog post, the welcome screen is more about convenience and getting the information that's important to you quickly and easily. Hopefully like me, you'll find this feature helpful and a nice addition to the game.
--- Optional Reading ---
As a side note, our original design idea for the welcome screen included an interface similar to iGoogle where you could move the widgets around and even decide which widgets were visible and which were hidden. We eventually opted for taking a simpler approach so the feature wouldn't take as long to implement. At some point in the future we might still look into a more customizable interface. For now, it'll be interesting to see what our modding community comes up with.
Tuesday, March 03, 2009
EQII Shadow Maps
If you log into the Test server you can see an early implementation of shadow maps for EQII. I'm really excited about this feature going in and its something we've been wanting for awhile. Ryan Favale has done an excellent job adding this new shadow system to EQII. Now you'll have a choice between shadow volumes (CPU) and shadow maps (GPU). There will certainly be situations depending on your computer hardware and the zone you're in where one might be faster than the other and vise versa. But this option will at least let you choose the shadow system that works best overall for your system.
There are some inherent differences between the two shadow technologies and if you're interested to read more about how they work, there is some decent information on Wikipedia. We are using something similar to the cascaded shadow map system described below.
Shadow Maps (GPU)
http://en.wikipedia.org/wiki/Shadow_mapping
Shadow Volumes (CPU)
http://en.wikipedia.org/wiki/Shadow_volume
If you get a chance to try these out on the Test Server, please give us your feedback in the following thread. Ryan is still tweaking for performance and quality as well as fixing bugs as he's made aware of them. So the more feedback we get, the better!
http://forums.station.sony.com/eq2/posts/list.m?topic_id=444895
There are some inherent differences between the two shadow technologies and if you're interested to read more about how they work, there is some decent information on Wikipedia. We are using something similar to the cascaded shadow map system described below.
Shadow Maps (GPU)
http://en.wikipedia.org/wiki/Shadow_mapping
Shadow Volumes (CPU)
http://en.wikipedia.org/wiki/Shadow_volume
If you get a chance to try these out on the Test Server, please give us your feedback in the following thread. Ryan is still tweaking for performance and quality as well as fixing bugs as he's made aware of them. So the more feedback we get, the better!
http://forums.station.sony.com/eq2/posts/list.m?topic_id=444895
Friday, February 06, 2009
EQII Update
Well, its been quite awhile since I've blogged about work. It seems that since November I've been working only on big projects that I can't talk about until they are released. Then, once they are out there, they aren't near as exciting to talk about. :)
Lately the team has really been focusing on performance optimizations to try to combat lag and improve overall performance both on the client and server. Josh (Autenil) has been making some great improvements to the server which should hopefully go out in the next week or so. Ryan Favale, our graphics guru, has been working on some really amazing things for the client but unfortunately I can't talk specifics. I was given the task to try to improve how our servers provision hardware for zones. Without going into too much detail, our zones are dynamically created when a player needs them and our server software determines which piece of hardware is best suited for the zone based on memory usage and CPU activity. This is working as it should, but I was able to make some optimizations that should improve this process a bit. Hopefully these changes will go out today or Tuesday and if all goes well we will see some improvement to the lag spikes.
Lately I've also been able to focus some time on improving the user-interface and provide more capabilities to the modding community. The group window was modified to include the pet health bar. The raid window was modified to include an advanced mode that will show detrimental effects for the entire raid. Hopefully these changes will help the healers out there. :)
One of the big changes that I made to the user interface will go out with GU52 and if all goes well, players won't even notice the difference. Currently, all of your UI Settings are stored in INI files in a binary format. Binary is the most efficient when it comes to space, but it can be a real pain to work with. I've recently modified the client to store the settings in XML format. The advantage from a code-standpoint is that its now much easier for us to add and modify the values that we store without breaking older versions of the file. Previously every change had to have a version number associated with it and the code had to be prepared for loading a file of any previous version number so we could properly deserialize the binary information. Now with the XML format, its much simpler. The big advantage for players is they can now easily see what information is stored there and manually tweak this file if they wish to. You can easily do things like remove pesky chat channels or modify the channel number that is associated with them. This change also paves the way for giving modders the ability to store custom data in the settings file for their UI mods.
In addition to these changes I've been quite busy with random bug fixes and improvements to the game. I'm also working on another "big" project that isn't due for release until GU53, so again, I'm being a tease and can't really talk about it. :) EQII has some of the richest lore in any MMO out there and one the most creative groups of players I've seen. I think it will be a very fun addition to the game.
Lately the team has really been focusing on performance optimizations to try to combat lag and improve overall performance both on the client and server. Josh (Autenil) has been making some great improvements to the server which should hopefully go out in the next week or so. Ryan Favale, our graphics guru, has been working on some really amazing things for the client but unfortunately I can't talk specifics. I was given the task to try to improve how our servers provision hardware for zones. Without going into too much detail, our zones are dynamically created when a player needs them and our server software determines which piece of hardware is best suited for the zone based on memory usage and CPU activity. This is working as it should, but I was able to make some optimizations that should improve this process a bit. Hopefully these changes will go out today or Tuesday and if all goes well we will see some improvement to the lag spikes.
Lately I've also been able to focus some time on improving the user-interface and provide more capabilities to the modding community. The group window was modified to include the pet health bar. The raid window was modified to include an advanced mode that will show detrimental effects for the entire raid. Hopefully these changes will help the healers out there. :)
One of the big changes that I made to the user interface will go out with GU52 and if all goes well, players won't even notice the difference. Currently, all of your UI Settings are stored in INI files in a binary format. Binary is the most efficient when it comes to space, but it can be a real pain to work with. I've recently modified the client to store the settings in XML format. The advantage from a code-standpoint is that its now much easier for us to add and modify the values that we store without breaking older versions of the file. Previously every change had to have a version number associated with it and the code had to be prepared for loading a file of any previous version number so we could properly deserialize the binary information. Now with the XML format, its much simpler. The big advantage for players is they can now easily see what information is stored there and manually tweak this file if they wish to. You can easily do things like remove pesky chat channels or modify the channel number that is associated with them. This change also paves the way for giving modders the ability to store custom data in the settings file for their UI mods.
In addition to these changes I've been quite busy with random bug fixes and improvements to the game. I'm also working on another "big" project that isn't due for release until GU53, so again, I'm being a tease and can't really talk about it. :) EQII has some of the richest lore in any MMO out there and one the most creative groups of players I've seen. I think it will be a very fun addition to the game.
Subscribe to:
Posts (Atom)