Thursday, March 20, 2008

WYWSITUN Volume 1: Adjacency System Changes

U R probably wondering WTF does WYWSITUN mean.

It means "What You Won't See In The Update Notes".

We actually do a lot of work here at SOE, contrary to popular belief. :) Sometimes the things we work on aren't really worthy of the update notes. Although, I think it would be cool if we started including more detailed info in the update notes. Many people would probably be interested to read some of the technical details.

Until then, I'll just blog about some of these things. So without further ado, welcome to Volume 1 of WYWSITUN.

Something I've worked on a little bit recently is tweaking our adjacency system. What is an adjacency system? Well, its a small part of the server that determines which objects are near you. I guess you could more accurately refer to it as a "sub-system" of the server. Why do we need an Adjacency Sub-System (A.S.S. for short)? Without our ASS, we would have to send you updates for every object in a zone.

When Kunark was in development, we needed a faster and more efficient ASS because the zones were so much larger than previous expansions. One of the things the new system does is cap out when a maximum number of objects are sent to a client. This keeps a player from being bombarded with too many updates in really crowded areas. Instead, you just don't see as many objects as you would normally see. However, with the new system, I've noticed a couple of different issues.

Issue #1: Sometimes doors that are really far away don't render and you see black holes beyond the door. The reason you see a black hole is because the client determines that the door is shut, so it doesn't need to render whats inside the building. But since the door isn't rendering either, it just looks really bad. This is noticeable in Neriak when you stand before the bridge heading to the docks. At the other end of the docks is a building with a door, and sometimes this door doesn't render. I've made a change that increases the priority of doors and other important objects so they should render now from much further away, even if you're in a crowded area like Neriak.

Issue #2: Inanimate objects were causing you to hit the cap of max objects, thus cutting down the number of objects you'd see in an area. Inanimate objects are very lightweight to send to the client because they don't move or do anything to cause constant updates. So basically once we inform the client they are there, there's not a lot of overhead in keeping them relevant to you. An example of where you could see this problem in action is the Village of Somborn in Loping Plains. If you stood outside the church facing the guy that sells Wargs in the corner, most of the time you won't see him or the wargs render if you have your graphics settings somewhat low. This is because the tents behind you have a lot of static objects in them like bottles, papers, etc. Static objects like these will no longer cause you to hit the max cap of adjacent objects.

When GU44 goes live you should now see objects rendering from further away without impacting performance much. There's one other thing to keep in mind. The first step in getting an object rendered is for the server to notify your client that it's there. The second step is for your client to determine if it should be drawn or not. If your graphics settings are set to favor performance over quality, some objects may not be drawn even though the client knows about them. Another change I made is to increase the default rendering distance for some of the lower graphics settings. But if you still see objects not rendering when you think they should be, all you need to do is adjust the value of "cl_lod_scale". Set this value to 1 and you should be fine. You can also find this setting under Options -> Display -> Model Detail -> Level of detail bias.

Bonus: What does Level of detail bias do?

Level of Detail (LOD) describes how an object is rendered. An object rendered at the highest level of detail will show every detail of the model with the highest quality texture. As the LOD goes down, the model gets less detailed. So objects that are far away render at a lower LOD. After all, you won't notice the intricate detail in something when its far away. Once it gets far enough away, it will stop rendering altogether.

Each object determines the LOD it should be rendered at a given distance. So for example, a chair might render at its highest LOD up to 10 meters away. Between 10 and 20 meters it might render at its lowest LOD. And beyond 20 meters it may not render at all.

So back to the original question, what does Level of detail bias do? The distance of an object is multiplied by this value to determine what LOD should be used. So if your cl_lod_scale value is set to 2, a chair at 8 meters away would render as if it were 16 meters away. This would cause the chair to render at the lower LOD instead of the higher LOD. So basically putting this value to "1" tells the client to render objects from the distances in which they were intended to render. A higher number renders them as if they were further away.

So if objects are disappearing sooner than you would like, just decrease this value until you're happy with the results. Tweaking this value in combination with the adjacency changes that go live in GU44 should help in making objects appear when they would normally be gone.

Stay tuned for WYWSITUN Volume 2: Nerfing Rangers
(just kidding)


Barx Atthemoon, Warden of Tunare said...


I'm so glad to hear about that, doors like that bug me to no end /grin.

I'm also one of the folks that would *love* to see more technical stuff in the patch notes, so this post is right up my alley =)

Oh, and when you nerf rangers can you give their stealth to wardens? /grin

((Wardens, being a druid and yet not having even a self-invis is my personal pet peeve, hehe))

Unknown said...

Great post, Greg. I do like seeing more of the technical back-end fixes as well.

Tiffany said...

While funny anyway, the "Nerfing Rangers" joke is probably even more amusing if your readers know that you spent the first two years plus of your EQ2 life (as well as most of your EQ1 life) as a ranger. ;)

I do miss that handsome guy (as opposed to your new alter-ego.)

Anonymous said...

This was fascinating. I love the detailed explanation. I think that it would be good to have information available at this level of detail for all patches, but as an additional document and not as part of the regular patch notes. However, I suspect that it might place an extra burden on the developers, especially as programmers are not always famous for their love of writing documentation.

Chuglet said...

Nice work Greg. As a programmer myself I love hearing about the more technical stuff and knowing that you guys are working to make the game more efficient. I always wondered if not every update made it to the update notes.

I'm not sure if you're aware of the problem, or if it is even a problem, but flying into Teren's Grasp always causes a lot of lag and things like the sokokar post don't always render in a timely fashion (this can be disastrous on a pvp server lol). Are the things you mentioned above helping this or is that a whole different issue (again, if it's even an issue or just my computer)?

Oh, and about rangers. I play a ranger myself and from what I've heard, we're getting some nice updates in the next GU. :)

Greg Spence said...

These changes might very well help the problems in Teren's Grasp. I'm not talking about the lag issue, but about things not rendering in fast enough. Another thing we do is limit the number of "new" objects we send you each server frame. I think the limit is something like 10. So if you were to instantly teleport to a new area that had 40 objects in it, it could take several frames for everything to load. This is to keep from bogging you down too much in the first frame. Since TG is so densely populated, it makes sense that some objects pop in a little later. Hopefully this will help, if not, let me know through my blog and I'll look into it more closely.

Nick McLaren said...

Excellent post Greg! THIS is the kind of stuff I thrive on.. The intricate geeky details that the average person doesn't care about. Looking forward to hearing more later!


Victoria said...

Great post Greg! I love these little bits of information, they really add to my appreciation of the game. I'm a techie at heart, if not always in practice :)

Keep up the great work and I look forward to future WYWSITUNs :)

Anonymous said...

WYWSITUNWTFBBQ? Even though I do not have the level of technical proficiency that most of your other readers probably have, I would just like to chime in that highly detailed posts of changes like this are fantastic.

I am curious if this sort of change would get at least a line in the eventual update notes, and if not, why not get a brief mention.

Also, I'd like to campaign for you to be the person to weed through what things should and shouldn't be mentioned in the notes.

Mikul said...

As a general note on technical notes for changes to game. Would it be worth, asking all the dev/coders to add "#single line effective comments" about changes that are made (then again any good coder does this anyhow ;)) and then just rip this information out for a technical update notes? That gets linked at the end of the normal update notes.

Greg Spence said...

It's basically up to the individual developers that make changes to decide if their change appears in the update notes or not.

I think many times if its a minor bug fix or a tweak like this that would normally require a more lengthy explanation, we may opt to just leave it out. However, I think I included a one-liner in the update notes about some of these changes.

Personally I am going to try to include something about each change I make because I know people want to see even the minor stuff.

However, there are somethings that just don't make sense to post, like modifying something on the back-end to reduce the size of a database record. Ultimately no one will be able to see the outcome of changes like that.

Anonymous said...

Will this affect how mobs don't render until you're right on top of them? This is pretty bad with the drachnids in the first part of KP. But I've recently been visiting PoA again and the mobs in there are doing it really bad now. PoA was never like this before. The Green Dragon up top is the perfect example. You clear the room until he and his three guards pop, and all you see are his three guards. The dragon doesn't render. So you start pulling the guards, and whammo, the dragon is suddenly right on top of you whacking you down. That room has a lot of chairs, bookcases, books, etc. so your changes to the ASS may help there. Was just curious.

Anonymous said...

I can only meow along with the other sheep (I'm so confused now).

Very interestin, keep doing this. Please.

Anonymous said...

These types of posts are always appreciated even by us "armchair developers".

Greg Spence said...

It might help with PoA and the Drachnids. Unfortunately I can't say for sure, with so many variables, those problems could be caused by something else. If you get the chance to try it again after GU44 goes live let me know if you still have the issues and I'll take a look at those areas.

Thundy said...

This post is great and should be on the official forums.

Anonymous said...

Just in general, I would say that any mob that is aggro and you are within 200% of its aggro radius, should be given a very high priority in the ASS. Mobs not rendering until you are *within* their aggro radius is very confusing and very frustrating.

Mikul said...

I would think the reason why some of us, want 99% of everything to be included in some kind of dev/coder notes. Is that we'd like to help you in return, by being able to speed up the liveQA testing. That unfortunately we actually do in the live environment because there isn't time for internal QA to do full scale testing that players would actually get up to. For example if we log in and an encounter has changed and is now a little buggy. If we have something more akin to “Encounter based triggers have now been made to lock to the engaged player group", rather than nothing. This would allow us to quicker feedback issues with encounters like Nexona when things don’t work right. I understand that there is one point of reasoning that would say, don’t post notes about "exploits" or workarounds as it highlights to people what is out there. But I feel that you should really go with the say it’s fixed and trust a section of your player base that actually cares about playing the game as coder intended to highlight those issues to be fixed.

Anonymous said...

Great stuff!!! MOAR!

Anonymous said...

I'm confused. In your arm chair programmer post it sounded like you could get the most gains from moving processes from the CPU to the GPU instead of multi-threading.

But on this post it seems like you are moving processing from the GPU (to render the objects) to the CPU (to decide if the the GPU should render it or not)

So I guess what I am asking is does the process of determining what needs to be rendered in high LoD take place on the GPU also?

If not then I am not sure I understand why you would be looking at one process to move from CPU to GPU and another to move from GPU to CPU at the same time... is one more efficient at a particular style of task?

Anonymous said...

Just adding my /vote for more posts like these. :-)

Anonymous said...

Chris: I think the ASS happens on the server, but I could be wrong.

Anonymous said...

You'd be surprised, Greg, how many people would read even minor bugfix notes, let alone changes to the game mechanics, and combat tweaks, quest adjustments, raid mob changes, etc.

You surely have an issue managment system and a source code repository... the list of fixed/implemented issues per GU would make a good read for those interested in it. A bit like you see on open source (sourceforge f.e.) project websites.

Added a new AOE to the Leviathan? Why, have a line about "new effect from Leviathan" in the notes. Fixed an issue with the "Taking it to the Source" quest? Have a line about how the named froglok inside the ruins has returned or whatever. The customers would be happier. ;)

Ah well. Big companies and transparency are no good friends.

Thundy said...

Also these changes are great but I would personally give my left testicle for the lag introduced in GU43 to be fixed.

Greg Spence said...

It seems we've been having some lag issues related to network hardware, so that might be what you're seeing and not necessarily something related to GU43. Our network guys are working on it with the hardware vender.

Thundy said...

The consensus from everyone is that it started as soon as GU43 went in, and I agree from my experience.

I didn't mean to "armchair program", just saying.

Greg Spence said...

It's possible the network changes also coincided with the GU43 release, but I'm not sure of the exact date. Hopefully we'll know something soon.

Anonymous said...


Thank you for this post. It is nice to get an insight in the work behind the stage. I am looking forward to your next piece.

KC said...

(Awesome post Greg! I tried commenting on this blog post the other day but my computer ate my reply =(

I was curious as to what EQ2's player-character's polygon counts were around?

By the looks of it, I would guess 4500-6000 polygons.. but I can't be too sure without wireframe inspections :)

(just really curious, always wondered how EQ2 stacks up against other Next-gen game titles these days)


Anonymous said...

Nice to read.

I don't know but I also hope this helps with characters not rendering.

Each time we meat at Thuga, there is at least one group member not being rendered. I always need to ask them to jump to get them rendered and then to be able to buff them accordingly. ;)

Anonymous said...

I've always wondered at some of the services that SOE is offering at 2.99 a Month (station super dooper subscription?), however, THIS is exactly the kind of thing I'd pay 2.99 a month for.

Now, we're getting it for free.

Just like free beefcakes. But with less beef. Mmmmh, beef. ;)


Where can I sign up to Greg's Fanclub?

Anonymous said...

In searching for sites related to web hosting and specifically comparison hosting linux plan web, your site came up.