This is an old revision of the document!
My interest in software engineering stemmed from my interest in what my deceased mother (since 2003) would do on the computer when I was a child. I've seen her using what was probably Photoshop (or some alike) as well as programs that I now recognize as being Dark Basic Classic, but those themselves are just vague memories that I can't even really verify. Alongside of a general intrigue with what the technology industry has to offer, I am very certain she would be proud of what I have since become and am still growing into.
I've started my actual programming experience making mods for Tribes 2 which involved writing code in a very peculiar language known as Torque Script which is specific to the Torque Game Engine and is actually used in the more modern successor of that engine known as Torque3D. That has gotten me essentially six years worth of general programming experience and 6 years of expertise its scripting language along with the general workings of the engine. There was a player on there who went by the name of “Naosyth” that had helped get me on the track of modding the game with a short introduction to the datablock system and the basic syntax of the language by using a Metallic Construction mod server as the playground.
I started off writing simple chat commands for Construction .70a, which were all eventually considered, personally, my own mod which I had dubbed “DDDX Mod”. Not much of the mod exists at this point in time, presumably due to poor data backup habits at the time which I now regret nearly eight years later writing this.
Eventually I began work on the Tribes 2: Birth of Legends mod which in reality has almost no correlation to the Birth of Legends project. It was intended to be Tribes 2 game-play infused with various RPG elements in an open sandbox type world. The only relation to the Birth of Legends project now is that it also featured the playable race, the Draakan in a form that is fairly comparable to how they are now. It was a ambitious project at the time, and it stretched the engine to its breaking points.
About a week ago I discovered a ghosting bug in the Tribes 2 engine. What's especially peculiar is that I discovered a networking bug in a listen server of all things. It takes the following form:
The above screenshot was taken in a Meltdown 3 listen game server. After a couple of seconds, everything snaps back into reality:
Before I get into the bug itself, I should explain how Torque Game Engine synchronizes game simulations across the network. To start off with, all interested parties (including the server) run at a tickrate of 32 milliseconds. This means that the game will update the game world in increments of 32 milliseconds each. So, 3 ticks = 3 ticks * 32ms = 96ms. On the server side this is precisely true because that's where the local simulation resides – no latency is accounted for. However, when operating with clients across the internet, network transmission latency has to be accounted for in some way.
The game deals with latency using a client-side simulation to fill in the time it takes for the game server to throw another frame at you which yields a more natural looking simulation according to game rules, but it is prone to desynchronization from what the game server wants you to see at any given point in time. The game has several scripting language exposed values that help control this (with their default values):
Fractional ticks are possible depending on the implementation, so the 0.5 values above represent 16ms instead of the usual 32ms. In my explanation, we'll focus primarily on $Player::maxPredictionTicks which holds the most precedence in the game simulation. In essence, $Player::maxPredictionTicks is used to express how many ticks the game will simulate for between receiving frames from the game server. In this case, it defaults to 30 which is 960ms, almost a full second of simulation. So it takes up to about a humanly visible second for the whole game simulation to freeze if the server crashes or if ping time is so bad that it exceeds this time.
Cool, so what does this have to do with the screenshot above? Or even the title for that matter? An optimized synchronization technique called ghosting. The client-side representation of a given game object is called the “ghost”, which is what's updated from server frames and simulated against using that recent data. However, the game has to deal with dialup connections and slow desktop processors (Pentium 3, Pentium 2). To help alleviate network stress and memory/CPU stress for clients, the game restricts the scope of ghosts to what's relevant to a given player. This means that objects far away enough out of visible distance flat-out does not exist on the client. The game server also performs ghosting based on indoor and outdoor presence – when indoors, many outdoor ghosts vanish from the client's simulation.
Now what do ghosts have to do with the screenshot? The second “vehicle” you see floating off to the side there is a couple of images that are supposed to be mounted to my vehicle. Think of images as holographic representations of an object that you can put on vehicles and players. These images are often used to make new looking objects and weapons without requiring genuine client-side content. Tribes 2 has an inherent bug with keeping these images attached to ghosts in some circumstances.
Not only does Tribes 2 have trouble keeping images mounted correctly, it has a problem with keeping actual objects mounted to the vehicle which are not corrected indefinitely and do not seem correctable aside from modder intervention unmounting and remounting the object. This is noticeable in Meltdown 3 (when the fix isn't applied) where a small cloaked sentry turret can sometimes be found floating around the map which doesn't seem to fully exist – because it doesn't. This turret is a weighting object mounted to the vehicle to help simulate weighting and it isn't supposed to be visible. When you see it detached, its still connected to a vehicle in reality.
This bug is very easily reproduced by placing a vehicle outdoors and going indoors on foot. Once inside (you have to be pretty far indoors), use the script to mount yourself back into the vehicle and move as quickly as possible. If done correctly, the images will lag behind briefly and any mounted objects may “detach” visibly.
As mentioned above, a fairly simple fix which is likely the most commonly used is to occasionally unmount and remount objects occasionally to fix the perpetually unmounted objects after the fact. However, there is no reliable way of timing this as its bound to the server side ghosting systems which means that this is only a partial bandaid, not a full fix.
The scripting engine exposes a function called %object.setScopeAlways, which would seem to be exactly what we need (always ghost the object to everyone) but it apparently simply expedites the problem and actually makes it worse in most cases:
The vehicle is also inaudible, which is really odd. Everything goes back to normal once I mount the vehicle, *and* the chassis weight object is detached.
The scripting engine also exposes a function called %object.scopeToClient(%client), which serves the same function as setScopeAlways except that it targets an individual client. This function seems to work well enough on a single client basis, though I have managed to crash the engine once with it and I have not been able to reproduce it since. Perhaps a WINE bug, hopefully not. And hopefully that function can be looped to scope the object to more than one person to properly resolve both the image bug and the object detachment bug.