Trion To Make Sense

You ever played a game in which there was a massive hardware failure, and the only way to make sure everyone’s account was safe was to roll the entire shard back over 24 hours?  This happened to the Tranquility server in 2012.  CCP Games’s response was a full rollback of about 24 hours, but as compensation to the players they immediately gave everyone who logged in a pretty sizeable amount of skill points.  Now given, in EvE Online, “character level” is measured in skill points, and characters can generally be given skill points equal to a specific formula to help balance out what may have been lost in the rollback.  Of course, this also means that ships lost are replaced, and anything you may have bought or found during the time between the rollback start and when it was rolled back to is lost.  That can be both good and bad, depending on how much Bob and RNGesus love you.

Now, let’s assume that a game’s expansion launched during the week.  Let’s pretend it was on a Wednesday that it launched.  The launch was pretty good: no massive issues except a slightly longer-than-anticipated downtime, and a couple of per-usual hotfixes.  No massive server lag spikes, no issues with questing.  It’s pretty good, and you’re thinking it’s doing really well.

Thursday rolls around.  People have taken these two days off to enjoy the new expansion and explore/level, as gamers are wont to do when a new expansion drops.  Once again, plenty of issues that were expected don’t show up, and you’re feeling really great.

Okay, now let’s put a name to this game: Rift.  The expansion that came out is called Nightmare Tide.

At 0400 Pacific Standard Time on Friday, three servers went down.  The official reason for this was some key databases for the Laethys, Wolfsbane, and Faeblight servers had become corrupted, and so the servers were taken down to facilitate the transfer of data to new databases.  No ETA for when it would be fixed, but we didn’t’ find out the reasoning until four hours after the problems began and the servers went down, and we were told they didn’t know what was causing the issue (rather, they were vague on the cause).

At 1200 PST, Laethys came back online, a full eight hours after it went down.  Wolfsbane followed 25 minutes later.  Faeblight remained offline.  During this time, a hotfix was announced.  However, Laethys and Wolfsbane found themselves rolled back after they came back up, but only to an hour at maximum from when they went down (so at most they were rolled back to 0300 PST on Friday, so people lost nine hours of playtime, though only one of it was actually played).  Faeblight was not so well off: the server was rolled back to 0300 Thursday morning.

That’s a 25 hour rollback, for those keeping track.

Now some people will point out that players on Faeblight didn’t actually lose 25 hours’ worth of gameplay.  After all, no one at that point is going to play that much on a single day.  We need time to rest and what-not.  And you’re correct, we did not.  But I want to explain it like this.

Three servers lost a total of nine hours minimum compared to the rest of the shards out there.  They were down, and so that is time at the beginning of an expansion in which those players are not playing, and getting ready to raid.  For progression raiding guilds—even in games outside World of Warcraft—that’s a big thing.  Sure, the individual server may be at an equal footing for server firsts, but can you imagine if <Ensidia>’s server dropped like that?  Could you imagine the shitstorm that would be caused, and Blizzard’s responses to get them back into contention (we’ve seen Blizzard previously coddle their favorite guild, allowing them to bypass mechanics and exploit for world firsts)?  Trion may not have the same issue in that guilds aren’t vying for world firsts for real-world money, but it still causes headaches for them in that it shows ineptitude on the part of the technicians.

Quite literally, the left hand does not know what the right hand is doing.  Technicians do not talk to GMs (Game Masters, the in-game fixers of problems) do not talk to CMs (Community Managers, who handle social media and forums).  I did a video a few months back in which I had reported some players to Trion about harassment that went beyond any type of trolling.  I had gotten in touch with Ocho, a member of the CM team who promised he would talk to the GMs.  A month later, the head GM (Daglar) contacted me literally five minutes after I had finished the initial video to talk to me about the problem.  It took a month to get on that, and that was after I had talked to a CM, whom Daglar said he never heard from.  So someone was lying, or just dropped the ball.  I would have been willing to give them the benefit of the doubt, but I had been a royal pain the ass with the issue: forum thread that I had updated two weeks and four weeks after the initial ticket-and-canned-answer fiasco, so it was a sign, to me, that they didn’t actually care.  But that is a topic for another posting.

The point is, here the GMs hadn’t talked to the technicians, and it took six hours to find out what the fuck had happened.  Just fucking think about that: the players didn’t know why it had happened, and given most of us probably didn’t care about the why as much as we cared about the how, as in “How do you plan to fix this?”  And we probably don’t care about how they plan to prevent this from happening in the future, because those of us on Faeblight feel completely cheated.

So let’s go back to the problem right now.  The server as a whole is a minimum of forty-eight hours behind every other server.  I say “minimum” because I’m really not taking into account the nine hours we had offlined alongside Laethys and Wolfsbane.  That really affects the morale of the players, who have routinely felt that Trion does not care as long as they are making money.  Now given, I’ve had far more positive experiences in this game than negative, but that doesn’t absolve them of blame when they fuck up big time.  A hardware failure may not be completely their fault, but the corruption of a database so severely that it requires you to go back twenty-five hours in order to restart everything screams negligence (and that’s according to some database analysts, who have said this to Daglar et al. on the forums).

Faeblight was called “venerable” by Daglar, which means two different things: highly respected, or old and antiquated.  Faeblight, being one of the few original launch servers remaining, is probably on the old architecture for the game, so it wasn’t ungraded with the other servers.  It also means that it isn’t backed up like the others (which appear to be once every hour).  That says that those of us playing on Faeblight can probably expect to have this happen again, and the same results will happen.

Players have lost credits they spent, which the company claims they will reimburse (and after the fiasco with Guild Wars II, I believe it will be).  They’ve lost currency, and they’ve lost items that were expensive, or powerful (we’re talking relic-level treasure boxes, epic-level BoE gear, etc.).  They’ve lost experience they gained.  But that can all be retrieved in short order, once people (including myself) get over the shock of what appears to be such a monumental failure in thinking.  More importantly, however, the people on Faeblight (and to a lesser extent Wolfsbane and Laethys) lost time.  Time is a precious commodity, and is the most expensive thing each and every one of us has.  Time is money is a common saying, and it’s true: the server is two days behind as a collective, which hurts us for progression raiding and the like.

But the most important thing Faeblight denizens have lost is trust: we’ve lost trust in the capabilities of Trion to successfully keep the server going.  It may not seem like much, but something as simple as properly backing up the database would have done far more to calm us.  As-is, the players feel slighted, and it’s going to take far more than a title, cosmetic pet, mount, or packaging of +160% EXP packages to alleviate that feeling.

As of this writing, Trion’s head GM Daglar has said that for reimbursement they are putting up an experience boosting event on Faeblight.  However, Patrons (that is, paying subscribers) are not seeing this bonus applied to them, thus alienating the people giving a consistent income further.  Try as the free-to-play crowd might to say that we aren’t special, we are a far more consistent income than the people who spend money to buy credits, because that is a one-time thing normally.  You simply can’t count on that every month, where-as we are generally paying for months upon months in a row.  Many of us have also received 750 free credits to spend in the store as we see fit, which is the equivalent to buying $5.00 worth of credits.  It’s not much, but it’s a start.  You may not be able to buy us completely, but you can definitely make us feel better with reimbursement that is commensurate with the level of fuck-up presented.  I’m not saying that we should all get a million credits, but something as simple as some special vials that might last four hours and over +400% EXP gain, as well as everyone on Faeblight affected by this server issue receiving a free 3-day Patron pass to make up for the time we are missing out on because we have to redo what we already did would go a logn way to showing us we care.  And that’s just for starters.

The EULA might say we can’t expect anything from Trion, and in fact Trion doesn’t have to give us anything as compensation, but companies have tried that before, and the negative publicity alone has caused them endless amounts of headaches.  I fully expect Trion will try to repair the damage, but the repairing needs to happen sooner rather than later, and it needs to be seen as something serious.  If Trion does that and restores the faith of the playerbase—because players on other shards are certainly watching this, understanding it could happen to them—they will have a lot more loyal customers, and the media will pick up on it.

I will update as more information becomes available.

Head GM Daglar's Public Response

Edit At the beginning of the post I mentioned that CCP Games rolled the Tranquility server back 24 hours.  This was incorrect.  At the time, the Tranquility server was under a concentrated DDoS attack that kept the server down for a period of 18+ hours, which meant that during this time skills were not training.  As such, CCP reimbursed players a specified formula of skill points in order to apologize for this issue and worked to resolve the problem.  CCP did not roll the server back.