PR Nightmares (and they're not Rifts)

For the past few weeks I’ve been silent, and what I have posted has been about Trion Worlds’s inaction on the part of the customer support, community management, and in general public relations teams to try to keep the customerbase of Rift updated on the issue that occurred three weeks ago.  Now it may seem like a long time to hold a grudge, but truth be told it’s no longer a grudge about the rollback, but rather the lack of Trion to actually take the complaints seriously, and to treat the paying players (really, all the players) as something of a money faucet.  Generally speaking, the anger at this point that is still simmering amongst the Faeblight denizens is that the company does not care, and looking back on many of the posts and what-not, I have to echo this sentiment.

Before I go into the actual posting, I’d like to take a moment to congratulate Turbine for their quick response today when the Dungeons and Dragons Online authentication server issue happened.  In under an hour the server was restarted, and the players were kept updated as to the cause, the status, and the final resolution in a timely manner.  Not once did anyone feel like there was an issue that they were being ignored or kept in the dark, and Turbine’s Twitter, Facebook, and forums kept the players in the loop.  Dungeons and Dragons Online doesn’t look as pretty as Rift, it doesn’t play as well as Rift, and in general feels like a lower-quality game than Rift; however, Turbine has continuously shown a good customer support team when it comes to PR (their in-game support is a bit lacking).

Now, I’m going to go into the root cause of the issue, but I’m going to touch on some secondary issues as well.  Let’s get those secondary issues out of the way first.

Rollbacks happen

Yes, they do.  Unfortunately, sometimes servers shit the bed and hardware problems occur.  There is very little that companies can do to prevent this: it comes with age and the like.  Server maintenance and downtime generally focuses on the software side, and that can help mitigate the problems that can appear when hardware problems happen.  I don’t think anyone is ever going to complain about scheduled downtime, because it’s a necessity in MMOs (CCP Games has recently begun doing minor updates and patchwork without bringing the Tranquility server offline for daily maintenance, but this is not the norm).

I ended up asking around and doing some research to find out what could have theoretically happened to the servers that would cause the need for a 25-hour-plus rollback of the servers, and the people I talked to were dumbfounded.  After all, it is simply something that doesn’t happen unless you have a huge clusterfuck happen, the starts align, and a virgin has been sacrificed by a competitor to Aule.  Properly maintained, software and databases don’t implode like they did, especially if you have multiple redundancies in effect to do so.  What happened to the databases for Rift (specifically, the Faeblight server) appears to show the redundancies and multiple “save points” on different servers does not exist.  And since we are told Faeblight is not on any type of architecture different than the rest of the game, that can only lead one to believe that safeguard is not being used for any shard in Rift.  That is a dangerous gamble, as we saw with the rollback: you risk hours—if not days—of information when something goes wrong.

So, how often do rollbacks happen?  Minor ones are actually more common than people think: usually it’s only minutes or hours at a time (more the former than the latter), and while specific amounts are unknown, it’s safe to say that each game has rollbacks of different servers at least every month because of :reasons:.  Major ones—rather, ones that require 6+ hours of rolling back—are nigh unheard of.  Not once that I can think of has there been a need for a game company to rollback servers for more than three hours, otherwise there would have been a huge uproar on the internet about incompetence when it was picked up on.

Now given, there were rumors of a five-hour rollback in 2013 when it came to Star Wars: The Old Republic.  In truth, I haven’t found any actual corroboration from EA/Bioware concerning it, but given that SWTOR is a half-assed game thanks to the efforts of EA, and how shifty the company is, I wouldn’t be surprised.

Just to make sure I was correct in not knowing of any major rollbacks with regards to WoW, I asked a friend of mine who works with their customer service department.  She can’t recall any, and as she put it, “If that ever happened to Blizzard, it would be everywhere.”

We can all agree.

Reimbursement Not Required

One of the more common claims by apologists is that the company does not have to give you anything.  By a strict reading of the EULA/ToS, you’re correct: the company is not bound legally or otherwise to provide you with any type of reimbursement for a failure of the game.  In fact, should the game die a day after you pay for a year-long subscription, the company doesn’t even have to give you a refund, pro-rated or otherwise (though good business practice is to do so, especially if your company has more than one game.  It gives the consumer a sense of honesty, and customers like being able to trust a company).

The problem is that in the past, Trion has done admirably with regards to reimbursing people for mistakes.  When the chat server issue was prevalent earlier this year, every player received three bundles of +160% vials, and they got these if they made new characters for a specific frame of time.  Likewise, when Patron-status players found their “patron buffs” not working for a handful of days (three, I think) each one got a stack of seven bundles, and once again new characters made during a specific timeframe on those accounts also received the same number of bundles.  So it truly is no wonder that the players not only of Faeblight, but the other affected servers as well as the playerbase-at-large expected better than what we got.

What the players got was a temporary 48-hour buff to non-quest experience gain, and 750 credits if you either (a) spent credits during the affected time, or (b) were a “paying customer”.  That, unfortunately, didn’t stack, so you didn’t get 1,500 credits if you filled out both parts of the criteria, and many Patrons pointed out that the buff didn’t affect them.  More to the point, this buff was during a set time, meaning many players couldn’t take advantage of it.  Also, since the credits were given unequally, it was impossible to determine what a “paying customer” was, as some Patrons received the credits and others didn’t; some people who bought Collectors Editions of Nightmare Tide received the credits, and some didn’t.  it made for unequal treatment, and the question of “What constitutes a paying customer’?” was never answered by a Trion employee.

Now you can imagine why after this was announced a week after the rollback had occurred, and the playerbase had been kept in the dark about what was allegedly going on, that said playerbase would rightly be angry.  It may seem like entitlement, but the playerbase had become accustomed to being given a very good reimbursement when something went wrong.  To many of us, this seemed like a slap in the face.

So then, what is the problem?

This is one of those questions that everyone will have a multitude of answers for.  The two chief concerns are Trion’s lack of communication during the entire ordeal, and Trion’s lack of giving a shit when the players say something.  This has not exactly been something new, as it has been happening for some time prior to this event.  For example, on 09 August Daglar responded to criticism of the proposed auction house changes with the following:

“I never want to see someone leave the game over a change, however the nature of the game is change.  We will make changes, sometimes ones that seem moronic to specific players (or even most players) that have the potential to make people stop playing.  We will make changes none the less.”

This was widely touted as Daglar saying that even if the majority of players disagreed with the changes, if he would want to continue on even at the expense of lost revenue he would push on.  I can see where people get that, but I also see that, while poorly worded, Daglar is saying that they cannot allow a vocal group to dictate the direction of the game.

The problem being that Daglar admitted in that very post that even if a majority of players found the change to be counterintuitive or “moronic”, he’d still go along with it.  Because fuck you, it’s his game to be the executive producer of, not yours you unwashed heathen.

Now, this is the beginning, as far as I’m concerned, of a very frightening trend.  But even going back further, my own experience with Trion’s customer service/GM group has been lackluster.  As I posted to my YouTube account, I had a ticket in on people who had been harassing me consistently for over a month.  Even after getting in touch with Ocho, the Community Manager most frequently seen, nothing happened.  As Ocho told me that he would talk to Daglar et al. about my problem, I thought things would be fine.  Two weeks after I finished my chat with Ocho (and a full four weeks after my ticket was initially submitted), I eventually updated the thread I had created, explaining my frustration with Trion’s lack of customer service or taking harassment complaints seriously (my ticket had had a GM copy-paste a form response to my ticket about a week prior with no response after I responded to it).  I had been in the process of editing a YouTube video I had done on Trion’s failure to take such things seriously when a GM magically contacted me in-game to talk about what had been happening.  In true form, they hadn’t read a thing from my ticket, despite it being stupid amounts of specific (listing dates, times, locations, and chat channels that the harassment had taken place in; specifically, the constant use of the term “pedophile” to describe me was mentioned).

That has, however, been a problem with free-to-play games: they put public relations on the back burner a lot of the time, and instead focus on the here-and-now.  Only when things explode and they need damage control do you see how competent your PR/community teams are.

I keep harping back to 2012, when the EvE online “Monoclegate” scandal broke.  This is for a specific set of reasons.  Chief among them is that it’s very simple for me to recall accurately, because I was at the forefront of the protests.  But it also served as a terrific primer on what not to do when it comes to PR, and how a gaming company—even one as big as Blizzard Entertainment—can make huge mistakes, and mistakes snowball, they don’t add up.

The initial thread started pretty much told us that CCP Pann, the CM they sent in to play early damage control, was simply biding time until the company could come up with a response they thought we wanted to hear.  At this point, the players were not happy, and CCP Pann disappeared, ostensibly because her daughter had gotten sick and she needed to be rushed to the hospital (I can neither confirm nor deny this is the actual excuse, and if it is not true it is a poor excuse when you realize someone is out of their depth.  More to the point, a few months later CCP Pann was among the people from the community team fired after Monoclegate happened, and the CEO blamed the players in interviews for “having” to fire them).

Trion To Make Sense

You ever played a game in which there was a massive hardware failure, and the only way to make sure everyone’s account was safe was to roll the entire shard back over 24 hours?  This happened to the Tranquility server in 2012.  CCP Games’s response was a full rollback of about 24 hours, but as compensation to the players they immediately gave everyone who logged in a pretty sizeable amount of skill points.  Now given, in EvE Online, “character level” is measured in skill points, and characters can generally be given skill points equal to a specific formula to help balance out what may have been lost in the rollback.  Of course, this also means that ships lost are replaced, and anything you may have bought or found during the time between the rollback start and when it was rolled back to is lost.  That can be both good and bad, depending on how much Bob and RNGesus love you.

Now, let’s assume that a game’s expansion launched during the week.  Let’s pretend it was on a Wednesday that it launched.  The launch was pretty good: no massive issues except a slightly longer-than-anticipated downtime, and a couple of per-usual hotfixes.  No massive server lag spikes, no issues with questing.  It’s pretty good, and you’re thinking it’s doing really well.

Thursday rolls around.  People have taken these two days off to enjoy the new expansion and explore/level, as gamers are wont to do when a new expansion drops.  Once again, plenty of issues that were expected don’t show up, and you’re feeling really great.

Okay, now let’s put a name to this game: Rift.  The expansion that came out is called Nightmare Tide.

At 0400 Pacific Standard Time on Friday, three servers went down.  The official reason for this was some key databases for the Laethys, Wolfsbane, and Faeblight servers had become corrupted, and so the servers were taken down to facilitate the transfer of data to new databases.  No ETA for when it would be fixed, but we didn’t’ find out the reasoning until four hours after the problems began and the servers went down, and we were told they didn’t know what was causing the issue (rather, they were vague on the cause).

At 1200 PST, Laethys came back online, a full eight hours after it went down.  Wolfsbane followed 25 minutes later.  Faeblight remained offline.  During this time, a hotfix was announced.  However, Laethys and Wolfsbane found themselves rolled back after they came back up, but only to an hour at maximum from when they went down (so at most they were rolled back to 0300 PST on Friday, so people lost nine hours of playtime, though only one of it was actually played).  Faeblight was not so well off: the server was rolled back to 0300 Thursday morning.

That’s a 25 hour rollback, for those keeping track.

Now some people will point out that players on Faeblight didn’t actually lose 25 hours’ worth of gameplay.  After all, no one at that point is going to play that much on a single day.  We need time to rest and what-not.  And you’re correct, we did not.  But I want to explain it like this.

Three servers lost a total of nine hours minimum compared to the rest of the shards out there.  They were down, and so that is time at the beginning of an expansion in which those players are not playing, and getting ready to raid.  For progression raiding guilds—even in games outside World of Warcraft—that’s a big thing.  Sure, the individual server may be at an equal footing for server firsts, but can you imagine if <Ensidia>’s server dropped like that?  Could you imagine the shitstorm that would be caused, and Blizzard’s responses to get them back into contention (we’ve seen Blizzard previously coddle their favorite guild, allowing them to bypass mechanics and exploit for world firsts)?  Trion may not have the same issue in that guilds aren’t vying for world firsts for real-world money, but it still causes headaches for them in that it shows ineptitude on the part of the technicians.

Quite literally, the left hand does not know what the right hand is doing.  Technicians do not talk to GMs (Game Masters, the in-game fixers of problems) do not talk to CMs (Community Managers, who handle social media and forums).  I did a video a few months back in which I had reported some players to Trion about harassment that went beyond any type of trolling.  I had gotten in touch with Ocho, a member of the CM team who promised he would talk to the GMs.  A month later, the head GM (Daglar) contacted me literally five minutes after I had finished the initial video to talk to me about the problem.  It took a month to get on that, and that was after I had talked to a CM, whom Daglar said he never heard from.  So someone was lying, or just dropped the ball.  I would have been willing to give them the benefit of the doubt, but I had been a royal pain the ass with the issue: forum thread that I had updated two weeks and four weeks after the initial ticket-and-canned-answer fiasco, so it was a sign, to me, that they didn’t actually care.  But that is a topic for another posting.

The point is, here the GMs hadn’t talked to the technicians, and it took six hours to find out what the fuck had happened.  Just fucking think about that: the players didn’t know why it had happened, and given most of us probably didn’t care about the why as much as we cared about the how, as in “How do you plan to fix this?”  And we probably don’t care about how they plan to prevent this from happening in the future, because those of us on Faeblight feel completely cheated.

So let’s go back to the problem right now.  The server as a whole is a minimum of forty-eight hours behind every other server.  I say “minimum” because I’m really not taking into account the nine hours we had offlined alongside Laethys and Wolfsbane.  That really affects the morale of the players, who have routinely felt that Trion does not care as long as they are making money.  Now given, I’ve had far more positive experiences in this game than negative, but that doesn’t absolve them of blame when they fuck up big time.  A hardware failure may not be completely their fault, but the corruption of a database so severely that it requires you to go back twenty-five hours in order to restart everything screams negligence (and that’s according to some database analysts, who have said this to Daglar et al. on the forums).

Faeblight was called “venerable” by Daglar, which means two different things: highly respected, or old and antiquated.  Faeblight, being one of the few original launch servers remaining, is probably on the old architecture for the game, so it wasn’t ungraded with the other servers.  It also means that it isn’t backed up like the others (which appear to be once every hour).  That says that those of us playing on Faeblight can probably expect to have this happen again, and the same results will happen.

Players have lost credits they spent, which the company claims they will reimburse (and after the fiasco with Guild Wars II, I believe it will be).  They’ve lost currency, and they’ve lost items that were expensive, or powerful (we’re talking relic-level treasure boxes, epic-level BoE gear, etc.).  They’ve lost experience they gained.  But that can all be retrieved in short order, once people (including myself) get over the shock of what appears to be such a monumental failure in thinking.  More importantly, however, the people on Faeblight (and to a lesser extent Wolfsbane and Laethys) lost time.  Time is a precious commodity, and is the most expensive thing each and every one of us has.  Time is money is a common saying, and it’s true: the server is two days behind as a collective, which hurts us for progression raiding and the like.

But the most important thing Faeblight denizens have lost is trust: we’ve lost trust in the capabilities of Trion to successfully keep the server going.  It may not seem like much, but something as simple as properly backing up the database would have done far more to calm us.  As-is, the players feel slighted, and it’s going to take far more than a title, cosmetic pet, mount, or packaging of +160% EXP packages to alleviate that feeling.

As of this writing, Trion’s head GM Daglar has said that for reimbursement they are putting up an experience boosting event on Faeblight.  However, Patrons (that is, paying subscribers) are not seeing this bonus applied to them, thus alienating the people giving a consistent income further.  Try as the free-to-play crowd might to say that we aren’t special, we are a far more consistent income than the people who spend money to buy credits, because that is a one-time thing normally.  You simply can’t count on that every month, where-as we are generally paying for months upon months in a row.  Many of us have also received 750 free credits to spend in the store as we see fit, which is the equivalent to buying $5.00 worth of credits.  It’s not much, but it’s a start.  You may not be able to buy us completely, but you can definitely make us feel better with reimbursement that is commensurate with the level of fuck-up presented.  I’m not saying that we should all get a million credits, but something as simple as some special vials that might last four hours and over +400% EXP gain, as well as everyone on Faeblight affected by this server issue receiving a free 3-day Patron pass to make up for the time we are missing out on because we have to redo what we already did would go a logn way to showing us we care.  And that’s just for starters.

The EULA might say we can’t expect anything from Trion, and in fact Trion doesn’t have to give us anything as compensation, but companies have tried that before, and the negative publicity alone has caused them endless amounts of headaches.  I fully expect Trion will try to repair the damage, but the repairing needs to happen sooner rather than later, and it needs to be seen as something serious.  If Trion does that and restores the faith of the playerbase—because players on other shards are certainly watching this, understanding it could happen to them—they will have a lot more loyal customers, and the media will pick up on it.

I will update as more information becomes available.

Head GM Daglar's Public Response

Edit At the beginning of the post I mentioned that CCP Games rolled the Tranquility server back 24 hours.  This was incorrect.  At the time, the Tranquility server was under a concentrated DDoS attack that kept the server down for a period of 18+ hours, which meant that during this time skills were not training.  As such, CCP reimbursed players a specified formula of skill points in order to apologize for this issue and worked to resolve the problem.  CCP did not roll the server back.