I recently purchased a new car. A few days later I got an email from Audi asking me to rate my experience. I clicked the link to the survey and ended up seeing this:
Ok, this should be easy.
“Ease of looking at dealer’s inventory” – great, no problems there. A 10, right? Well… was it OUTSTANDING? How about TRULY EXCEPTIONAL? No, it wasn’t those… I can’t say someone’s inventory was truly exceptional. I can’t put my name on that sort of endorsement. So…?
Comfort in the office where we cut the deal? It was fine – I couldn’t imagine it to be better, but was it TRULY EXCEPTIONAL? No. That doesn’t fit. So does that make it a 6 or 7? No, it was better than that… But… So…?
I see this sort of thing in surveys all the time. A simple 1-10 scale (or 1-5, it doesn’t matter), but the labeling of the numbers is so sensationalized that it turns me off. As far as the number goes, I’m happy to give something the highest rating, but the language overshoots the number and then I don’t know how to respond.
I find these sorts of things great reminders of how important it is to choose the right words. Don’t overshoot, don’t sensationalize. Be modest with language. Find the right fit and leave it alone.
Mattias
on 02 Jan 13Isn’t the point of the survey to get you to add wonderful words to your memory? The whole thing is just a scam where it forces you to spend time writing positive things, while reinforcing that opinion in your mind.
You know, like contests where customers writes slogans. This is just an easier task to get the customer to do.
Scott Higdon
on 02 Jan 13Your post was outstanding, if not truly exceptional.
Joe Taylor Jr.
on 02 Jan 13Surveys like these just rob us of the conversations that dealers need to have with their customers. Dealers need to be able to print ads that say “more than X% of customers rated us as outstanding,” even though this scale rates “average” at 4 and “outstanding” at 6-7. Skewing the scale makes it easy to pass less-than-stellar numbers as great ad copy for customers who may only make purchases every 3-6 years.
Michael
on 02 Jan 13I hate scales like this, and usually don’t fill out the survey when I see them. Give me three choices: sucked, ok, great.
Charlie Hayes
on 02 Jan 13Maybe Audi is trying very hard to create a Truly Exceptional experience and really wants to know how well they are doing. It could also be a litmus test to see if users are just picking all 10s.
Jason Roelofs
on 02 Jan 13There’s another even stranger aspect to these surveys. I recently purchased a new Volkswagen and was told numerous times that if I chose anything other than a 10 on these surveys, VW would consider that a failure mark against the dealership.
I liked the dealership so I just did all 10s and didn’t read much of the survey. I have received other ones after maintenance / checkups but threw them away.
10 choices, but in the end it’s nothing but a Good / Bad survey that most people either won’t fill out, or will just put all 10s and not really care about the questions. What does VW/Audi intend to learn from this?
Graham
on 02 Jan 13The explosion in popularity of customer surveys in retail in the recent times has got to be linked to some common retail-insider report that figured out that customers who are forced to answer surveys get that brand ‘stuck’ in their head for a little longer than if they didn’t.
Seems like I am bombarded with the same spiel from all big box outlets when handed a receipt: “Here, take the survey on this receipt and get a chance to win {crappy thing}!”
Perhaps the whole thing is just to glean a valid email or IP address out of customers. Regardless, I find it annoying, and one more reason to shop at either truly local places (who don’t do this) or online at Amazon.
Jean-Sébastien Dussault
on 02 Jan 13I think the words are, contrary to what you say, spot on. This scale just shows how flawed a simple 1-10 scale could be misread. Everyone’s idea of a 10 differs. By telling yourself “It was great, but I wouldn’t call it truly exceptional”, you are admitting that your 10 still could be better, even if you were totally satisfied.
By putting words into the scale, Audi does several things right: –They CLARIFY the scale. Yep, you had to think and putting your mark wasn’t automatic as a usual 1-10, but at least, everyone’s on the same page on your appreciation. –They tell you that Audi standards are high: they believe that going above and beyond what’s expected by the customer is their goal, and that “satisfactory” or “great” is not worth printing. How’s that for branding? –They really care about your answers. By making you reflect a bit about what you put as an answer, they’re telling you they’ll probably actually put effort in analysing data.
In the end, I think your conclusion is all wrong. If something is in excess in these kind of polls, it’s the numbers, as they mean something different for everyone.
Mark Copeman
on 02 Jan 13Jason, I couldn’t agree with you more, which is exactly why we built CustomerThermometer.com
We learnt how to build it from you. Simplicity rules. We were completely inspired by Getting Real! We built a diary around the build watchusgettingreal.com which readers may relate to.
Thanks, Mark
scott
on 02 Jan 13What Audi did you buy?
Fabian
on 02 Jan 13I also think it’s just a measure to get a legal basis for advertising with customer testimonials à la “93% rate us as outstanding”, even though the average might be 6.5. Sorry, but for me forms like this lack one big checkbox: “[x] Bullshit”.
Dylan
on 02 Jan 13Now, onto more important questions…what did you buy?
Matt Van Horn
on 02 Jan 13Also – how the hell is ‘4’ average on a scale of one to ten?
Bob D
on 02 Jan 13Someone should recommend Net Promoters Score (aka NPS) to Audi. There really should be a single question on any survey, On a scale of 1 – 10 (extremely unlikely up to highly likely), how likely are you to recommend our XYZ service to a friend or colleague? 10 responses leaves a lot of subjective interpretation, but if you couldn’t get a 9 or 10 you failed to delight the customer enough that they would recommend your xyz service to their friends. 9’s and 10’s are what drive future growth, they promote you, they were happy with what you provided. 7 & 8, they are passive, they won’t help growth, but they won’t hurt your growth either. 6 and below are detractors, they will hurt your growth. For our company, every survey with a 6 and below gets a call from a mid level manager to discover what we did wrong. With a single clear question our survey return rates went up +400% and it really gets us what we need to know…did we delight the customer enough to the point they would recommend us.
proctime
on 02 Jan 13well, thats what you get when you buy a volkswagon I guess :) bean counters making anemic surveys that are pointless and I doubt they even read them or use them for anything – what they tell to their boss is how many surveys they sent out and maybe how many were returned – beyond that, they could care less and you should not care either – disregard, just fluff in our already overloaded information rich lives
Jesper
on 02 Jan 13@Jean-Sébastien Dussault: Audi being honest about their aspirations is one thing, but that doesn’t make the scale any saner. “Average” and “Outstanding” being adjacent is completely out of whack. If this is intended to move the needle so that I will think of an ordinary “good” experience as an outstanding one, it will backfire if I consciously notice the discrepancy and it will spoil my assessment if I don’t. Playing games with the scale ruins the data.
And even if people actually put a 10 out of 10 only when it’s the best experience they’ve had, they could have a way better experience tomorrow for all they know. These scales can never actually be about absolute measurements, only about what score you are willing to assign yourself. You can’t right everyone to score to the same standards on the same scale.
I’m hoping that whoever designed this survey or are responsible for tallying it or following it up knows that the aggregate, the broad trends and possibly the sequential change are what’s important. That relies on data that makes sense across the board. I would love to hear what 37s-Noah has to write about this.
Dave M
on 02 Jan 13Jason Roelofs, you’re right about getting steered to the 10’s (“everything else is a FAIL!”) – this from a waitress at a TGI-Crackerbee’s type place. And I recently filled out a 20-minute survey from Barnes and Noble for a self-service, one-item trip that took about 15 minutes, and would have taken less if they’d staffed the registers.
I don’t mean this to be a big “me too!” comment, but these surveys are crap. Too many questions, too wide a numeric scale, flowery rating language (I’d rather see something short like 0-3: Fail, Meh, Cold Beer, Chuck Norris’s Beard). InfoWorld’s Bob Lewis has been saying this for years – measure the wrong thing or measure it the wrong way, and you’re either doing damage or at least wasting your time and money.
Zeynel
on 03 Jan 13Hello, sorry for the off topic comment, but I was wondering if you are planning to make this blog platform to the public. I really like the design. It is great. Thank you.
Anonymous Coward
on 03 Jan 13I was working for an other German brand. Actually, the result of surveys were used by the importer to calculate the gross margin (for the following year) of the local dealer. There were also other evaluations (percentage of attributed quota of “end-of-life” cars sold, # contract signed, # warranty signed, etc.) taken into account.
Anonymous Coward
on 03 Jan 13I was working for an other German brand. Actually, the result of surveys were used by the importer to calculate the gross margin (for the following year) of the local dealer. There were also other evaluations (percentage of attributed quota of “end-of-life” cars sold, # contract signed, # warranty signed, etc.) taken into account. These surveys are important for the importer because he has no direct conversation with the customer. It is also really important to note that the most exclusive brands tend to have the least information, because only the seller has a contact with the customer (that’s usually the customer desire). And these sellers tend to keep the information of “their” customers.
Jake
on 03 Jan 13My fiancée and I recently purchased a car and our salesman let us know in advance that this survey was coming. As he explained it, the corporate office words these surveys as they do to illicit just the kind of response you wrote about. For every X amount of 10s a dealership/salesman gets, corporate gives out a bigger “bonus”.
I wouldn’t be surprised to find that our salesman was trying to milk some sympathy and have us rate him and the experience he provided with all 10s, but my time working in a large corporation tells me I can’t write off this explanation as absurd either.
EH
on 03 Jan 13A car salesman milking it? Perish the thought.
Ben Garvey
on 03 Jan 13When I bought my last car (a Ford), I was told several times that they get eviscerated by corporate for anything less than a 10 and that their bonuses are calculated using these surveys.
Darcy Fitzpatrick
on 03 Jan 13Jason, this is very reminiscent of the surveys I’ve received from VW.
I had the exact same experience as Jason Roelofs, where I was told by several people at my dealership throughout the buying process that if I didn’t give them a 10 on the impending survey, VW would view it as a fail. That really soured what should have been a pleasant buying experience, and it’s tainted my overall impression of the dealership.
Even if VW is putting this kind of pressure on their dealerships, which Jason Roelofs experience combined with mine would seem to indicate, it’s up to the dealership to strive to meet these standards without having to resort to pressuring their customers. That defeats the whole purpose.
As do these types of surveys. Even now that I’ve started being honest on my VW surveys (I caved into the pressure at first), it’s hard to give a true sense of your experience when they word the surveys so poorly.
Matt B.
on 03 Jan 13I recently took my sister into the VW dealership and at the end of the sale, the salesman pulled her aside and said that when she gets the online survey if she would be so kind as to give him perfect scores. He then said it can’t even be one down from the best score, it has to be a perfect score or “it doesn’t count” for them. Okay, we sort of just chuckled about that on the way out. Then a few months later when I was up the street at the Audi dealership getting my S4, that salesman pulled me aside and said the EXACT same thing. He said they have to have perfect scores.
Nothing against sales positions, but I think they get so caught up in living their pitch and gabbing with people, that sometimes they don’t know when to turn it off.
Nic
on 03 Jan 13I hate these surveys – too many questions and too much range on the scales. The easy way out is just to rate everything as (1). At worst you may get a phone call, at best you will stop getting surveys – they don’t like negative responses.
JZ
on 04 Jan 13I’ve had similar reactions to survey language. “Extremely satisfied”, in particular, gets under my skin. I’ve often been satisfied with a product, but has my level of satisfaction ever been “extreme”? I don’t think so. Nor have I ever been “extremely likely” to recommend, re-visit, purchase, etc. It’s as if on one ever considered that a customer might try to read these surveys.
paul
on 04 Jan 13Jason, we bought an Audi recently and received the same survey as well. The problem is not just the scale usage, but creating scales for things such as “inventory” like you say. As I am sure you did, you called first, verified inventory and then went. So they need some “not applicable” options. Companies use 10-point scales to try to get variance in responses that they can then run models on to get at things like derived importance and so forth. If they did “yes/no” they would not be able to run the models, even if the data were to be cleaner and easier to answer.
Personally, I am a fan of 4-point or 5-point scales that are are labeled. As you note, people have different perceptions of what an 8 vs. 9 means. But a scale such as poor/okay/good/great is pretty straight forward. It is also easier for the dealerships to understand they moved from 20% great to 25% great vs. from 8.2 to 8.7. Unfortunately, all dealerships (even Audi) tell you “rate all 10 or please call us, we get in trouble if it is not all 10). Some offer free gas to bring the survey in or let them fill it out (in the old paper days) and so forth. I run a market research company and getting customer feedback is important, helps companies innovate, improve the UX and more, but the car survey system is completely broken and the data is not useful.
Jean-Sébastien Dussault
on 08 Jan 13@Jesper
What I’m saying is that this scale would be better with the same words and no numbers (with unmarked radio buttons between them) than any plain vanilla 1-10 scale with average in the middle.
People who had a great experience will check before “outstanding” and those with a satisfactory one will check after “average”. I really don’t see the use for putting more degrees between those (the same way we don’t need more than two degrees before falling in the “unacceptable” disgrace). THAT, in my opinion, would be a loss of time to interpret.
Also, I’m not talking about Audi being clear about their aspirations (that part is more in the unsaid spectrum), but rather about making sure everyone is on the same page about the value a 10 (or 4 or 7) means on that particular scale. I think the word-values are clear enough in this design so people don’t go and put a “5” -average or 10-good by mistake and will rather notice the unusual and think about it before grading.
Carrie Herrera
on 09 Jan 13I wouldn’t have given this scale a second thought, but your comments on it are spot on! Notice how there are more opportunities for a positive response compared to a negative one (after number 3 it’s all pretty great, or least middle of the road). The two extremes should be opposites; unacceptable and acceptable, or truly unacceptable and truly acceptable. I’m all about being fair and this survey isn’t.
This discussion is closed.