Category: News

Ever wonder why political opinion polls can claim a margin of error of 3% when they survey only 1067 people out of an electorate of, say, 2 million?  1067 is a tiny percentage of 2 million – way less than 1% in fact.  So how is this possible?  For most people, such a low response rate (in percentage terms) would provoke a sceptical reaction.

Response rates get a lot of attention when it comes to surveys.  Whether it’s a business conducting market research or an organisation conducting an online survey to gauge member satisfaction, the focus is often on reaching some target response rate, for example, 40%.

Of course there is merit in this focus on response rates – the higher the response rate, the more representative the data.  But response rates are only part of the picture.  The margin of error is also very important.

Imagine two online surveys achieve an identical response rate – 20%.  The first survey was of 200 people – so it got 40 responses.  The second survey was of 1000 people – so it got 200 responses.  But guess what?  The first survey’s margin of error is 13.89% whereas the second’s is 6.2%.

This has some interesting implications :

• As the size of a population increases, and if the percentage response rate stays the same, the margin of error decreases.
• As the size of a population increases, the required sample size to achieve a given margin of error decreases!

So – be mindful of response rates – but don’t focus exclusively on them!

Recently, while travelling, I came across a customer satisfaction comment card for a hotel.  These cards were being left in guests’ rooms.  Besides questions about guests’ experiences, there were also some profiling questions.  One of these questions asked guests their age.  Such questions can be very useful.  For example, if a hotel is surprised that the satisfaction level among its younger guests is far higher than among retirees, then there is some underlying issue that they can aim to uncover.

However, in this example, the lowest selectable age group was 25-34.  What about the under-25s?  What about their opinions?  Were they simply forgotten?  Or was there an assumption that all guests would be over 25?  When designing your questionnaires or online surveys, be careful that you ask the right questions!

There has been a lot of talk recently about how it’s no longer enough to satisfy customers.  We’re told that we need to “delight” them.  If you are running an online survey to assess customer satisfaction you may well be looking to know how to delight your customers.  But be careful – you might be better off first finding out whether you’ve irritated them, whether you’ve let them down, whether you’ve ignored them.  Providing a smooth, hassle-free interaction in a responsive manner may be all they’re looking for from you.  And if you mess up on any of those you risk them telling others.  So yes – reach for the stars – but do so from a firm foundation!  Make sure your survey design doesn’t neglect the basics!