Audit

A CEO recently questioned the results of a pulse survey. The survey was short – 6 quick questions – and was part of an ongoing programme to measure the extent to which colleagues engage with the new corporate brand. And the results were good with double digit percentage increases across most of the 6 indicators. So why the challenge and how best to respond?

The first angle to cover is the statistical one. Can we treat the results as statistically significant and representative of the whole population? This sounds complex but actually isn’t with easy to use online calculators available to determine minimum sample sizes. As a rule of thumb, if the sample size is small and if the overall size of the population is less than 2,000 then you will have to settle for a wider confidence interval (which means less certainty) than if both are larger.

It’s unlikely though that concerns about statistical validity were uppermost in his mind. More likely is that he has a different perception of what people are saying, thinking and doing than respondents say they have.

There might therefore be an issue with the questions being asked. Are they specific enough or do they allow for too much interpretation? An obvious consequence of keeping a survey short is that questions tend to get more general. So rather than ask whether respondents see evidence of the brand being lived every day, maybe we ask whether they see evidence of prompter or more courteous customer service or whatever else may make up the brand promise.

What about the degree of direction respondents were given? A lot of organisations push surveys hard and in some cases hold managers responsible for response rates. This can obviously have an impact on the mindset of the respondent and it can be positive or negative. In this case, the survey was live for 2-3 weeks and other than a couple of emails to make people aware of it, no particular pressure was applied. Importantly, all responses were given anonymously.

And is employee perception really a reliable measure? As an alternative, what about using clear objective measures (that are not usually tracked) such as arriving at meetings on time or responding to email within a particular timeframe and observing behaviour. This moves us away from perception and into cold, hard reality. But it also requires more effort and will almost certainly cost more to do.

Organisations often rely on the perceptions of employees as a way of measuring success or otherwise. It’s easy to do, creates involvement and can be a reliable indicator of performance. Agendas are often based on the results and managers held accountable for future performance in key areas. All of which suggests that the CEO should simply accept the results and take the credit he and his team deserve.

But that allows those of us who run surveys on a regular basis off the hook. A challenge like this is good as it forces us to really assess the validity of what we are doing. Are we really asking the right questions given the wider objective (in this case, stronger brand engagement)? How certain and open are we about the statistical validity of what we have learnt? And is a survey of employee perception really the best way of determining success? All good questions and all ones we should ask next time the brief requires us to run a simple survey.

We recently uploaded case studies featuring internal communication reviews we undertook for two very different organisations. One was for British Airways and was based around an extensive survey questionnaire with a strong channel focus. The other was for a UN agency called UNRWA and was based around a series of focus groups and interviews, undertaken during two fascinating weeks in the Middle East. Two projects with a very similar purpose but with quite different approaches.

The output was also quite different. In the case of BA, our main focus was an extensive analysis of the answers we received from the survey which led to a series of conclusions and recommendations. With UNRWA, our output was focused less on what we’d learnt and more on the channels and supporting infrastructure needed if communication was to improve. A quite different challenge altogether.

But the two projects were similar in one very important way. Both were one-off exercises that need only happen every 2-3 years and both were undertaken on behalf of those directly responsible for communication. And both also included the recommendation to introduce an ongoing measurement programme which brings me to my main point.

Communication reviews and audits are great as diagnostic tools for understanding how communication is currently viewed and where improvement can be made. But they are essentially inward looking and designed to inform future planning. More a means to an end. A measurement programme is an ongoing assessment of performance against pre-agreed criteria. Its purpose is to summarise the impact of communication activity and provide the organisation with evidence that the investment is worthwhile. An end therefore in itself.

In my view, both are equally necessary and valid exercises. But the crucial point is not to confuse the two. Put very simply, review once and measure often!