Measurement

Measurement has been one of the main themes of this blog since we started it and this time we’re using it to explain some of the background to ICEmarq™, our brand new internal communication service.

With spending in many organisations under the microscope, demonstrating return on investment has never been more important.  Our goal with ICEmarq™ is to help organisations measure that return by providing them with what they need to run an effective measurement programme. We explain how ICEmarq™ works on the ICEmarq™ site but here I’d like to share some of our observations about measurement and how they have influenced our approach to ICEmarq™.

• No single ‘right’ way. If you ask an organisation what aspects of internal communication it would like to measure you will get many answers. Some will focus on cut-through of key messages, others on the performance of key channels and a few will focus on the wider of benefits of good communication such as advocacy or higher discretionary effort. There is no ‘one way’ of approaching the measurement challenge and what an organisation measures will ultimately come down to what it really wants to achieve with its communication. 
• In-house tools and expertise. We’ve observed an increase in the number of internal surveys in recent years thanks largely to the availability of easy-to-use tools such as Survey Monkey™.  The existence of such tools and the expertise to use them means that organisations no longer require the full-service approach to research and instead prefer the opportunity to buy only those components it actually needs. At the top of that list sits establishing the right questions to ask and interpreting results effectively.
• Placing results into context. Whilst we believe an organisation can always draw conclusions from a set of survey results, we also believe those results are more meaningful if they can be compared to an external benchmark. We tested this point repeatedly with those we’ve researched ICEmarq™ with which is why we’ve made the availability of statistically reliable and representative data a cornerstone of the service. We’ve also sought to extend the principle by offering ICEmarq™ Unique for any organisation wanting a precise benchmark sourced from a panel made up exclusively of people working in its industry sector and/or countries in which it operates.

We’re very excited about ICEmarq™ and over the next few weeks we intend to share some of the most interesting ICEmarqs™ via this blog. But for now, we’d love to hear any feedback you have about internal communication measurement in your organisation!

A CEO recently questioned the results of a pulse survey. The survey was short – 6 quick questions – and was part of an ongoing programme to measure the extent to which colleagues engage with the new corporate brand. And the results were good with double digit percentage increases across most of the 6 indicators. So why the challenge and how best to respond?

The first angle to cover is the statistical one. Can we treat the results as statistically significant and representative of the whole population? This sounds complex but actually isn’t with easy to use online calculators available to determine minimum sample sizes. As a rule of thumb, if the sample size is small and if the overall size of the population is less than 2,000 then you will have to settle for a wider confidence interval (which means less certainty) than if both are larger.

It’s unlikely though that concerns about statistical validity were uppermost in his mind. More likely is that he has a different perception of what people are saying, thinking and doing than respondents say they have.

There might therefore be an issue with the questions being asked. Are they specific enough or do they allow for too much interpretation? An obvious consequence of keeping a survey short is that questions tend to get more general. So rather than ask whether respondents see evidence of the brand being lived every day, maybe we ask whether they see evidence of prompter or more courteous customer service or whatever else may make up the brand promise.

What about the degree of direction respondents were given? A lot of organisations push surveys hard and in some cases hold managers responsible for response rates. This can obviously have an impact on the mindset of the respondent and it can be positive or negative. In this case, the survey was live for 2-3 weeks and other than a couple of emails to make people aware of it, no particular pressure was applied. Importantly, all responses were given anonymously.

And is employee perception really a reliable measure? As an alternative, what about using clear objective measures (that are not usually tracked) such as arriving at meetings on time or responding to email within a particular timeframe and observing behaviour. This moves us away from perception and into cold, hard reality. But it also requires more effort and will almost certainly cost more to do.

Organisations often rely on the perceptions of employees as a way of measuring success or otherwise. It’s easy to do, creates involvement and can be a reliable indicator of performance. Agendas are often based on the results and managers held accountable for future performance in key areas. All of which suggests that the CEO should simply accept the results and take the credit he and his team deserve.

But that allows those of us who run surveys on a regular basis off the hook. A challenge like this is good as it forces us to really assess the validity of what we are doing. Are we really asking the right questions given the wider objective (in this case, stronger brand engagement)? How certain and open are we about the statistical validity of what we have learnt? And is a survey of employee perception really the best way of determining success? All good questions and all ones we should ask next time the brief requires us to run a simple survey.

We recently uploaded case studies featuring internal communication reviews we undertook for two very different organisations. One was for British Airways and was based around an extensive survey questionnaire with a strong channel focus. The other was for a UN agency called UNRWA and was based around a series of focus groups and interviews, undertaken during two fascinating weeks in the Middle East. Two projects with a very similar purpose but with quite different approaches.

The output was also quite different. In the case of BA, our main focus was an extensive analysis of the answers we received from the survey which led to a series of conclusions and recommendations. With UNRWA, our output was focused less on what we’d learnt and more on the channels and supporting infrastructure needed if communication was to improve. A quite different challenge altogether.

But the two projects were similar in one very important way. Both were one-off exercises that need only happen every 2-3 years and both were undertaken on behalf of those directly responsible for communication. And both also included the recommendation to introduce an ongoing measurement programme which brings me to my main point.

Communication reviews and audits are great as diagnostic tools for understanding how communication is currently viewed and where improvement can be made. But they are essentially inward looking and designed to inform future planning. More a means to an end. A measurement programme is an ongoing assessment of performance against pre-agreed criteria. Its purpose is to summarise the impact of communication activity and provide the organisation with evidence that the investment is worthwhile. An end therefore in itself.

In my view, both are equally necessary and valid exercises. But the crucial point is not to confuse the two. Put very simply, review once and measure often!

Change as we all know is the one constant in any organisation. And the effective communication of change remains one of the core roles of any communicator. So as a new year, indeed decade begins and my focus set firmly on a couple of change projects, what better time to reflect on some of the main priorities when communicating change. What’s really interesting is how little the list has changed in 10 years.

• Clarify the objective. One area of change management I believe organisations have improved in is the clarification of programme goals and success factors. I absolutely believe communicators should follow this lead and be very clear about their objectives if only to demonstrate that their role on the project has been a success. And in an age when survey and polling tools sit on most Intranets, ongoing measurement is a must.
• Consult, consult, consult. How often have you heard people ask who exactly was consulted? It is of course code for this doesn’t work for me (or something like that) and is often a consequence of the late introduction of the professional communicator to the process. A key priority for any change communicator is to grab the ongoing consultation and testing so dialogue is constant, professionally managed and received in time to be of value.
• Experiential, peer to peer communication. This strictly is two points but between them, sum up by far the most effective way of getting change messages across. First, don’t just talk about it, show people how it works. There really is no substitute for demonstrating the new system or procedure in action. And second, enrol people whose opinion is valued by your audience to do the talking. Any other way simply smacks of unwanted imposition.
• Manage the whole brand experience. Change teams are often disparate combinations of contractors, subject matter specialists and enthusiastic in-housers. They are often transitory and have a tendency to ignore the truism that a brand communicates in many ways. Many people come into contact with a change brand long before it is introduced and ensuring that all interactions with that brand are everything they need to be (responsive, speedy etc) is for me another big priority.
• Get the product right. This one goes well beyond the compass of the communicator but it is fair to say that a bad product will undermine, if not totally negate the very best communication. Clearly it is our job to make the change owners aware of any potential issue and if we’ve done each of the four earlier steps, the chances of this happening are greatly reduced. But we must always be prepared to cope with a crisis and ensure that the damage is limited.

The list could of course go on. But there is a wealth of knowledge and experience to tap into out there. And a good place to start is probably http://www.melcrum.com/topics/change.shtml.  Good luck!

The subject of my (newly) teenaged son’s homework came up the other day, a familiar if delicate subject for any parent. Is he getting enough? Is he doing enough? And what are his marks like? Partly in an attempt to move on (is October of Year 8 really the right time?), I suggested ‘worrying about outputs not inputs’ by which I think I meant ‘exam results’. It’s not the first time I’ve used those words, as anyone with whom I’ve ever done a comms survey will testify. But with a number of communication and engagement surveys currently on the go, it seems a good time to revisit the argument.

First, let me clarify my meaning. By outputs, I mean the wider organisational outcomes we are seeking to bring about which in the case of an internal communication function might include building staff engagement with strategy or boosting trust in leadership. By inputs, I mean the activities we undertake to make this happen which might include updating the Intranet or running CEO roadshows.  In other words, one is an end in itself and the other a means to an end.

But what if our outputs are not properly defined? What if our very best efforts to clarify the role of the function still mean multiple interpretations of our purpose? And what if we ‘own’ only part of the outcome (there’s only so much influence we can have over staff engagement with strategy or perception of leadership)? Under these circumstances, measuring output becomes difficult and focusing on inputs suddenly appears more attractive. After all, no one’s going to get fired for measuring hits on a website or reaction to the CEO roadshow, sharing the results (always important) and learning from the feedback.  

My big concern and the basis of my argument is that by focusing on the inputs, we overlook the organisational outcomes the IC function is really trying to bring about and then double the crime by failing to use the opportunity presented by a measurement exercise to define it. That said, this can be a long and tiring process and deflect from the primary function of any good IC function which is to facilitate better internal communication. 

I make no apology for beating the ‘outputs’ drum as I believe every IC function should be clear about the wider outcomes it is seeking to influence. But to not measure all those tangibles that the function is best known for is equally negligent. So a small but important revision to my favourite expression; internal comms measurement is about outputs and inputs. Better go and check on that homework then.