To benchmark or not to benchmark – that is the perennial question when reporting organisational surveys. That is understandable as there are many compelling reasons, both for and against, and it can be a minefield to navigate, so let us lay out some pointers to help you plot a course.
What do we mean by benchmarking?
This is a whole topic in itself, as benchmarking is a very generic term for a whole host of different types of comparisons. It could mean benchmarking a section of the population against the overall population, or it could mean comparing the data with equivalent historic data from the same organisation. In this instance, we are specifically going to explore external benchmarking – making comparisons with data found elsewhere in the industry or commensurable organisations. Even within that specific case there are a number of different ways of benchmarking: comparing to the average, looking at the range of the benchmark data (lowest to highest) or measuring by percentile (which quartile is your organisation in, relative to the benchmark).
Why do people consider benchmarking?
This is a really important question to answer, and the conclusion may change slightly each time it is asked. When starting out with an organisational survey, benchmarking to other similar organisations can be reassuring. It can help to provide a sense of perspective and context. Understanding whether the results you are looking at are within expected norms can be difficult to begin with, and benchmarking can feel like it helps to answer than question in the early days. However, once a baseline has been established and an understanding of the organisation’s unique ‘footprint’ has been achieved, the relevance and value of external benchmarking begins to dwindle. That is not to say that it becomes entirely redundant, particularly when external pressures and changes buffet an entire industry, but it is generally less relevant as an organisation matures on their journey. Benchmarking usually carries a significant cost premium (sometimes more than doubling the cost of a survey), so its value has to be assessed in that light.
The Do’s and the Don’ts
Do look for organisations in similar circumstances
It might seem obvious, but it is important to compare like for like. Simply being in the same sector or industry vertical may not provide this for you. Are the organisations a similar size? Do you have some local advantages or disadvantages? Are you at the same stage in the organisational cycle? Comparing with disimilar organisations can create a false sense of failure, or an equally false sense of security.
Do research to ensure the comparative organisations have not had a significant event prior to their survey – change in leadership, long overdue payrise, etc…
As with the previous point, significant events like these are likely to impact the results that organisations in the benchmark have recently achieved, in either direction. In order to achieve a meaningful benchmark, you need to be aware of events that may impact it. Benchmarks are use usually based on a random section of your sector, and will obviously predate your survey and recent events in your industry, so they are impacted by historic events, but not current ones.
Do look for outliers and exclude them
Just as you would look for outliers in any other data set, you also need to be mindful of this in a comparative data set. If you have chosen a group of organisations to benchmark with, but find for example that one has a very wary culture which depresses their responses, it may be pertinent to exclude these from your comparisons. If you are using externally provided benchmark data, or an external agency is doing the benchmarking, it is important to understand what is in the external data and what filtering has been applied.
Do ensure the metrics you are benchmarking against are comparable
Accurate and meaningful benchmark comparisons can only be achieved if the questions and the metrics created from them are, in themselves, comparable. Were the questions used in the same context, and would they have been interpreted in the same way? Have any of the questions or answer choices been modified or removed? Were they presented in a different part of the survey or with different instruction text. All of these things affect the data.
Don’t benchmark against an ‘industry standard’
Just because an organisation is in the same sector doesn’t necessarily mean it is comparable to your own. It may be brand new, in a different part of the country or world where the economic situation is different, or immediately post-merger or restructuring, etc. In reality, when it comes to benchmarking, there is no such thing as an industry standard, and if the data set you are benchmarking to is made up of an unrepresentative set of incomparable organisations, the benchmark will not provide you with much useful insight.
Don’t focus only on what can be benchmarked
Your main focus should be on what is happening within your own organisation, uncovering any differences of experiences between groups, informing your strategy, with any benchmarks providing a datapoint in addition to this. Overly focusing on the questions which can be benchmarked will be a distraction from the important insights.
Don’t place more importance on the external benchmark than your employee’s opinions
As with the above point, do go back to the reason that you are conducting your survey. Unless this was to outdo the Jones’, your benchmark needs to take its place in the hierarchy of data. If the sentiment around a particular topic is above the benchmark data, you might view that as a strength. However the reality is that if the sentiment is below your organisational norm, or falling, then actually you still have an issue to address. Employee’s do make external comparisons, but they place significantly more importance on their historical experience and the local narrative. You might be the top scoring organisation, but if employees felt that things were better a year ago, then that is still a key issue to address.
Don’t doubt your own ability to know what is ‘good’ in your situation
Whilst benchmarks are useful, particularly when talking with people outside of your organisation, if the comparisons don’t ring true, don’t feel you need to ignore your intuition. Dig into what you know and try to find out if and where the disparities are coming from.
Benchmarks are for context, don’t turn them into a competition
This is the most important point which can’t be stressed enough. A survey is not a test or an exam, and as soon as it used this way, at best the data quality will begin to fall, and at worst, a toxic culture will begin to take hold. If you start to ‘chase’ the benchmark as a target, you are charting a course towards average, at best. If you fall short of that, then your organisation is going to be in a very challenged position.
Above all, keep in mind that any external benchmark when used well merely adds a little additional context, it is never the be-all-and-end-all to your survey results. Historic and internal comparisons within the well-known organisational context will always provide richer insights, and capability modelling will allow realistic stretch goals to be set.
Benchmarks and Goodhart’s Law
The famous economist Charles Goodhart made an observation: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” – it can be more simply be restated as this: “When a measure becomes a target, it ceases to be a good measure.” If managers or leadership are incented on engagement relative to a benchmark (an output measure) then it creates pressure to increase the measurement, rather than improve the organisation. People are encouraged to answer more positively, disengaged staff are discouraged from completing the survey, there is a focus on the questions that are benchmarked, rather than the questions that improve the organisation. It might initially sound unlikely, but experience has shown us that it is inevitable. A benchmark sets starting context, after that it has diminishing meaningful use. It usually provides comfort when actually challenge is required.
If any of this has provided food for thought and you would like to chat further, please get in touch using the chat button below – one of our team would be very happy to explore these ideas with you.
For further reading, you might also be interested in our post about Conducting surveys after a Reduction in Force, a significant event within an organisation which may impact your approach to benchmarking.