Our Head of Reporting and Analytics talks about evolving methodologies around measuring engagement.
One of the most exciting things about developing a new product is putting it in clients hands and watching how they use it.
When we developed Workometry the original use case was for it to capture and analyse frequent employee feedback in a ‘Pulse’ survey. A second use case was for it to be used to understand experiences throughout the employee lifecycle.
To recap, Workometry is our employee feedback tool, launched in 2015 whose differentiator is a focus on capturing and analysing open text feedback. It is a responsive survey front end on top of some very sophisticated computational linguistics and machine learning algorithms.
Our clients usually ask 3 to 5 scale based questions and 2 or 3 open text questions in each questionnaire. Employees typically take no more than 5 minutes to answer the questions.
We first code the responses, using an approach which is unsupervised meaning we detect topics automatically from the data that we capture. Typically we capture between 30 to 50 topics (things such as “shortage of staff”, “better work-life balance”, “the car park situation”) from each open text question. We can do this across multiple languages. We tune our algorithms at the question level.
We then link the perception data to other employee data on an individual-level and use a probabilistic modelling approach to identify groups of employees who are especially likely or unlikely to discuss each topic. These segments can be quite refined, e.g. “women under the age of 30 with university education are especially likely to mention a shortage of career development opportunities”.
Most of our clients started by asking traditional employee-engagement or satisfaction type questions. However increasingly they’ve (and we’ve) realised that the tool can be used for a much more business-specific set of questioning.
One client recently used it to capture and understand large volumes of employee feedback on a newly announced strategic initiative. The two open questions were asking for examples of good-practice where the new approach had been applied and the barriers to execution of the initiative. We collected over 8000 responses during a 1 week period.
Almost immediately after the feedback period closed we were able to start interpreting the results. As often is the case things that are perceived in one part of the organization as a strength were seen in another as a weakness. We can see how manager involvement and buy-in to the initiative reflected on the perception and even engagement of the initiative.
Responses typically were categorised with more than one category. By linking the data together in a co-occurence graph we were able to identify groups of topics most frequently mentioned together. We were able to link the two questions together and look at the linkages between both. We can also filter the scale questions by the topics or vice versa.
Getting such rich data on management challenges usually takes a consulting team considerable time. Workshops are expensive, and by their very nature only include a smallish number of people. Traditional surveys often don’t ask some of the really important questions (who would have thought engaged folks are most frustrated by the car park situation & therefore this was asked in the survey?). What the open text approach does is enable you to apply a qualitative approach in a wide, quantitative manner.
Of course executives love being able to quickly gauge the opinions of their employees, especially those at the coal-face whose thoughts are often filtered by layers of middle management.
What’s clear to us is that a conversational approach, asking open questions to the whole organization, summarising and communicating the results quickly encourages participation.
With several hundreds of thousands of rows of data captured and analysed in the last months I think we’re only just starting to understand how to get the most out of this approach. However, what we are seeing is the passion of employees to have their say on key aspects of how the firm treats not only its’ employees but the customers and broader community they operate in.
Being able to ask people in their own words is revealing the collective wisdom of the whole organization.
postscript: why should HR be involved?
One of the most powerful parts of this approach is using employee data ranging from demographics, performance data, job information and even business data to identify segments of employees who are more likely to answer in particular ways. In most organizations HR is the controller of such information.
What is important though is for HR to feel confident to move the conversation with employees from exclusively their perceptions of their jobs to a broader, hybrid approach which includes business topics. We know from comments that employees really value being listened to about their views on how the organization can better respond.
This post originally appeared here.