Every ethics and compliance conference I’ve been to for at least the last four years - including the conferences I’ve had a hand in running - has included Measuring Effectiveness as one of its major themes - and for good reasons:

• The Sentencing Guidelines and best practices call for periodic assessments;
• Boards and executives want to see empirical evidence of successes, failures, and trends as well as charts and graphs that help them understand the program and justify its expense and effort;
• Measuring your own program is a prerequisite to benchmarking with others; and, most importantly,
• We’ve all heard the maxim “you only get what you measure.”

This demand for metrics has driven more and more companies to rely on in-house surveys to gauge employee morale, assess culture and track opinions about ethics and compliance program elements. There’s no doubt that surveys are a useful tool, but are in-house surveys as reliable and accurate as we think?

Some of the problems with surveys are not new and are the result of poor question writing or faulty implementation. Leading questions are a common culprit: “Do you know that you can call our Helpline anonymously - Yes or No?” One can imagine survey-takers thinking, “Well I know it now. I’ll answer yes.” And how much of our data is skewed by “social desirability”. Most people like to present themselves in a favorable light, so their responses may be biased toward what they believe is socially desirable or, in this context, toward what they think is the most ethical answer.

Then there’s the problem of trust: “Is this survey really anonymous? Why are they asking so many demographic questions? Won’t they be able to identify me?” Like the problems above, these too can be addressed fairly easily. In this case simply ask if you really need all those demographic questions. Probably not, and consider what you gain by asking them and what you loose in terms of trust. In addition, it’s usually a good idea to have employees send their surveys to a third party for compilation. Trust is also an issue with on-line surveys. While they can be efficient and very cost effective, few believe they’re untraceable.

But what worries me more than these perennial survey problems is a new trend that seems to be occurring especially at companies that put the most emphasis on metrics. When ethics and compliance metrics are tied to performance reviews or when they are used to identify corporate problem sites, it doesn’t take long for everyone to realize what answers are most desirable. People know that a “wrong” answer will mean a manger’s bonus might be dinged or that poor numbers will result in remedial training or other unwanted attention from Corporate. In cases like these the pressure is strong to be sure to get the answers right. I’ve had employees tell me, “We don’t want to jam up our supervisor. When she’s happy, we’re happy…so we do what we can to help.”

It can get worse. I’ve seen examples of managers prepping their employees under the guise of training. And given the pressure, don’t be surprised when outright cheating occurs including manipulating the employee selection process to ensure that only those with “positive attitudes” are surveyed. Fortunately, in most cases actual cheating isn’t necessary because it’s safe to assume that everyone knows what answers are desirable. They’ll know what to do on their own.

Data-mining and consolidating survey results into a single overall number to be reported to the Board – another recent trend - only compounds the problems.

Most of these survey errors and problems tend toward a positive bias, and so the cumulative result can be greatly inflated data. Recently when conducting an assessment of a company’s corporate culture I ran into exactly this situation. Their in-house survey data was spectacularly good, but our focus groups told a very different story. Digging a little deeper, employees told us that so much emphasis was being placed on ethics and compliance metrics they had begun to “game” the system. The result was a false sense of optimism that went all the way up to the Board.

So what’s the solution? The quick answer is to avoid over-reliance on in-house surveys, perhaps by alternating annual surveys with focus groups. But there’s much more to it than that. We’ve scheduled sessions on the topic at this spring’s Conference Board meetings in San Diego and New York to explore the matter in more detail and I’ll have more to say about solutions in upcoming blogs. In the meantime, what are your thoughts? Have you seen problems like those I’ve mentioned? Do you have solutions that are working?