Typically, Ventana’s analytical methods employ large amounts of both quantitative and qualitative date. This is because most assignments involve serious attempts to validate the cause-and-effect represented in a given model – as well as the ranges of uncertainty that devolve from errors in both the input data and the model formulation. Historically, survey data has proven to be one of the more important qualitative or quasi-quantitative data sources. We put surveys in this category despite the common presentation of survey data as quantitative, i.e., average (and variability around) customer satisfaction scores, product ratings, perception of product/service attributes – as well as a host of different quantitative ways of describing what a sampled population thinks, estimates, feels or perceives about something. Likert scales are common, but other ordinal/metric measures are also common.

Ventana’s almost universal finding is that the proper interpretation of what survey respondents are trying to communicate is so hard to fathom that results are only interpretable in the context of models that also incorporate “hard” quantitative data. Indeed, in a few instances, the survey data has proven so unreliable that it is unclear that respondents were trying to convey accurate information. (This is one reason that personal interviews and focus group situations remain such an important part of Ventana’s consulting practice.)

While Ventana experience and conclusions about survey data are consistent with academic psychological research as well as research focusing on voter opinion/behavior, we find that many organizations with whom we work are consistently fooled by the survey results. One reason is that survey work is almost always done as an exercise that stands apart from other research, and little effort is devoted to corroborating or disproving survey findings. “They are what they are – let the buyer beware.” This is distinctly not Ventana’s approach, which requires more systemic understanding by using a variety of data types and models. As a consequence, in working with clients, we have discovered ways to revise the surveys, the sampling, and the analysis procedures to get more of the knowledge the clients are seeking, more reliably, for less money.

Customer satisfaction surveys

In analyzing customer surveys, Ventana’s approach is to build a small model connecting individuals’ detail responses to overall perceptions about a product or service, their stated purchasing intentions, and “hard data” on purchasing behavior. We have done this for both consumer and B2B purchasers. Typically, but not always, respondents show excellent internal consistency in the way they answer a battery of questions. Yet, the external validity proves to be terrible despite serious attempts to rescale, renormalize and/or reanalyze the data from various perspectives. This inconsistency between what respondents report and how they behave can be vexing. Examples abound. Purchase managers give a vendor a perfect score and then give most of their business to a different supplier; purchase managers improve their score for a vendor by a factor of two, but they decrease their business volume with the vendor and give it to a company with a declining score. Hotel guests rate hotels at the low end of their scale on all dimensions. Yet, they continue to frequent them; or conversely, they state their intention to never visit a hotel again – but do so within the next quarter. Large consumer panels can rate products low that have dominant market shares and vice versa.

Ventana shows why common procedures for interpreting surveys give wrong answers for some types of data, and provide more reliable guidelines. Properly analyzed, the survey results show strong evidence that common survey formats, while effective for examining details of the customer experience or measuring the penetration of marketing messages, are poor instruments for determining the key drivers of customer choice. Understanding the causes of customer behavior requires both a dedicated and markedly different surveying scheme, and an analytical approach that triangulates survey data against other kinds of data.

Physician surveys (use of “expert” opinion)

Ventana has worked with more of these surveys than any other kind over the last decade. The survey issues are basically the same as customer surveys, but a physician is generally considered a medical “expert”, and surveys delve into a variety of very technical topics. Yet, like other “customer” surveys, they manifest vexing interpretation problems. Even their internal consistency can be poor. The syndicators that perform many of the widely distributed survey results (as opposed to the primary research of individual pharmaceutical companies) often provide what they call a “derived satisfaction” analysis in addition to the actual respondents’ data. While provided as a kind of “bonus” supplement, more often than not, this analysis essentially shows that physicians’ stated overall satisfaction with a given product is inconsistent with their detailed responses to questions about individual product attributes.

Far more important, however, are the external validity issues with surveys. When survey responses for individual physicians are compared with their actual prescribing behavior there can be startling discrepancies. In one instance, Ventana discovered that physicians did not even accurately report the medication they prescribed. Indeed, the discrepancies were so great that the linking between individual physician responses and their prescribing records had to be triple-checked, because no one could believe the magnitude of the differences. Physicians are busy people and syndicated surveys tend to be long and detailed. And even though respondents are paid for their answers, inattention and memory dynamics influence how questions are answered. Ventana must account for these influences in its simulation models in order to explain market behavior.

More important interpretation issues emerge from the detailed technical content of questions and responses. The ordering of questions and very subtle differences in language affect responses. Moreover, physician motivation, legal issues, and the obvious “test taking” acumen of physician candidates influence the way questions are answered. For example, consider physicians’ use of narcotics. These are very commonly prescribed drugs, but trying to figure out when and why physicians use them using a survey instrument is hard. Public policy, legal and popularity with patients converge to make straightforward responses problematic for the honest physician.

Employee surveys

One key finding is that despite organizational pressures to conduct only a few “special” surveys, companies are better served by using more frequent surveys and controlling the cost of this practice by scientifically sampling employees for each survey. There are several reasons for surveying employees, and not all reasons apply to all surveys, but in Ventana’s experience at least one of them will apply in a given circumstance. For example, in trying to assess the impact of morale on an aerospace company’s manufacturing productivity, we found intermittent to annual surveying to provide no useful information about morale’s impact. In the rare cases where monthly or quarterly surveys were available, however, we could resolve the impact. The reason was that with the more frequent measurement we could distinguish the difference between the longer term trends in feelings and the ephemeral gripes and joys of a given day or week.