The following story just appeared in The Economist:

Internet trends: Companies are eavesdropping on online discussion forums to
find out what their customers really think about them

ONE of the things that makes the internet so appealing is that for any
subject, no matter how obscure, there is almost guaranteed to be at least
one website, blog or discussion forum where people congregate to talk about
it. Online discussion forums cover a huge range of topics: it is not just
stereotypical geeky bickering about Macs versus PCs, or Windows versus
Linux. Worried mothers compare the fat content of different brands of potato
crisps; car enthusiasts discuss the merits of forthcoming models; fans of
obscure bands swap trivia.

The internet’s oldest discussion system, called Usenet, dates back to 1979
and can now be easily reached via Google, which also maintains archived
discussions going back 25 years. More recently, web-based discussion boards,
and the comments that can be appended to blogs, have become the most popular
forums for online debate. Most of these discussions are of little interest
except to their participants. But the direct, unfiltered, brutally honest
nature of much online discussion is gold dust to big companies that want to
spot trends, or find out what customers really think of them. As a result,
many firms now monitor online chatter as an adjunct to more traditional
forms of market research.

For example, ConAgra, an American food giant known for its Butterball
turkeys and Healthy Choice ready meals, tracks discussion groups to keep
abreast of new diet trends, such as Atkins and organic food. “These
discussion groups are very useful to determine whether a trend is really a
trend, or just a fad,” says Nick Mysore, ConAgra’s director of strategy.
“They also help you get a read on the marketplace quickly and cost
effectively, and provide you with the ‘baseline hypothesis’ that you can
test further, using conventional market-research techniques.”

This sort of thing has been going on, in an informal way at least, for
several years. Sometimes public-relations teams are asked to keep an eye on
online discussions, and switched-on company executives visit forums to stay
in touch with the public mood. But companies are now looking to do it in a
more systematic way. Initially, this involved throwing lots of computing
power at the problem. Accenture and IBM have each built computer systems
that trawl the web in search of trends and insights. But the extensive use
of jargon, abbreviations and obscure slang can make it hard for computers to
figure out what people are saying.

So smaller specialist firms, such as Wavemetrix in Britain and Nielsen
BuzzMetrics in America, are now using a combination of computers and human
researchers. They track general discussions and try to spot trends early, by
identifying the members of online communities who are most likely to
influence other participants. BuzzMetrics does this kind of work on behalf
of 100 of the world’s biggest firms, including General Motors, Ford and
Microsoft.

Such research has a number of weaknesses, not least the fact that it
excludes non-internet users, so it is not about to replace telephone surveys
and focus groups any time soon. But it also has many advantages, notably its
high speed and low cost. Opinions appear on the web within minutes of an
event. Convening a focus group or performing traditional market research to
gauge the impact of a campaign, in contrast, might take weeks or even
months.

The results can often be surprising. Wavemetrix, for example, carried out a
study for a European mobile operator that was aware that its network was
less reliable than those of its rivals. Online research, however, showed
that the general public believed the opposite to be true. Other surprising
findings arise because participants in discussion groups can say anything
they like, whereas people answering a survey answer only the questions that
researchers think to ask.

What of privacy? Max Kalehoff of BuzzMetrics insists that “openness and
transparency are the most important things in our industry.” Even so, many
companies are reluctant to discuss their activities in this area. Although
firms track only public forums, they do not obtain the participants’ formal
consent, and many users may be disconcerted to learn that their
conversations are being listened to. That said, people who post their
thoughts online generally want them to be read. While the ethics of
monitoring public discussion boards are a matter of debate, there is general
agreement that the active abuse of forums-in particular, posting poorly
disguised product plugs-is unacceptable.

Occasionally, the monitoring of discussion groups itself becomes a topic of
conversation. In one car forum, a discussion of BuzzMetrics’ research for
General Motors produced no objections-just disbelief that the carmaker could
listen to their conversations and still produce such unappealing products.
Consumers often moan that companies do not listen to them. Might the
monitoring of discussion groups provide an answer to that problem? Discuss.