Saturday’s FT ran a piece by Ellen Kelleher about the rise of personal finance blogs. In it the former Wall Street analyst Henry Blodget was quoted: “The blogosphere functions the same way the stock market does–by incorporating millions of individual opinions into a general consensus. By itself, the influence of any one blogger is small, but if the ideas are persuasive, they will rapidly begin to influence the “blogosphere” as a whole.”

This is a profound remark. The blogosphere indeed functions as a marketplace in information, where spam takes the place of hype, and where a measurable consensus emerges around which companies have good products, and which ones are poor. Where a company’s employees, channel partners and customers spill the beans on how the company is doing 24 hours a day, 7 days a week. The blogosphere can be seen as “setting the price” of goods by forcing those with a bad reputation to discount in the search for buyers. Conversely those with a good reputation can charge a premium.

The interesting thing about this marketplace is that - unlike the stock exchange - the numbers are very hard to extract. You have to use social network analysis, natural language processing and statistical profiling to establish authority and sentiment. Having said that, these techniques exist (we and others are using them) and over time Wall Street and The City will track reputation indices as avidly as they track Standard and Poors ratings.

The phenomenon of consumer blogs having such a disproportionate influence on major brands, which we highlighted in our “Search is Brand” study, derives in part from the failure of brands to create functional, easily indexed sites for themselves, and to produce lively, relevant and topical content.

But some companies get over this by cheating. They do deals with unethical agencies who use spam and dummy websites to create phoney relevance and push a brand up to the top of a search term willy-nilly. We have been anxious for a while about this, as large brands have been using this technique not realising the risks (pace the BMW ban) or suffering from the fact that their competitors are doing so.

Ashley Friedlein, writing on e-Consultancy’s forums, points out that some black hat tactics appear to be working for some SEO-related consultancies. A company called Oyster-web seems to have got itself up to number 2 in the “Search engine optimisation” search, but its business seems to be selling leads to SEO companies.

At the same time Google appears to have woken up to this to the extent of going after first BMW and now some SEO companies to punish. It’s very surprising that they have picked a highly reputable agency like Scotland’s Bigmouthmedia to zero-rank. Bigmouth have some very respectable clients and it would be surprising if they have knowingly done anything unethical, but it is a chilling development.

Technorati have launched a mode button on their search which allows the user to sort by authority. They are trying to solve a problem that bedevils the consumer blog search companies: how to filter out spam results. If spammers are targetting keywords - last week we saw a blog targeting the christian name “Alan” - then live searches become gradually less and less effective. One approach is to say that if a blog has links back to it, it has authority. Increase the number of links and you increase the authority. The idea depends on links having a certain value.

This was the approach taken to internet search by Altavista around 1997-98. Altavista favoured sites with lots of links over sites with fewer. It worked for a while and made Altavista the most effective internet search engine. Then the spammers discovered that links could be spammed just as effectively as keywords and the Altavista approach failed. The problem is that a link does not have a universal value. A link from a spam site has less value than a link from the New York Times.

This is the weakness of the current Technorati approach. A full analysis of the weakness of the Technorati system can be found by our partner Flemming Madsen here.

Google’s BMW blacklisting is an affair with many ramifications.

We have noticed a huge increase in “black hat” SEO by a number of companies, although few as high profile as BMW. This gives us issues because it has a big effect on blog spam. Suddenly a humble keyword search on a customer’s brand is flooded with a blizzard of bogus sites. The sites look like this:

http://www.mownhose.com/

They have no archive, the content is nonsense and the sole function of the site is to spam a keyword.

I got this one by using the search term

“aluminium”

What is intriguing about this is that the link is misformatted.

It looks like this

http://www.mownhose.com/7//

and gives a 404 error.

I don’t entirely understand why this is so …

These companies are using techniques which are designed to give them authority in particular search strings. They are then selling this authority to … well, who knows, but probably someone in the Aluminium business.

The fact that companies as big as BMW, as Ricoh have been doing this shows that spammers have evolved from working for the gambling and porn industry into having a roster of “legit” customers. Work done by our company recently identified a highly respectable company as the beneficiary of a web of complicated links from affiliate schemes via invisible sites. Google say that they have a fix for the problem, but my guess is that they wouldn’t have chosen as big a beast as BMW to blacklist unless they wanted to fire a loud shot across the “black hat” SEOs’ bows.

Update:

Even weirder: here is someone trying to spam the name “Alan”

http://alanblog.morsalan.com/

An excellent article in Red Herring reporting on work by Umbria in Boulder, Colorado, draws attention to the increasing problems posed by spam blogs or splogs. Apparently spam bloggers have targeted 44 out of the top 100 brands.

This problem has vastly increased in severity since September/October 2005 when the blog spammers seemed to change their game. It is worth pointing out here that blog spam differs from email spam. Email spam is chiefly aimed at getting the recipient to visit a website, or engage in a transaction. Blog spam is partly about that, but to a greater measure it is about ensuring that the spam site appears in search results for a popular topic. It might also be useful, once it is has succeeded in gaining search engine authority on a particular topic, in lending that authority to a legitimate business which is trying to spoof its own way to greater search engine prominence. We will write more about this topic in our forthcoming white paper, a sequel to our “Measuring blogging influence”.

But back to the spam bloggers: previously they had tried to boost the traffic to their sites by posting direct links to gambling and poker sites. From mid-Autumn onwards we noticed increasingly that product names (cars, mobile phones, broadband suppliers) and common search terms were being systematically targeted, often in the context of material which might itself be returned in response to common searches.

This strategy suggests a return to the bad old days before Google, when search prominence could be (and was) spoofed. The retrospective results in Google are still reliable, but live results (and this business is all about live results) are polluted by junk returns.

The immediate problem from our perspective and the perspective of anyone in the corporate intelligence business was simple: how did we maintain the integrity of our live blog search results in the face of this issue? If you are searching on “product name” + keyword, and that product name is suddenly the target of spammers, you are screwed.

In the article the research group Umbria announces that it is pinning its hopes on a linguistic approach, hoping that the spammers will betray themselves by using characteristic patterns. It is an approach that is definitely worth trying. However, Market Sentinel’s researchers find that spam blogs are cunning at reusing genuine content, and that oftentimes you cannot identify that a blog is a spam blog until you have clicked through from the search return to the blog posting itself. This kind of disguise means that any algorithmic filtering is likely to be hard to implement. If a human being cannot spot a fake blog, what chance does a machine have?

For our customers the key question is: is this result important? Is it relevant to me? We have established that the best approach is to filter all our results by the writer’s relevance to a particular issue, using an algorithm developed by our partners at influence-specialists Onalytica to assess the writer’s influence on a particular issue, and then highlighting the most relevant returns in the results we provide the customer. This approach ensures that any spam-polluted result can be eliminated, saving time and server space. It also helps to create a more valuable monitoring service, since you are highlighting only those commentators who are authoritative.