To get a better understanding of what constitutes best practice in social media measurement and evaluation, we thought the simplest thing to do was to “eat our own dog food” and use our own social media measurement tools to determine who had influence on the topic of “social media measurement” and then look in detail at what they had to say. We wanted to know who owned the consensus around the topic.
First we used Market Sentinel’s proprietary web crawler to undertake a MarketInfluence study. In this case, a “stakeholder” is any individual or organisation who is mentioned in the context of “social media measurement”. Our web crawler finds these stakeholders by following a three step process:
1) Search the web and identify documents (web pages, Word, pdf or PowerPoint documents) that reference “social media measurement”
2) Search those documents for references to other documents
3) Visit the referenced document and determine whom they reference
The analysis is focused on identifying who references whom. The process continues until the context is lost. When all possible citations have been found we perform a “citation analysis” to determine who in the conversation is
a) most popular (i.e. most linked in the context)
b) most influential (i.e. most linked by those who themselves are most linked)
c) more central to the network
Citaton analysis has been used since the sixties to calculate the authority of academic journals. Here, we use it to analyse web documents for references. For example, a document by Seth Godin refers to a document by BBC News in the context of “social media measurement”, then we take it that Seth Godin deems BBC News to be relevant to the issue. It also means that, on average, BBC News to some extent influences Seth Godin. References like these are turned into systems of mathematical equations which we can solve to determine “popularity”, “influence” and centrality.
Our influence metric works much like that of academic journals, where academic authors cite the works of other academics. They mainly do this because they believe that the articles they cite are relevant to the context. In doing so, they point to other publications that are relevant to their topic thereby revealing which publications have influenced them. We use a similar approach to measure influence in social media measurement.
At the heart of this type of influence measurement is a simple idea: Person X has influence on Person Y regarding a particular issue if Person Y is dependent on Person X for information or to support point of view about the issue. For example, BBC News influences Seth Godin regarding “social media measurement” if Seth Godin depends on BBC News for information. Seth Godin may demonstrate this dependency by linking to a BBC News article in a blog post. Our influence measurement works by searching relevant documents for such links and transforming these relationships into a system of equations which we then solve as a relative measure of influence.
What emerges from this process is a league table of influencers, which might be compared to Google’s pagerank, except that it is topic-specific, a network map showing how these speakers or entities are linked, and various mathematical measures showing how their status in the network can be determined.
The value of this procedure, which Market Sentinel has used many times for its large brand customers, is to achieve a God-like understanding of any given conversation, who is involved in it and how it is articulated, and to understand where the major themes in that conversation spring from and can therefore be influenced.
Here is the leage table of influencers, showing who is most authoritative on the topic of social media measurement/evaluation. [Update in response to Katie Paine's comment] The analysis was completed during August 2008 and updated earlier in October.
In calculating influence, we obtain a “virtual social network” which shows how the top influencers influence eachother. The map helps us determine if any stakeholder groups cluster together in interesting ways.
(Click for full-size network. Please note that the figure does not show all the stakeholders, but just a very detailed close-up of the centre of the influence network. If all stakeholders were included it would be virtually impossible to identify individual stakeholders.)
The size of each stakeholder’s marker indicates their relative influence.
The network maps are constructed using an approach called “minimum spring force layout”. Imagine the relations between the nodes are springs and that the stiffness of each spring is proportional to the strength of the relation. The nodes are then arranged so the total force used to “stretch” all springs is at a minimum.
The main advantage of the network map is that it clusters stakeholders based on the strength of their influence relations. The closer two stakeholders appear on this map the more they influence each other, even if that influence is indirect via one or more other stakeholders. This gives an idea of how closely related individual stakeholders are.
So what’s the story here? Google is here because of its omnipresence in matters internet-related. It has also recently applied for a patent for applying a version of pagerank within social networks. YouTube, Facebook, Technorati and Yahoo! are prominent for similar reasons: a combination of mindshare in the context (particularly true of Facebook, where the lack of metrics is a major issue) and of relevant content.
The story thereafter is of a conversation dominated by Forrester, whose Charlene Li (she left in 2008, but has been consolidated in this data) has been extremely important in atomising the debate around the development of social media and whose “Groundswell” encapsulates many of the great cases studies of the last 3-4 years. Her co-author on the book Josh Bernoff maintains the Groundswell blog. Next is Jeremiah Owyang. He has not been consolidated with Forrester because he was part of this conversation in his own right before he joined them. Jen McClure’s Society for New Communications Research has – equally – published a number of great resources on the topic. Katie Paine is a guru of PR measurement in all its flavours and so finding her here is no surprise.
Jeff Jarvis of BuzzMachine and Robert Scoble are here in part because of their relevance in two of the stories that defined the era which witnessed the arrival of social media: Dell Hell (the story of Dell’s failure to comprehend social media and then their enthusiastic conversion to it) and in Scoble’s case the opening up of Microsoft to the outside world (symbolised by Scoble’s work) and Scoble’s own subsequent career as a “star” of social media in his own right. Scoble’s co-author Shel Israel’s Global Neighbourhoods follows hard on Scoble’s heels.
The presence of Seth Godin (marketing guru) and of Steve Rubel (PR guru) demonstrates the twin poles around which the area of social media evaluation revolves. Marketing folk need to understand how people have responded to their campaigns, public relations folk want to understand key conversations and influencers with whom they can connect.
Actually, they have interests that are increasingly identical and interchangeable, but marketing and PR stubbornly refuse to converge, partly because they employ such different methods. Marketing people create assets (campaigns, websites, virals, creative) and PR people create conversations, person to person engagement. The pressure of social media’s rise is forcing the two disciplines to borrow increasingly from one another’s toolkit, but they are not yet in the same world.
As we examine the measurement methodologies each apply it will be interesting to establish how much they are beginning to ask the same questions of the data. How many responses received? How many conversations started? What is my awareness level? How many products have I sold?
An interesting sidelight on this story is provided by a glance at the list of the top ten connectors. This group is defined by their characteristic of disproportionately citation of the most authoritative sources.
1. Jeremiah Owyang
2. Constantin Basturea
3. Josh Hallett’s Blog
4. Livingston Communications
5. Forrester
6. Peter Kim
7. Kami Watson Huyse’s Blog
8. Shel Holtz
9. Market Sentinel
10. Nathan Gilliatt
These folk offer a kind of directory of quality sources in this context and I suppose this blog post continues Market Sentinel’s work in this area.
In our next posts, we will look in detail at the ideas of individual stakeholders about how social media should be measured and see if we can define a growing consensus.
Interesting analysis, thanks for the transparency and insight into your methodology. I think, however, that you need to tell us the time frame. I’m guessing it goes back a year or two, since it’s been awhile since Charlene Li put forth her now-discredited AVE approach to social media measurement. Given how fast things are changing in this space, I wonder how different the network would look if you only included the last 6 months.
Thank you for the comment Katie – I have updated the study to clarify the timeframe. It would certainly be worth running the topic again in a few weeks to see the changes. Charlene Li and Josh Bernoff have benefited hugely from the attention they have paid to this area over the last few years and from the detail of the case studies they explore in “Groundswell”