Putting data to work

Leadership is a critical factor in the success of traditional analytics programs, and it’s just as important for big data. We don’t have a lot of examples of leaders of companies with substantial commitments to big data… However, we can begin to make some generalisations about the approaches followed by these relatively few big data leaders... One key leadership trait for big data seems to be a willingness to sponsor experimental activity with data on a large scale. Big data, at least today, requires some educated faith. ROI is difficult to define in advance — particularly when it involves new products and services or faster decisions. This is especially true in the discovery phase. 

There are some leaders who are willing to venture into big data on faith, however. At LinkedIn, for example, co-founder Reid Hoffman had also been a founder of PayPal, and knew there were substantial opportunities from exploiting online transaction data. It was primarily his decision to begin hiring data scientists into the product engineering organisation. He encouraged them not only to try to develop new products and services, but also to contact him directly if their ideas got stuck in the process or the hierarchy.

That’s exactly what Jonathan Goldman, a data scientist at LinkedIn whom Hoffman had helped recruit, did when he had an idea for a new application that became People You May Know (PYMK). The application recommends people you may want to network with who have background attributes in common with you. Goldman created an early prototype of PYMK, but had difficulty getting the product engineering organisation to incorporate it into the LinkedIn site... Hoffman allowed him to create a test ad on the LinkedIn site. 

The click-through rate on those ads was the highest ever seen. Goldman continued to refine how the suggestions were generated, incorporating networking ideas such as triangle closing — the notion that if you know Larry and Sue, there’s a good chance that Larry and Sue know each other. Goldman and his team also got the action required to respond to a suggestion down to one click.

LinkedIn’s top managers quickly made PYMK a standard feature. That’s when things really took off… PMYK messages achieved a click-through rate 30% higher than the rate obtained by other prompts to get people to return to the site. Millions of people paid repeat visits who would not have done so otherwise. Thanks to this feature, LinkedIn’s growth trajectory shifted significantly upward... It wouldn’t have happened without Goldman’s idea — and Hoffman’s support of it.

It won’t be necessary for data scientists to go directly to the company’s chairman, but it’s not a bad idea for senior executives to open a direct channel to them in the early days of the big data era. Part of taking an interest in experimentation is eliminating barriers to the implementation of innovative ideas and offerings.

Leaders of big data-intensive organisations also need some degree of patience. A good deal of “mucking around in data” may be necessary before there is any sense of a payoff. It may even be necessary to keep data around for multiple years before its value is known. Jeff Bezos of Amazon is known for saying, “We never throw away data,” simply because it is difficult to know when it may become important for a product or service down the road.