Advance trend catches on! Gartner says Asia’s big data sparks rush into Business Intelligence

20110905-head

Mounting Big Data in Asia, is sparking a rush to analytic, like Business Intelligence in Asia!

In Asia, the middle class is getting larger, at a very fast pace. Globally, between 1990 and 2005, more than 1 billion people worldwide entered the middle class, according to Wikipedia. That means more and more people who gain money will become more literate which in turn leads to information growth.

But at Harvard Business Review, the old question of human vs machine is highlighted, with some sound advice, how human input, into understanding of big data, can never be replaced.

  • Gartner, the research unit, says:

“THIS year’s buzzword – big data – had a profound effect on Asia’s enterprise IT spending choices, industry watchers say. Ian Bertram, Gartner’s managing vice-president, said data analytics was the top priority for companies in 2012, as firms sat up and paid attention to making sense of the growing storehouse of data generated by PCs and mobile devices belonging to workers.

Mobile devices are undoubtedly adding to the cacophony, but much of the data is simply coming from more work done digitally, as well as more modes of communication online internally, as well as with customers outside.

All of this bodes well for business intelligence and analytics vendors, which promise to help companies extract value from all the noise.

And next year, they’ll be floated by an increase in IT spending in the region, as well. Gartner said next year’s IT budgets in Asia Pacific alone are expected to grow 7.9 per cent by the end of 2013 to reach US$743 billion.”

  • According to Wikipedia, there are a number of business intelligence vendors, often categorized into the remaining independent “pure-play” vendors and consolidated “megavendors” that have entered the market through a recent trend[when?] of acquisitions in the BI industry.[20] Some companies adopting BI software decide to pick and choose from different product offerings (best-of-breed) rather than purchase one comprehensive integrated solution (full-service).[21]

Specific considerations for business intelligence systems have to be taken in some sectors such as governmental banking regulations. The information collected by banking institutions and analyzed with BI software must be protected from some groups or individuals, while being fully available to other groups or individuals. Therefore BI solutions must be sensitive to those needs and be flexible enough to adapt to new regulations and changes to existing law.

  • Wikipedia on Business Intelligence:

Business Intelligence is the ability of an organization to collect, maintain, and organize data. This produces large amounts of information that can help develop new opportunities. Identifying these opportunities, and implementing an effective strategy, can provide a competitive market advantage and long-term stability.[1]

BI technologies provide historical, current and predictive views of business operations. Common functions of business intelligence technologies are reporting, online analytical processing,analyticsdata miningprocess miningcomplex event processingbusiness performance managementbenchmarkingtext miningpredictive analytics and prescriptive analytics.

The goal of modern business intelligence deployments is to support better business decision-making. Thus a BI system can be called a decision support system (DSS).[2]Though the term business intelligence is sometimes a synonym for competitive intelligence (because they both support decision making), BI uses technologies, processes, and applications to analyze mostly internal, structured data and business processes while competitive intelligence gathers, analyzes and disseminates information with a topical focus on company competitors. If understood broadly, business intelligence can include the subset of competitive intelligence.[3]

  • Wikipedia on Big Data: 

“Big data” has increased the demand of information management specialists in that Software AGOracle CorporationIBMMicrosoftSAP, and HP have spent more than $15 billion on software firms only specializing in data management and analytics. This industry on its own is worth more than $100 billion and growing at almost 10 percent a year, about twice as fast as the software business as a whole.[5]

Developed economies make increasing use of data-intensive technologies. There are 4.6 billion mobile-phone subscriptions worldwide and there are between 1 billion and 2 billion people accessing the internet.[5] Between 1990 and 2005, more than 1 billion people worldwide entered the middle class which means more and more people who gain money will become more literate which in turn leads to information growth. The world’s effective capacity to exchange information through telecommunication networks was 281 petabytes in 1986, 471 petabytes in 1993, 2.2 exabytes in 2000, 65 exabytes in 2007[15] and it is predicted that the amount of traffic flowing over the internet will reach 667 exabytes annually by 2013.[5]

In information technologybig data[1][2] is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage,[3] search, sharing, analysis,[4] and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to “spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.”[5][6][7]

As of 2012, limits on the size of data sets that are feasible to process in a reasonable amount of time were on the order of exabytes of data.[8][9] Scientists regularly encounter limitations due to large data sets in many areas, including meteorologygenomics,[10] connectomics, complex physics simulations,[11] and biological and environmental research.[12] The limitations also affect Internet searchfinance andbusiness informatics. Data sets grow in size in part because they are increasingly being gathered by ubiquitous information-sensing mobile devices, aerial sensory technologies (remote sensing), software logs, cameras, microphones, radio-frequency identification readers, and wireless sensor networks.[13][14] The world’s technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s;[15] as of 2012, every day 2.5 quintillion (2.5×1018) bytes of data were created.[16]

Big data is difficult to work with using relational databases and desktop statistics and visualization packages, requiring instead “massively parallel software running on tens, hundreds, or even thousands of servers”.[17] What is considered “big data” varies depending on the capabilities of the organization managing the set, and on the capabilities of the applications that are traditionally used to process and analyze the data set in its domain. “For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration.”[18]

  • Harvard Business Review gives a word of caution:

Big data, it has been said, is making science obsolete. No longer do we need theories of genetics or linguistics or sociology, Wired editor Chris Anderson wrote in a manifesto four years ago: “With enough data, the numbers speak for themselves.”

Last year, at the Techonomy conference outside Tucson, I heard Vivek Ranadivé — founder and CEO of financial-data software provider TIBCO, subject of a Malcolm Gladwell article on how to win at girls’ basketball, and part owner of the Golden State Warriors — say pretty much the same thing:

I believe that math is trumping science. What I mean by that is you don’t really have to know why, you just have to know that if a and b happen, c will happen.

Anderson and Ranadivé are reacting to something real. If the scientific method is to observe, hypothesize, test, and analyze, the explosion of available data and computing power have made observation, testing, and analysis so cheap and easy in many fields that one can test far more hypotheses than was previously possible. Quick-and-dirty online “A/B tests,” in which companies like Google and Amazon show different offers or page layouts to different people and simply go with the approach that gets the best response, are becoming an established way of doing business.

But that does that really mean there are no hypotheses involved? At Techonomy, Ranadivé made his math-is-trumping-science comments after recommending that the Federal Open Market Committee, which sets monetary policy in the U.S., be replaced with a computer program. Said he:

The fact is, you can look at information in real time, and you can make minute adjustments, and you can build a closed-loop system, where you continuously change and adjust, and you make no mistakes, because you’re picking up signals all the time, and you can adjust.

As best I can tell, there are three hypotheses inherent in this replace-the-Fed-with-algorithms-plan. The first is that you can build U.S. monetary policy into a closed-loop system, the second is that past correlations in economic and financial data can usually be counted on to hold up in the future, and the third is that when they don’t you’ll always be able to make adjustments as new information becomes available.

These feel like pretty dubious hypotheses to me, similar to the naive assumptions of financial modelers at ratings agencies and elsewhere that helped bring on the financial crisis of 2007 and 2008. (To be fair, Ranadivé is a bit more nuanced about this stuff in print.) But the bigger point is that they are hypotheses. And since they’d probably prove awfully expensive to test, they’ll presumably stay hypotheses for a while.

There are echoes here of a centuries-old debate, unleashed in the 1600s by protoscientist Sir Francis Bacon, over whether deduction from first principles or induction from observed reality is the best way to get at truth. In the 1930s, philosopher Karl Popper proposed a synthesis, in which the only scientific approach was to formulate hypotheses (using deduction, induction, or both) that were falsifiable. That is, they generated predictions that — if they failed to pan out — disproved the hypothesis.

Actual scientific practice is more complicated than that. But the element of hypothesis/prediction remains important, not just to science but to the pursuit of knowledge in general. We humans are quite capable of coming up with stories to explain just about anything after the fact. It’s only by trying to come up with our stories beforehand, then testing them, that we can reliably learn the lessons of our experiences — and our data. In the big-data era, those hypotheses can often be bare-bones and fleeting, but they’re still always there, whether we acknowledge them or not.

“The numbers have no way of speaking for themselves,” political forecaster Nate Silver writes, in response to Chris Anderson, near the beginning of his wonderful new doorstopper of a book, The Signal and the Noise: Why So Many Predictions Fail — But Some Don’t. “We speak for them.” He continues:

Data-driven predictions can succeed — and they can fail. It is when we deny our role in the process that the odds of failure rise. Before we demand more of our data, we need to demand more of ourselves.

One key role we play in the process is choosing which data to look at. That this choice is often made for us by what happens to be easiest to measure doesn’t make it any less consequential, as Samuel Arbesman wrote in Sunday’s Boston Globe (warning: paywall):

Throughout history, in one field after another, science has made huge progress in precisely the areas where we can measure things — and lagged where we can’t.

In his book, Silver spends a lot of time on another crucial element, how we go about revising our views as new data comes in. Silver is a big believer in the Bayesian approach to probability, in which we all have our own subjective ideas about how things are going to pan out, but follow the same straightforward rules in revising those assessments as we get new information. It’s a process that uses data to refine our thinking. But it doesn’t work without some thinking first.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s