Analytics

Analytics
A brief
overview

An Introduction
• Big data analytics is where advanced analytic techniques operate on big data
sets. Hence, big data analytics is really about two things plus how the two
have teamed up to create one of the most profound trends in business
intelligence (BI) today.
• Analytics helps us discover what has changed and how we should react as
well as this is the best way to discover new customer segments, identify the
best suppliers, associate products of affinity, understand sales seasonality.
• . To help user organizations select the right form of analytics and prepare big
data for analysis, this report will discuss new options for advanced analytics
and analytic databases for big data so that users can make intelligent
decisions as they embrace analytics.
• This is a collection of related techniques and tool types, usually including
predictive analytics, data mining, statistical analysis, and complex SQL.
• The three Vs of big data (volume, variety, and velocity) constitute a
comprehensive definition, and they bust the myth that big data is only about
data volume. In addition, each of the three Vs has its own ramifications for
analytics.
• It’s obvious that data volume is the primary attribute of big data. With that
in mind, most people define big data in terabytes

• It’s obvious that data volume is the primary attribute of big data. With that
in mind, most people define big data in terabytes. . Some organizations find
it more useful to quantify big data in terms of time.
• One of the things that make big data really big is that it’s coming from a
greater variety of sources than ever before. Many of the newer ones are Web
sources, including logs, clickstreams, and social media.
• The few organizations that have been analyzing this data now do so at a
more complex and sophisticated level. Big data isn’t new, but the effective
analytical leveraging of big data is. The recent tapping of these sources for
analytics means that so-called structured data is now joined by unstructured
data and semi-structured data.
• , with big data, variety is just as big as volume. In addition, variety and
volume tend to fuel each other. Big data can be described by its velocity or
speed.
• . For example, think of the stream of data coming off of any kind of device or
sensor, say robotic manufacturing machines, thermometers sensing
temperature, microphones listening for movement in a secure area, or video
cameras scanning for a specific face in a crowd.
• Web sites for years, using streaming data to make purchase
recommendations to Web visitors. With sensor and Web data flying at you
relentlessly in real time, data volumes get big in a hurry

Defining Big Data Analytics

Why Put Big Data and Analytics Together Now?
• Big data provides gigantic statistical samples, which enhance analytic tool
results. Most tools designed for data mining or statistical analysis tend to be
optimized for large data sets.
• The general rule is that the larger the data sample, the more accurate are the
statistics and other products of the analysis. Instead of using mining and
statistical tools, many users generate or hand-code complex SQL, which
parses big data in search of just the right customer segment, churn profile, or
excessive operational cost.
• The economics of analytics is now more embraceable than ever. This is due to
a precipitous drop in the cost of data storage and processing bandwidth. The
fact that tools and platforms for big data analytics are relatively affordable is
significant because big data is not just for big business.
• . The preparation of big data for advanced analytics rarely follows the same
best practices we associate with mainstream data warehousing, reporting,
and OLAP. Even more compelling, however, is the prospect of discovering
problems that need fixing and opportunities that merit leverage.
• . Many small-to-midsize businesses (especially those deep into digital
processes for sales, customer interactions, or supply chain) also need to
manage and leverage big data

The State of Big Data Analytics

• Big data is an all-encompassing term for any collection of data sets so large
and complex that it becomes difficult to process using on-hand data
management tools or traditional data processing applications
• The trend to larger data sets is due to the additional information derivable
from analysis of a single large set of related data, as compared to separate
smaller sets with the same total amount of data, allowing correlations to be
found to "spot business trends, prevent diseases, combat crime and so on."
• The primary goal of big data analytics is to help companies make better
business decisions by enabling data scientists and other users to analyze
huge volumes of transaction data as well as other data sources that may be
left untapped by conventional business intelligence (BI) programs.
• Big data analytics can be done with the software tools commonly used as
part of advanced analytics disciplines such as predictive analytics and data
mining.
• The technologies associated with big data analytics include NoSQL
databases, Hadoop and MapReduce
• Predictive analytics uses techniques like simulation, statistics, and machine
learning to extrapolate from past data or behavior to predict what might
happen. Variations might be introduced so that you can get an idea of future
results if you increase your sales force by 10%, decrease your price by 5%, or
increase your manufacturing capacity.

• Prescriptive analytics often uses serious mathematical optimization
techniques, simulation, and algorithms to help you understand how you
should reach your goals
• Scientists regularly encounter limitations due to large data sets in many
areas, including meteorology, genomics, connectomics, complex physics
simulations, and biological and environmental research. The limitations also
affect Internet search, finance and business informatics.
• Big data can also be defined as "Big data is a large volume unstructured data
which cannot be handled by standard database management systems like
DBMS, RDBMS or ORDBMS".

Big science
• The Large Hadron Collider experiments represent about 150 million sensors
delivering data 40 million times per second. There are nearly 600 million
collisions per second. After filtering and refraining from recording more
than 99.999% of these streams, there are 100 collisions of interest per second.
• As a result, only working with less than 0.001% of the sensor stream data,
the data flow from all four LHC experiments represents 25 petabytes annual
rate before replication (as of 2012). This becomes nearly 200 petabytes after
replication.

Science and research
• When the Sloan Digital Sky Survey (SDSS) began collecting astronomical
data in 2000, it amassed more in its first few weeks than all data collected in
the history of astronomy. Continuing at a rate of about 200 GB per night,
SDSS has amassed more than 140 terabytes of information
• The NASA Center for Climate Simulation (NCCS) stores 32 petabytes of
climate observations and simulations on the Discover supercomputing
cluster
• Decoding the human genome originally took 10 years to process, now it can
be achieved in less than a day: the DNA sequencers have divided the
sequencing cost by 10,000 in the last ten years, which is 100 times cheaper
than the reduction in cost predicted by Moore's Law.

International development
• Research on the effective usage of information and communication
technologies for development suggests that big data technology can make
important contributions but also present unique challenges to International
development.

• Advancements in big data analysis offer cost-effective opportunities to
improve decision-making in critical development areas such as health care,
employment, economic productivity, crime, security, and natural disaster
and resource management

Market
• Big data has increased the demand of information management specialists in that
Software AG, Oracle Corporation, IBM, FICO, Microsoft, SAP, EMC, HP and Dell
have spent more than $15 billion on software firms only specializing in data
management and analytics
• In 2010, this industry on its own was worth more than $100 billion and was growing
at almost 10 percent a year: about twice as fast as the software business as a whole.
• Developed economies make increasing use of data-intensive technologies. There are
4.6 billion mobile-phone subscriptions worldwide and there are between 1 billion and
2 billion people accessing the internet
• The world's effective capacity to exchange information through telecommunication
networks was 281 petabytes in 1986, 471 petabytes in 1993, 2.2 exabytes in 2000, 65
exabytes in 2007 and it is predicted that the amount of traffic flowing over the internet
will reach 667 exabytes annually by 2014.

Technologies
• Big data requires exceptional technologies to efficiently process large
quantities of data within tolerable elapsed times.
• A 2011 McKinsey report suggests suitable technologies include A/B testing,
crowdsourcing, data fusion and integration, genetic algorithms, machine
learning, natural language processing, signal processing, simulation, time
series analysis and visualisation.
• The practitioners of big data analytics processes are generally hostile to
slower shared storage, preferring direct-attached storage (DAS) in its various
forms from solid state drive (SSD) to high capacity SATA disk buried inside
parallel processing nodes

Big Data and CSPs
• Communications service providers (CSPs) know an inordinate amount of
personal information about their customers, such as who their contacts are
and their phone numbers, addresses (home, work, email), Internet usage,
applications downloaded, travel history, even how long it takes them to
commute to work each morning. A customer’s smartphone usage becomes a
snapshot of their daily lives—data that any social media company would
love to possess.

Need for Big Data
• Giant companies like Amazon and Wal-Mart as well as bodies such as the
U.S. government and NASA are using Big Data to meet their business
and/or strategic objectives.
• The days of keeping company data in Microsoft Office documents on
carefully organized file shares are behind us, much like the bygone era of
sailing across the ocean in tiny ships. That 50 gigabyte file share in 2002
looks quite tiny compared to a modern-day 50 terabyte marketing database
containing customer preferences and habits
• Interpretation of Big Data can bring about insights which might not be
immediately visible or which would be impossible to find using traditional
methods
• Apache Hadoop is one such technology, and it is generally the software most
commonly associated with Big Data. Apache calls it "a framework that
allows for the distributed processing of large data sets across clusters of
computers using simple programming models."
• The use of Big Data is becoming a crucial way for leading companies to
outperform their peers. In most industries, established competitors and new
entrants alike will leverage data-driven strategies to innovate, compete, and
capture value.

Five Ways to Leverage Big Data
• Big Data can unlock significant value by making information transparent.
There is still a significant amount of information that is not yet captured in
digital form, e.g., data that are on paper, or not made easily accessible and
searchable through networks. We found that up to 25 percent of the effort in
some knowledge worker workgroups consists of searching for data and then
transferring them to another (sometimes virtual) location. This effort
represents a significant source of inefficiency.
• As organisations create and store more transactional data in digital form,
they can collect more accurate and detailed performance information on
everything from product inventories to sick days and therefore expose
variability and boost performance. In fact, some leading companies are using
their ability to collect and analyse big data to conduct controlled experiments
to make better management decisions
• Big Data allows ever-narrower segmentation of customers and therefore
much more precisely tailored products or services. Sophisticated analytics
can substantially improve decision-making, minimise risks, and unearth
valuable insights that would otherwise remain hidden
• Big Data can be used to develop the next generation of products and services

Value Created By The Use Of Big Data
• If the U.S. healthcare system were to use big data creatively and effectively to
drive efficiency and quality, the sector could create more than $300bn in
value every year
• In the developed economies of Europe, government administrators could
create more than €100bn ($123bn) in operational efficiency improvements
alone by using Big Data.
• Some of the most significant potential to generate value from Big Data will
come from combining separate pools of data
• The era of Big Data could yield new management principles. In the early
days of professionalized corporate management, leaders discovered that
minimum efficient scale was a key determinant of competitive success.
Likewise, future competitive benefits are likely to accrue to companies that
can not only capture more and better data but also use that data effectively at
scale
• The era of Big Data could yield new management principles. In the early
days of professionalized corporate management, leaders discovered that
minimum efficient scale was a key determinant of competitive success.
Likewise, future competitive benefits are likely to accrue to companies that
can not only capture more and better data but also use that data effectively at
scale

Results of a Research by MGI on Big Data
MGI studied big data in five domains—healthcare in the United States, the
public sector in Europe, retail in the United States, and manufacturing and
personal-location data globally. The research offers seven key insights.
• Data have swept into every industry and business function and are now an
important factor of production, alongside labor and capital. We estimate
that, by 2009, nearly all sectors in the US economy had at least an average of
200 terabytes of stored data (twice the size of US retailer Wal-Mart's data
warehouse in 1999) per company with more than 1,000 employees.
• The use of big data will become a key basis of competition and growth for
individual firms. From the standpoint of competitiveness and the potential
capture of value, all companies need to take big data seriously. In most
industries, established competitors and new entrants alike will leverage
data-driven strategies to innovate, compete, and capture value from deep
and up-to-real-time information.
• There will be a shortage of talent necessary for organizations to take
advantage of big data. By 2018, the United States alone could face a shortage
of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million
managers and analysts with the know-how to use the analysis of big data to
make effective decisions

Need of Big Data and Analytics Strategy
• Creating Smarter, Leaner Organizations.
• Equipping Your Organization to Have Cross-Channel Conversations.
• Preparing Your Organization for the Inevitable Future.

15 Important Big Data Facts for IT Professionals
•
•
•
•
•
•
•
•
•
•
•
•
•
•

How Much Data is There?
Structured Versus Unstructured Data.
Big Data Generates Jobs
The Big Data Talent Shortage
Rethinking Job Roles and Titles
Disparate Systems
Getting Business Value from Big Data.
Data Quality
Create a Stronger Business
Better Manage Data
Top 3 Big Data Business Drivers.
Big Data Implementations
Big Data Tools
Big Data Spending

Drivers

So what drivers make businesses tick?
• Data Driven Innovation, having the ability to drive innovation through those
uber targeted data indicators.
• Data Driven Decision Making, Data driven decision-making is the inherent
ability of analytics to sieve through globs of data and identify the best path
forward
• Data Driven Discovery, Having a discovery mechanism will help you
understand hidden insights that were not visible through traditional means.
• Data Science as a competitive advantage, With a proper data driven
framework, businesses could build sustainable capabilities and further
leverage these capabilities as a competitive edge
• Sustained processesData driven approach creates sustainable processes,
which gives a huge endorsement to big data analytics strategy as a go for
enterprise adoption
• Cost advantages of commodity hardware & open source software. No more
overpaying of premium hardware when similar or better analytical
processing could be done using commodity and open source systems.
• Quick turnaround and less bench times. A good bigdata and analytics
strategy could reduce the proof of concept time smoothly and substantially.
It reduces the burden on IT and gets more high quality, fast and cost effective
solutions baked

•
•
•
•
•
•
•
•
•
•
•
•

Automation to backfill redundant/mundane tasks
Optimize workforce to leverage high talent cost
Data continues to grow exponentially
Data is everywhere and in many formats
Alternate, Multiple Synchronous & Asynchronous data streams
Low barrier to entry
Traditional solutions failing to catch up with new market conditions
The Quest for Business Agility
Growing variation in types of data assets for analysis.
Alternate and unsynchronized methods for facilitating data delivery
Rising demand for real-time integration of analytical results.
Technology Trends Lowering Barriers to Entry: Application development
 Commoditized platform
 Big data management.
 Utility Computing

Analytics of Big Data

• Analytics is the discovery and communication of meaningful patterns in
data. Especially valuable in areas rich with recorded information, analytics
relies on the simultaneous application of statistics, computer programming
and operations research to quantify performance.
• Analytics often favors data visualization to communicate insight. Firms may
commonly apply analytics to business data, to describe, predict, and improve
business performance.
• Specifically, arenas within analytics include enterprise decision management,
retail analytics, store assortment and stock-keeping unit optimization,
marketing optimization and marketing mix modeling, web analytics, sales
force sizing and optimization, price and promotion modeling, predictive
science, credit risk analysis, and fraud analytics.
• Since analytics can require extensive computation, the algorithms and
software used for analytics harness the most current methods in computer
science, statistics, and mathematics.
• Analytics is increasingly used in education, particularly at the district and
government office levels.
• Basel III and future capital adequacy needs are likely to make even smaller
banks adopt internal risk models. In such cases, cloud computing and open
source can help smaller banks to adopt risk analytics and support branch
level monitoring by applying predictive analytics.

Analytics vs. analysis
• Analytics is a multi-dimensional discipline. There is extensive use of
mathematics and statistics, the use of descriptive techniques and predictive
models to gain valuable knowledge from data - data analysis.
• The insights from data are used to recommend action or to guide decision
making rooted in business context. Thus, analytics is not so much concerned
with individual analyses or analysis steps, but with the entire methodology.
• There is a pronounced tendency to use the term analytics in business settings
e.g. text analytics vs. the more generic text mining to emphasize this broader
perspective.
• There is an increasing use of the term advanced analytics, typically used to
describe the technical aspects of analytics, especially predictive modeling,
machine learning techniques, and neural networks

Marketing optimization
• Marketing has evolved from a creative process into a highly data-driven
process. Marketing organizations use analytics to determine the outcomes
of campaigns or efforts and to guide decisions for investment and consumer
targeting

• Demographic studies, customer segmentation, conjoint analysis and
other techniques allow marketers to use large amounts of consumer
purchase, survey and panel data to understand and communicate
marketing strategy
• Analysis techniques frequently used in marketing include marketing
mix modeling, pricing and promotion analyses, sales force optimization,
customer analytics e.g.: segmentation.
• Web analytics and optimization of web sites and online campaigns now
frequently work hand in hand with the more traditional marketing
analysis techniques
• These tools and techniques support both strategic marketing decisions
(such as how much overall to spend on marketing and how to allocate
budgets across a portfolio of brands and the marketing mix) and more
tactical campaign support in terms of targeting the best potential
customer with the optimal message in the most cost effective medium at
the ideal time
• A focus on digital media has slightly changed the vocabulary so that
marketing mix modeling is commonly referred to as attribution
modeling in the digital or Marketing mix modeling context.

Portfolio analysis
• A common application of business analytics is portfolio analysis. In this, a
bank or lending agency has a collection of accounts of varying value and
risk. The accounts may differ by the social status of the holder, the
geographical location, its net value, and many other factors. The lender must
balance the return on the loan with the risk of default for each loan.
• The analytics solution may combine time series analysis with many other
issues in order to make decisions on when to lend money to these different
borrower segments, or decisions on the interest rate charged to members of a
portfolio segment to cover any losses among members in that segment.

Risk analytics
• Predictive models in banking industry are widely developed to bring
certainty across the risk scores for individual customers. Credit scores are
built to predict individual’s delinquency behaviour and also scores are
widely used to evaluate the credit worthiness of each applicant and rated
while processing loan application. Furthermore, risk analyses are carried out
in the scientific world and the insurance industry

Techniques for Analysing Big Data – A New Approach
•
•
•
•
•

Discovery
Iteration
Flexible Capacity
Mining and Predicting
Decision Management

Big Data Use Cases

• Machine-Generated DataAs the “Internet of Things” grows steadily each
year, researchers predict that the amount of data generated by machines will
one day outstrip the amount of data generated by humans. Machina
Research, a UK-based research firm, believes there will be 12.5 billion
“smart” connected devices—excluding phones, PCs and tablets—in the
world in 2020, up from 1.3 billion today
• Online Reservations If you were running an online travel booking website,
there are lots of interesting things you could do with your data to better
understand your users
• Multi-Channel Marketing and Sentiment Analysis, Today’s retailers must
contend with a multitude of overlapping touch-points including social,
digital, direct, instore, mobile, and call center

Big Data Analysis Requirements
There are five key approaches to analyzing big data and generating
insight:
• Discovery tools are useful throughout the information lifecycle for
rapid, intuitive exploration and analysis of information from any
combination of structured and unstructured sources. These tools
permit analysis alongside traditional BI source systems. Because there
is no need for up-front modeling, users can draw new insights, come to
meaningful conclusions, and make informed decisions quickly
• BI tools are important for reporting, analysis and performance
management, primarily with transactional data from data warehouses
and production information systems. BI Tools provide comprehensive
capabilities for business intelligence and performance management,
including enterprise reporting, dashboards, ad-hoc analysis,
scorecards, and what-if scenario analysis on an integrated, enterprise
scale platform.
• In-Database Analytics include a variety of techniques for finding
patterns and relationships in your data. Because these techniques are
applied directly within the database

• Hadoop is useful for pre-processing data to identity macro trends or find
nuggets of information, such as out-of-range values. It enables businesses to
unlock potential value from new data using inexpensive commodity servers.
Organizations primarily use Hadoop as a precursor to advanced forms of
analytics
• Decision Management includes predictive modeling, business rules, and
self-learning to take informed action based on the current context. This type
of analysis enables individual recommendations across multiple channels,
maximizing the value of every customer interaction. Oracle Advanced
Analytics scores can be integrated to operationalize complex predictive
analytic models and create real-time decision processes

Types of Processing and Analysis with Hadoop
• Hadoop is a popular choice when you need to filter, sort, or pre-process
large amounts of new data in place and distill it to generate denser data that
theoretically contains more “information”. Pre-processing involves filtering
new data sources to make them suitable for additional analysis in a data
warehouse

How Big Data Effects Businesses and Firms
•
•
•
•

Big data expands customer intelligence.
Big data improves operational efficiencies
Big data + mobile means new business processes
No time to lose — big data and analytics go “as a service”.

How Real-World Enterprises Are Using Big Data?
• IBM’s Institute for Business Value (IBV) and the University of Oxford just
released their information-rich and insightful report “Analytics: The realworld use of big data.”
• Based on a survey of over 1000 professionals from 100 countries across
25+ industries, the report provides insights into organizations’ top
business objectives, where they are in their big data journey, and how they
are advancing their big data efforts. It also provides a pragmatic set of
recommendations to organizations as they proceed down the path of big
data.
• One very interesting factoid in the study is that 63% of respondents
indicated that the use of information (including big data) and analytics is
creating a competitive advantage for their organizations–a 70% increase in
the past two years alone. As an increasingly important segment of the
broader information and analytics market, big data is having a big impact

•

The study found that organizations are taking a pragmatic approach to
big data. 5 key findings highlighted in the study:-

 Across all industries, the business case for big data is strongly
focused on addressing customer-centric objectives
 A scalable and extensible information management foundation is a
prerequisite for big data advancement
 Organizations are beginning their pilots and implementations by
using existing and newly accessible internal sources of data
 Advanced analytic capabilities are required, yet often lacking, for
organizations to get the most value from big data.
 As awareness and involvement in big data grows, four key stages of
big data adoption emerge along a continuum
Thus, based on the results of the above study, following were the ways devised
for the big players as well as small firms that would help them compete in the
market:• Predict exactly what customers want before they ask for it.
• Get customers excited about their own data.
• Improve customer service interactions.
• Identify customer pain points and solve them.
• Reduce health care costs and improve treatment

The World's Top 10 Most Innovative Companies in Big
Data
•
•
•
•
•
•
•
•
•

GE
KAGGLE
AYASDI
MOUNT SINAI ICAHN SCHOOL OF MEDICINE
THE WEATHER COMPANY
KNEWTON
SPLUNK
GNIP
EVOLV

How different Sectors can reap benefits using
Analytics?

Case and Return Estimation.

Market Opportunities
Big Data is the biggest game-changing opportunity for marketing and sales
since the Internet went mainstream almost 20 years ago. . New technologies
as well as rapidly proliferating channels and platforms have created a
massively complex environment. At the same time, the explosion in data and
digital technologies has opened up an unprecedented array of insights into
customer needs and behaviorsThose that use Big Data and analytics effectively
show productivity rates and profitability that are 5 – 6 percent higher than
those of their peers. Data on its own, however, is nothing more than 1s and 0s.
The companies that succeed today do three things well:
• Use analytics to identify valuable opportunities.
• Start with the consumer decision journey.
• Keep it fast and simple.

Market Scenario

Chances of Growth

Recent Deals

• Apple-IBM Deal-With the increasing use of Apple mobile devices in the
enterprise, the recent pact between Apple and IBM promises to bring more
Big Data analytics capabilities into the hands of mobile users. The two
companies emphasized right up front the goal of "bringing IBM's Big Data
and analytics capabilities to iPhone and iPad" in their joint announcement.
IBM's Big Data know-how, combined with Apple's widespread mobile
presence, will boost the ongoing trend of putting large-scale analytics into
the hands of more mobile workers no matter where they are or what devices
they're using.
• IBM- Star Analytics Deal, . IBM will acquire Star Analytics' entire portfolio of
business intelligence automation solutions, which automate the integration
of data, BI applications, and other reporting tools across both on-premise
and cloud-based computing environments. According to IBM, Star Analytics'
solutions also eliminate the need for custom coding and other manual
processes when integrating BI solutions across disparate data sources.
• Loyalty management firm Aimia buys stake in Fractal Analytics
• Accenture Teams Up With Hortonworks, and Capgemini Extends Cloudera
Pact. Accenture's agreement with Hortonworks will bring the two
organizations together to collaborate on delivering big data solutions to
clients, including helping them integrate the Hortonworks data platform
with existing environments.

• Jan 2013 – Sapient acquires (m)PHASIZE,
• Dec 2013 – Apple acquires Topsy Labs. On its API Services support page,
Topsy mentions that it is the “only Twitter partner certified as both a data
reseller and an analytics provider” with “access to any and all social
conversations, in their entirety or summarized.”
• Dec 2013 – NPD launches Receipt Pal iPhone App to collect Consumer
receipt data.
• Oct 2013 – SGI Acquires FileTek, . “With the addition of FileTek solutions,
SGI enables existing and new customers to align both unstructured and
structured data with the most cost effective storage throughout its lifecycle,
with seamless user access and reliable petascale protection. This acquisition
also reflects our strategy to build on SGI’s leadership in High Performance
Computing, expertise in Big Data, and experience delivering over 600
petabytes of storage capacity annually, to become a global leader in petascale
storage solutions
• July 2014 – LinkedIn acquires Bizo, “The combination of LinkedIn and Bizo
greatly increases our ability to be the most effective platform for B2B
marketers to reach their audiences, nurture prospects and acquire
customers.” writes Russell Glass, CEO, Bizo on the company blog.

Some Info graphics