Data, Analytics, Intelligence

Business Intelligence

Empower users to access, combine and visualize data from a range of sources.
Summarize key performance indicators in a form suited to continuous monitoring.

BI products are often large and complex and can demand a large investment in people and IT before returns are realised. Basic BI offerings are strongly rooted in the tradition of reporting and dashboards and do not effectively address questions of insight.

BI is typically concerned with data selection and presentation. Data is often extracted from “live” systems and organised into more user-friendly structures to allow “drill-down”, “roll-up” and “slice and dice” selection and filtering.
The BI product category is characterized by quite large integrated software suites that work with structured data . Modern BI systems emphasize the online dashboard, a visual summary of key metrics with some personalisation and interaction possible.

The major BI vendors are extending their offering in response to a widening of perspective on analytics, for example to offer support for analysis of unstructured data and for more sophisticated statistical and data-mining analysis.

Microsoft BI is a good stereotype

IBM’s acquisition of SPSS (formerly the Statistical Package for the Social Sciences) and incorporation with its Cognos BI suite is a good example of where BI is goingvii.

Oracle and SAP form the remainder of the “big 4” BI vendors and SAS is particularly strong on advanced methods and statistics

http://publications.cetis.org.uk/wp-content/uploads/2012/12/Analytics-Brief-History-Vol-1-No9.pdf

==============================================================================================

Business Analytics

Business analytics is “a process of transforming data into actions through analysis and insights in the context of organizational decision making and problem solving” (Liberatore & Luo, 2010).

Evans has defined Business analytics as: “the use of data, information technology, statistical analysis, quantitative methods, and mathematical or computer-based models to help managers gain improved insight about their business operations and make better, fact-based decisions”.

Business analytics is commonly viewed from three major perspectives: descriptive, predictive, and prescriptive.

Descriptive analytics summarize data into meaningful charts and reports, for example, about budgets, sales, revenues, or cost. They allow managers to obtain standard and customized reports, and drill down into the data and to make queries to understand the impact of an advertising campaign, for example, review business performance to find problems or areas of opportunity, and identify patterns and trends in data. Typical questions that descriptive analytics help answer are: How much did we sell in each region? What was our revenue and profit last quarter? How many and what types of complaints did we resolve? Which factory has the lowest productivity? Descriptive analytics also help companies to classify customers into different segments, which enable them to develop specific marketing campaigns and advertising strategies.

    Predictive analytics analyze past performance in an effort to predict the future by examining historical data, detecting patterns or relationships in these data, and then extrapolating these relationships forward in time. For example, a marketer might wish to predict the response of different customer segments to an advertising campaign, a commodities trader might wish to predict short-term movements in commodities prices, or a skiwear manufacturer might want to predict next season’s demand for skiwear of a specific color and size. Predictive analytics can predict risk and finds relationships in data not readily apparent with traditional analyses. Using advanced techniques, predictive analytics can help to detect hidden patterns in large quantities of data to segment and group data into coherent sets in order to predict behavior and detect trends. For instance, a bank manager might want to identify the most profitable customers or predict the chances that a loan applicant will default, or alert a credit card customer to a potential fraudulent charge. Predictive analytics helps to answer questions such as: What will happen if demand falls by 10 percent or if supplier prices go up five percent? What do we expect to pay for fuel over the next several months? What is the risk of losing money in a new business venture?

    Prescriptive analytics uses optimization to identify the best alternatives to minimize or maximize some objective. Prescriptive analytics is used in many areas of business, including operations, marketing, and finance. For example, we may determine the best pricing and advertising strategy to maximize revenue, the optimal amount of cash to store in ATMs, or the best mix of investments in a retirement portfolio to manage risk. The mathematical and statistical techniques of predictive analytics can also be combined with optimization to make decisions that take into account the uncertainty in the data. Prescriptive analytics addresses questions like: How much should we produce to maximize profit? What is the best way of shipping goods from our factories to minimize costs? Should we change our plans if a natural disaster closes a supplier’s factory and if so, by how much?

    While the tools used in descriptive, predictive, and prescriptive analytics are different, many applications involve all three.

    Business analytics is a convergence of disciplines that have been taught and used for a long time:

statistics,

business intelligence

information systems

modeling and optimization (traditionally, operations research and management science)

 

 

 

 

 

 

 

 

Figure 1. One perspective on business analytics.

 

Figure 1 shows a perspective of the relationships and synergies that are defining business analytics. While the core topics are traditional in nature, the uniqueness lies in their intersections. For example, data mining is focused on better understanding characteristics and patterns among variables in large databases using a variety of statistical and analytical tools. Many standard statistical tools, such as data summarization, PivotTables, correlation and regression analysis, and other techniques are used extensively in data mining. However, data mining also brings to the table more advanced statistical methods such as cluster analysis and logistic regression. Risk analysis relies on spreadsheet models and statistical analysis to examine the impacts of uncertainty in the estimates and their potential interaction with one another on the output variable of interest, and is often facilitated by Monte Carlo simulation. Spreadsheets and formal models allow one to evaluate “what-if” questions—how specific combinations of inputs that reflect key assumptions will affect model outputs. What-if analysis is facilitated by systematic approaches that manipulate databases and models, such as data tables, the Excel Scenario Manager, and goal seek tools, and parametric sensitivity analysis used by Excel add-ins such as Risk Solver Platform, which makes it easy to create data tables and tornado charts that provide useful what-if information.

    Perhaps the most useful component of business analytics, which makes it truly unique, is the center of Figure 1—visualization (BA635 Bonus Link: Visualization Broadens Business Intelligence’s Appeal). Visualizing data and results of analyses provide a way of easily communicating data at all levels of a business, and can reveal surprising patterns and relationships.

Software such as IBM’s Cognos system exploits data visualization for query and reporting, data analysis, dashboard presentations, and scorecards linking strategy to operations. The Cincinnati Zoo, for example, has used this on an iPad to display hourly, daily, and monthly reports of attendance, food and retail location revenues and sales, and other metrics for prediction and marketing strategies. ARAMARK corporation developed visual “interactive simulators” to display the results of multivariate regression models on dials similar to those on an automobile dashboard, while allowing users to manipulate independent variables using simple sliders. UPS uses telematics to capture vehicle data and display them to help make decisions to improve efficiency and performance.3 IBM has predicted that data visualization will soon overtake historical trend analysis and standardized reporting as the analytic technique that provides the most value.

As academics in business schools, we have been teaching these topics for over 40 years, albeit in a disjointed and compartmentalized fashion. Business analytics provides the framework to exploit the synergies between traditionally-diverse topics in a more practical, application-driven format. Perhaps the fields of quantitative methods, OR/MS, DSS, or whatever we’ve known for the past 40 years will gain the respect they deserve.

 

Here is a typical example in retail operations. As you probably know from your shopping experiences, most department stores and fashion retailers clear their seasonal inventory by reducing prices. The key question they face is what prices should they set, and when should they set them to meet inventory goals and maximize revenue? For example, suppose that a store has 100 bathing suits of a certain style that go on sale April 1, and wants to sell all of them by the end of June. Over each week of the 12-week selling season, they can make a decision to discount the price. They face two decisions: when to reduce the price, and by how much? This results in 24 decisions to make. For a major national chain that may carry thousands of products, this can easily result in millions of decisions that store managers have to make! Descriptive analytics can be used to examine historical data for similar products, such as the number of units sold, price at each point of sale, starting and ending inventories, and special promotions, newspaper ads, direct marketing ads, and so on, to understand what the results of past decisions achieved. Predictive analytics can be used to predict sales based on pricing decisions. Finally, prescriptive analytics can be applied to find the best set of pricing decisions that maximize the total revenue.

 

http://www.cbpp.uaa.alaska.edu/afef/business_analytics.htm

=======================================================================================================

 

Tsallis entropy

https://en.wikipedia.org/wiki/Tsallis_entropy

Gini is Tsallis_entropy with q=2 and Boltzmann–Gibbs is  the limit Tsallis_ continuous entropy with q->1 

Given a discrete set of probabilities  undefined  with the condition  undefined , and  undefined  any real number, the Tsallis entropy is defined as

undefined

where  undefined  is a real parameter sometimes called entropic-index. In the limit as  undefined , the usual Boltzmann–Gibbs entropy is recovered, namely

undefined

For continuous probability distributions, we define the entropy as

undefined

where  undefined  is a probability density function.

The Tsallis Entropy has been used along with the Principle of maximum entropy to derive the Tsallis distribution.

Various relationships

The discrete Tsallis entropy satisfies

undefined

where Dq is the q-derivative with respect to x. This may be compared to the standard entropy formula:

undefined

 

 

==============================================================================================

Gini impurity and Entropy

Gini is Tsallis_entropy with q=2 and Boltzmann–Gibbs is Tsallis_entropy with q=1

=

= Is the the probability of obtaining two identical outputs, independently of their category

Gini is the chance of no Identicals in any category =  Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset. 

Gini is intended for continuous attributes and

Entropy is for attributes that occur in classes

Gini is to minimize misclassification
Entropy is for exploratory analysis

https://victorzhou.com/blog/gini-impurity/

Generally, your performance will not change whether you use Gini impurity or Entropy.

Laura Elena Raileanu and Kilian Stoffel compared both in “Theoretical comparison between the gini index and information gain criteria“. The most important remarks were:

  • It only matters in 2% of the cases whether you use gini impurity or entropy.

For the case of a variable with two values, appearing with fractions f and (1-f),
the gini and entropy are given by:
gini = 2*f(1-f)
entropy = f*ln(1/f) + (1-f)*ln(1/(1-f))
These measures are very similar if scaled to 1.0 (plotting 2*gini and entropy/ln(2) ):

Gini (y4,purple) and Entropy (y3,green) values scaled for comparison

Ref: https://datascience.stackexchange.com/questions/10228/when-should-i-use-gini-impurity-as-opposed-to-information-gain

============================================================================================

TPR=True positive rate= sensitivity = recall = probability of detection  =  p(+o|+a)=positive detection rate

FPR= False positive rate = probability of false alarm= p(+o|-a)

ROC == (TPR vs FPR) as the criterion changes

=====================================================================================

 Karl Pearson’s phi coefficient or (Matthews correlation coefficient)

evaluate each test performance through the Matthews correlation coefficient (MCC), instead of the accuracy and the F1 score, for any binary classification problem.  F1 score depends on which class is defined as the positive class. The MCC doesn’t depend on which class is the positive one, which has the advantage over the F1 score to avoid incorrectly defining the positive class.

https://en.wikipedia.org/wiki/Matthews_correlation_coefficient

=====================================================================================

Data Visualization in R