Think Stats on Amit Rajan

Correlation

Mon, 10 Sep 2018 11:13:31 +0100

9.1 Standard scores

The main challenge in measuring correlation is that the variables we want to compare might not be expressed in the same units. There are two common solutions to this problem:

Transform all values to standard scores. This leads to Pearson coefficient of correlation.
Transform all values to their percentile ranks. This leads to Spearman coefficient.

Normalizing the score means subtracting mean from every value and dividing it by standard deviation.

Estimation

Tue, 04 Sep 2018 14:07:28 +0100

8.1 The estimation game

In a rudimentary way, sample mean can be used to estimate a distribution. The process is called estimation and the statistic we used (sample mean) is called an estimator. If there are no outliers, sample mean minimizes the mean squared error (MSE). A maximum likelihood estimator (MLE) is an estimator that has the highest chance of being right (value with highest probability).

Exercise 8.1: Write a function that draws 6 values froma normal distribution with \(\mu\) = 0 and \(\sigma\) = 1. Use the sample mean to estimate m and compute the error \(\bar{x} - \mu\). Run the function 1000 times and compute MSE.

Hypothesis Testing

Thu, 30 Aug 2018 06:17:39 +0100

The process of Hypothesis Testing can be summarized by following three steps:

Null Hypothesis: The null hypothesis is a mode of the system based on the assumption that apparent effect was actually due to chance.
p-value: The p-value is the probability of the apparent effect under the null hypothesis.
Interpretation: Based on the p-value, we can conclude that whether the effect is statistically significant or not.

Hence to test a hypothesis, we assume that it is not real (called as null hypothesis). Based on the assumption (i.e. null hypothesis), we compute the probability of the apparent effect (this is called p-value). If p-value is low enough, we can conclude that the null hypothesis is unlikely to be true.

Operations on Distributions

Sun, 26 Aug 2018 16:27:09 +0100

6.1 Skewness

Skewness is a statistic that measures the assymetry of a distribution. Sample skewness is defined as:

\[Skewness = \frac{Mean\ Cubed\ Deviation}{Mean\ Squared\ Deviation^{\frac{3}{2}}}\]

Negative skewness means that the distribution skews left and positive skewness means that the distribution is skewed to right. Outliers have a disproportionate effect on skewness.

Another way to evaluate the asymmetry of a distribution is to compare mean and median. For distributions that skews left, mean is less than the median. This statistic is denoted by Pearson’s Median Skewness Coefficient which is given as:

Probability

Wed, 22 Aug 2018 09:12:36 +0100

The belief in which probability is expressed in terms of frequencies is called as frequentism. Frequentists believe that if there is no set of identical trials, there is no probability.

An alternative is Bayesianism, which defines probability as a degree of belief that an event will occur. By this definition, the notion of probability can be applied in almost any cicumstance. One drawback of Bayesian probability is that it depends on person’s state of knowledge. People with different information might have different degree of belief about the same event. This is the reason that many people think that Bayesian probability is more subjective than frequentists.

Continuous Distributions

Fri, 17 Aug 2018 19:02:16 +0100

4.1 The exponential distribution

CDF of the exponential distribution is defined as (where lambda determines the shape of the distribution):

\[CDF(x) = 1-e^{-\lambda x}\]

Mean and Median of exponential distribution is computed as follows:

\[Mean = \frac{1}{\lambda}\]

\[Median = \frac{ln(2)}{\lambda}\]

In the real world, exponential distribution is encountered when we look at the series of events and measure the time between them which is called as interval times. The exponential distribution for different values of lambda is shown below. It can be seen that lambda decides the shape of the distribution.

Cumulative Distribution Functions

Sun, 12 Aug 2018 09:22:46 +0100

3.1 The class size paradox

For a probability distribution, the mean calculated from its PMF is lower than the one calculated by taking a sample from it. This happens because the larger classes tend to get oversampled.

Exercise 3.1 Build the PMF of the college class-size data and compute the mean as perceived by the Dean. Find the distribution of class sizes as perceived by students and compute its mean.

Descriptive Statistics

Wed, 08 Aug 2018 04:17:12 +0100

2.1 Means and averages

Mean of a sample is a summary statistics that can be computed as (where n is the total number of samples):

\[\mu = \frac{1}{n} \sum_i{x_i}\]

An average is one of many summary statistics that can be used to describe the typical value or the central tendency of a sample.

2.2 Variance

As mean describes the central tendency of a sample, Variance is intended to describe the spread. The variance is defined as:

Statistical Thinking for Programmers

Sun, 05 Aug 2018 10:02:08 +0100

1.1 Do first babies arrive late?

Anecdotal Evidence is based on data that is unpublished and usually personal. For example,

“My two friends that have given birth recently to their first babies, BOTH went almost 2 weeks overdue before going into labour or being induced.” Anecdotal Evidence usually fail because of Small number of observations, Selection bias (People who join a discussion of this question might be interested because their first babies were late.), Confirmation bias (People who believe the claim might be more likely to contribute examples that confirm it) and Inaccuracy.