On this page

Distributions

Geometric Distribution

The geometric distribution frequently occurs in applications because we are often interested in how long we have to wait before a certain event happens.

A random variable \(X\) for which

\[\mathbb{P}[X=i]=(1-p)^{i-1}p,\text{ for }i=1,2,3,\cdots\]

is said to have the geometric distribution with parameter \(p\), abbreviated as \(X\sim Geometric(p)\).

As a santiy check, we can verify the total probability of \(X\) is equal to 1:

\[\sum_{i=1}^\infty\mathbb{P}[X=i]=\sum_{i=1}^\infty(1-p)^{i-1}p=p\sum_{i=1}^ \infty(1-p)^{i-1}=p\times\frac{1}{1-(1-p)}=1\]

If we plot the distribution of \(X\), we get a curve that decreases monotonically by a factor of \(1-p\) at each step.

Mean and Variance

Let \(X\) be a random variable that takes in values \(\{0,1,2,\cdots\}\). Then,

\[\mathbb{E}[X]=\sum_{i=1}^\infty\mathbb{P}[X\geq i]\]

Below is the proof of what we call the Tail Sum Formula.

Proof

For notational convenience, let’s write \(p_i=\mathbb{P}[X=i]\), for \(i=0,1,2, \cdots\).

\[\begin{split}\mathbb{E}[X]=(0\times p_0)+(1\times p_1)+(2\times p_2)+\cdots\\ =p_1+(p_2+p_2)+(p_3+p_3+p_3)+\cdots=(p_1+p_2+p_3+\cdots)+(p_2+p_3+\cdots)+\cdots \\=\mathbb{P}[X\geq 1]+\mathbb{P}[X\geq 2]+\mathbb{P}[X\geq 3]+\cdots\end{split}\]

In compact notation:

\[\mathbb{E}[X]=\sum_{j=1}^\infty j\times\mathbb{P}[X=j]=\sum_{j=1}^\infty\sum_ {i=1}^j\mathbb{P}[X=j]=\sum_{i=1}^\infty\sum_{j=i}^\infty\mathbb{P}PX=j]= \sum_{i=1}^\infty\mathbb{P}[X\geq 1]\]

For \(X\sim Geometric(p)\), we have \(\mathbb{E}[X]=\frac{1}{p}\).

Proof

The key observation is for a geometric random variable \(X\),

\[\mathbb{P}[X\geq i]=(1-p)^{i-1},\text{ for }i=1,2,3,\cdots\]

We can obtain this simply by summing \(\mathbb{P}[X=j]\) for \(j\geq i\). Another way of seeing this is to note that the event \(X\geq i\) means at least \(i\) tosses are required, or the first \(i-1\) tosses are all tails and the probability of this event is \((1-p)^{i-1}\).

\[\mathbb{E}[X]=\sum_{i=1}^\infty\mathbb{P}[X\geq i]=\sum_{i=1}^\infty(1-p)^ {i-1}=\frac{1}{1-(1-p)}=\frac{1}{p}\]

And for \(X\sim Geometric(p)\), we have \(Var(X)=\frac{1-p}{p^2}\).

Proof

We will show \(\mathbb{E}[X(X-1)]=\frac{2(1-p)}{p^2}\):

\[\begin{split}Var(X)=\mathbb{E}[X^2]-\mathbb{E}[X]^2=\mathbb{E}[X(X-1)]+\mathbb{E}[X]- \mathbb{E}[X]^2\\=\frac{2(1-p)}{p^2}+\frac{1}{p}-\frac{1}{p^2}=\frac{2(1-p)+p -1}{p^2}=\frac{1-p}{p^2}\end{split}\]

To show \(\mathbb{E}[X(X-1)]=\frac{2(1-p)}{p^2}\):

\[\begin{split}\frac{d^2x}{dy^2}(\sum_{i=0}^\infty(1-p)^i=\frac{d^2x}{dy^2}(\frac{1}{p})\\= \sum_{i=2}^\infty i(i-1)(1-p)^{i-2}=\frac{2}{p^3}\end{split}\]
\[\begin{split}\mathbb{E}[X(X-1)]=\sum_{i=1}^\infty i(i-1)\times\mathbb{P}[X=i]\\=\sum_{i=2}^ \infty i(i-1)(1-p)^{i-1}p=p(1-p)\sum_{i=2}^\infty i(i-1)(1-p)^{i-2}\\=p(1-p) \times\frac{2}{p^3}=\frac{2(1-p)}{p^2}\end{split}\]

Poisson Distribution

A random variable \(X\) for which

\[\mathbb{P}[X=i]=\frac{\lambda^i}{i!}e^{-\lambda},\text{ for }i=0,1,2,\cdots\]

is said to have the Poisson distribution with parameter \(\lambda\), abbreviated as \(X\sim Poisson(\lambda)\).

As a sanity check,

\[\sum_{i=0}^\infty\frac{\lambda^i}{i!}e^{-\lambda}=e^{-\lambda}\sum_{i=0}^ \infty\frac{\lambda^i}{i!}=e^{-\lambda}\times e^\lambda = 1\]

where in the second to last step, we used the Taylor expansion \(e^x=1+x+\frac{x ^2}{2!}+\cdots\).

The Poisson distribution is used as a widely accepted model for so-called “rare events”, such as car accidents or ICU admission at a hospital. It is appropriate whenever the occurrences can be assumed to happen randomly with some constant density in a continuous region (of time or space), such that occurrences in disjoint subregions are independent.

Mean and Variance

For a Poisson random variable \(X\) with parameter \(\lambda\), we have \(\mathbb{E} [X]=Var(X)=\lambda\),

Proof

\[\begin{split}\mathbb{E}[X]=\sum_{i=0}^\infty i\times\mathbb{P}[X=i]\\=\sum_{i=1}^\infty i \times\frac{\lambda^i}{i!}e^{-\lambda}=\lambda e^{-\lambda}\sum_{i=i}^\infty \frac{\lambda^{i-1}}{(i-1)!}\\=\lambda e^{-\lambda}e^\lambda=\lambda\end{split}\]

Similarly,

\[\begin{split}\mathbb{E}[X(X-1)]=\sum_{i=0}^\infty i(i-1)\times\mathbb[P][X=i]\\ =\sum_{i=2}^\infty i(i-1)\frac{\lambda^i}{i!}e^{-\lambda}\\ =\lambda^2e^{-\lambda}\sum_{i=2}^\infty\frac{\lambda^{i-2}}{(i-2)!}\\ =\lambda^2e^{-\lambda}e^{\lambda}=\lambda^2\end{split}\]

Therefore,

\[Var(X)=\mathbb{E}[X^2]-\mathbb{E}[X]^2=\mathbb{E}[X(X-1)]+\mathbb{E}[X]- \mathbb{E}[X]^2=\lambda^2+\lambda-\lambda^2=\lambda\]

Sum of Independent Poisson Random Variables

Theorem

Let \(X\sim Poisson(\lambda)\) and \(Y\sim Poisson(\mu)\) be independent Poisson random variables. Then, \(X+Y\sim Poisson(\lambda+\mu)\).

Proof

For all \(k=0,1,2,\cdots\), we have

\[\begin{split}\mathbb{P}[X+Y=k]=\sum_{j=0}^k\mathbb{P}[X=j,Y=k-j]\\=\sum_{j=0}^k\mathbb{P} [X=j]\mathbb{P}[Y=k-j]\\=\sum_{j=0}^k\frac{\lambda^j}{j!}e^{-\lambda}\frac{\mu^ {k-j}}{(k-j)!}e^{-\mu}\\=e^{-(\lambda+\mu)}\frac{1}{k!}\sum_{j=0}^k\frac{k!} {j!(k-j)!}\lambda^j\mu^{k-j}\\=e^{-(\lambda+\mu)}\frac{(\lambda+\mu)^k}{k!}\end{split}\]

where the second equality follows from independence, and the last equality from the binomial theorem.

By induction, we can conclude for \(X_1,X_2,\cdots,X_n\) independent Poisson random variables with parameters \(\lambda_1,\cdots,\lambda_n\).,

\[X_1+\cdots+X_n\sim Poisson(\lambda_1+\cdots+\lambda_n)\]

Poisson as a Limit of Binomial

Theorem

Let \(X\sim Binomial(n,\frac{\lambda}{n})\) where \(\lambda>0\) is a fixed constant. For every \(i=0,1,2,\cdots\),

\[\mathbb{P}{X=i}\rightarrow\frac{\lambda^i}{i!}e^{-\lambda}\text{ as }n \rightarrow\infty\]

Proof

Fix \(i\in\{0, 1, 2, \cdots \}\), and assume \(n\geq i\) (because \(n\rightarrow \infty\)). Then, because \(X\) is a binomial distribution,

\[\mathbb{P}[X=i]=\binom{n}{i}p^i(1-p)^{n-i}=\frac{n!}{i!(n-i)!}(\frac{\lambda} {n})^i(1-\frac{\lambda}{n})^{n-i}\]

Collecting the factors,

\[\mathbb{P}[X=i]=\frac{\lambda^i}{i!}(\frac{n!}{(n-i)!}\frac{1}{n^i})(1-\frac {\lambda}{n})^n(1-\frac{\lambda}{n})^{-i}\]

The first parenthesis above, as \(n\rightarrow\infty\), becomes

\[\frac{n!}{(n-i)!}\frac{1}{n^i}=\frac{n(n-1)\cdots(n-i+1)(n-i)!}{(n-i)!}\frac {1}{n^i}=\frac{n}{n}\frac{n-1}{n}\cdots\frac{n-i+1}{n}\rightarrow 1\]

The second partnhesis becomes

\[(1-\frac{\lambda}{n})^n\rightarrow e^{-\lambda}\]

Since \(i\) is fixed,

\[(1-\frac{\lambda}{n})^{-i}\rightarrow(1-0)^{-i}=1\]

Substituting these into the above equation,

\[\mathbb{P}[X=i]\rightarrow\frac{\lambda^i}{i!}e^-{\lambda}\]
Random Variables: Variance and Covariance Continuous Probabilities