Let $X$ and $Y$ be two discrete random variables, then the joint probability distribution of $X$ and $Y$ is given by

Since $X$ and $Y$ are discrete (and can therefore only take on a finite or countable number of values), the joint distribution $P_{XY}(x,y)$ is only non-zero at a countable number of values.

Joint probability distributions must satisfy the following properties

1. $0\leq P_{XY}(x,y) \leq 1$ for all $x,y$.
2. All the probabilities add up to one \[ \sum_{x}\sum_{y} P_{XY}(x,y) = 1, \] where the sum is over all values $x,y$ such that $P_{XY}(x,y) \neq 0$.

Note: The above summation is over a discrete set of values since $X$ and $Y$ are discrete. It can be an infinite sum.

Example 1 (Roll two fair dice)

Recall from the notes on discrete probability the problem of rolling two fair dice. Let

then the probability table is given by:

Since there are $36$ cells in the table, the total probability clearly sums up to one.

This table is pretty boring since it does not indicate an interesting relationship between the random variables $X$, and $Y$. In fact $X$ and $Y$ are independent.

As a more interesting case, consider another random variable \[ Z = 7-X, \] which takes what ever $X$ gives and "flips" it around 3. Then $X$ and $Z$ are clearly dependent, since the value of $X$ completely determines the value of $Z$, the probability table becomes:

If you were to look at the distribution of just $Z$, you wouldn't be able to tell that $Z$ was "copying" off of $X$. This can only be recognized by considering the joint distribution.

Example 2

Let $X$ and $Y$ be as in the previous example. What is the probability that $X < Y$?

The event $\{X < Y\}$ can be described explicitly as the set of $(x,y)$ pairs

This can be visualized as a triangular subset of the probability table considered in Example 1, colored salmon below:

Since there are $15$ such elements, we see that \[ P(X< Y) = \frac{15}{36}. \]

Let $X_1, X_2, \ldots X_n$ be discrete random variables, then the joint probability distribution of $X_1,X_2,\ldots X_n $ is given by

A joint probability density $f_{XY}(x,y)$ for two continuous random variables $X$ and $Y$ must satisfy

1. $0\leq f_{XY}(x,y)$ for all $x,y \in \mathbb{R}$.
2. The total integral is 1, \[ \int_0^\infty\int_0^\infty f_{XY}(x,y)\mathrm{d}x\mathrm{d}y = 1. \]

Suppose that $X$ and $Y$ are continuous random variables with values in $[0,1]$ and joint density given by \[ f_{XY}(x,y) = \begin{cases} cxy , & \text{if } 0\leq x,y\leq 1\\ 0 & \text{otherwise}\end{cases} \]

What is the value of $c$ that makes this a valid joint pdf? For this value of $c$, what is the probability that $X>1/2$ and $Y<1/2$?

Clearly $f_{XY}(x,y) \geq 0$, if $c \geq 0$, and therefore we simply need to check the normality condition 2. We can do this using iterated partial integrals:

Therefore we need $c = 4$ for this to be a valid joint pdf.

To find the probability that $X>1/2$ and $Y<1/2$ we note that

This problem is taken from Example 5.4 in the text.

Suppose that $Y_1$ and $Y_2$ are continuous random variables with values in $[0,1]$ such that $Y_2\leq Y_1$ and the joint density given by \[ f_{Y_1 Y_2}(y_1,y_2) = \begin{cases} cy_1 , & \text{if } 0\leq y_2\leq y_1\leq 1\\ 0 & \text{otherwise}\end{cases} \]

What is the value $c$ that makes $f_{Y_2 Y_2}(y_1,y_2)$ a valid joint density function? What is the probability that $0\leq Y_1 \leq 1/2$ and $Y_2 > 1/4$?

Let's check the normality condition. First, we should recognize that the domain of $f_{Y_1 Y_2}(y_1,y_2)$ is a triangular region (shown in the figure below)

This is a region bounded between two curves and there integral can be set up as

Therefore we must have $c=3$.

To compute the probability that $0\leq Y_1 \leq 1/2$ and $Y_2 > 1/4$ we need to carefully consider the region we are integrating over. Note, for instance, that if $Y_1 < 1/4$, then since $Y_2 \leq Y_1$, it's not possible for $Y_2 \geq 1/4$. We need to consider how the region $\{0\leq y_1\leq 1/2,y_2 > 1/4\}$ overlaps with the triangular domain $ D = \{0\leq y_2\leq y_1 \leq 1\}$. This is illustrated in the following figure.

The associated integral is

Let $X_1, X_2, \ldots X_n$ be continuous random variables, then the joint pdf, $f_{X_1 X_2 \ldots X_n}(x_1,x_2,\ldots,x_n)$, of $X_1,X_2,\ldots X_n $ is define in terms of the $n$-dimensional volume integral

where $R\subset \mathbb{R}^n$ is any subregion of $n$ dimensional space. In this setting, it is useful to think of \[ \mathbf{X} = (X_1,X_2,\ldots,X_n) \] as a random $n$-dimensional vector in.

If $X$ and $Y$ are discrete random variables with joint distribution function $P_{XY}(x,y)$, then the joint cdf is just given by

where the sum is over $u$ and $v$ such that $P_{XY}(u,v) \neq 0$.

Example 5

Let $X$ and $Y$ be as in the previous example be as in example 1. What is $F_{XY}(3.5,4)$?

$F_{XY}(3.5,4)$ is just the event $X\leq 3.5, Y\leq 4$, we can visualize this shading cells in the probability table, colored salmon below:

Since there are $12$ such elements, we see that \[ F_{XY}(3.5,4) = \frac{12}{36}. \]

If $X$ and $Y$ are continuous random variables with joint pdf $f_{XY}(x,y)$, then the joint cdf is just given by

We can also recover the joint pdf from the joint cdf using partial derivatives \[ f_{XY}(x,y) = \frac{\partial^2 F_{XY}}{\partial x \partial y}(x,y). \]

Example 6

Let $Y_1$ and $Y_2$ be as in Example 4. What is the joint cdf?

The event $Y_1 \leq y_1$ and $Y_2\leq y_2$ depends on whether $y_1 \leq y_2$ or $y_1 \geq y_2$. Both cases are illustrated below

If $0\leq y_1 \leq y_2\leq 1$, then the region of integration is a triangle

On the other hand, if $0\leq y_2 \leq y_1\leq 1$ then the region of integration can be split up into a triangle and rectangle

Putting this together gives

If $X$ and $Y$ are any random variables with joint cdf $F_{XY}(x,y)$ then the following properies hold

1. F_{XY}(x,y) is non-decreasing, meaning that if $x$ or $y$ increase, then $F_{XY}(x,y)$ can't decrease
2. If you send any variable to $-\infty$, then $F_{XY}(x,y)$ goes to zero, i.e. $F_{XY}(-\infty,-\infty) = F_{XY}(-\infty,y) = F_{XY}(x,-\infty) =0.$
3. If you send both variables to $+\infty$, then $F_{XY}(x,y)$ approaches $1$, i.e. $F_{XY}(\infty,\infty ) = 1$.

Example 7

Recall the die tossing experiment from Example 1. Suppose we consider the event $\{X = 3\}$. We know that this corresponds to the the following pairs

Using the additive law of probability, we can calculate $P(X=3)$ by summing along the rows of the table

Of course we already knew this!

Suppose that $X$ and $Y$ are discrete random variables with joint distribution function $P_{XY}(x,y)$. The marginal distribution functions $P_X(x)$ and $P_Y(y)$ for $X$ and $Y$ respectively are given by summing the joint distribution over the other variable

Suppose that $X$ and $Y$ are continuous random variables with joint pdf $f_{XY}(x,y)$. The marginal distribution pdfs $f_X(x)$ and $f_Y(y)$ for $X$ and $Y$ respectively are given by integrating out the other variable.

Example 8

Consider the joint pdf from Example 3

Find the $X$ and $Y$ marginals.

Example 9

Consider the more complicated pdf of Example 4

Find the $X$ and $Y$ marginals.

Taking into account the triangular region,

Let $X$ and $Y$ be two random variables with joint cdf $F_{XY}(x,y)$, then the marginal cdfs are given by

Note that if $X$ and $Y$ only take values in a rectangle $[a,b]\times [c,d]$, then this can be simplified to evaluating $x$ or $y$ at the right end point of it's respective domain,

Example 10

Lets consider the cdf we calculated from Example 6.

Find the $X$ and $Y$ marginal cdfs.

Evaluating at the $y_1 = 1$ and $y_2 = 1$ gives

Note that $F^\prime_{Y_1}(y_1) = 3y_1^2$ and $F^\prime_{Y_2}(y_2) = \frac{3}{2}(1-y_2^2)$, which is consistent with the answers to Example 9.

Let $X$ and $Y$ be two random variables with joint distribution $P_{XY}(x,y)$, then the conditional distribution function of $X$ given $Y$ is

Let $X$ and $Y$ be two continuous random variables with joint pdf $f(x,y)$ and joint cdf $F(x,y)$, then the conditional cdf of $X$ given $Y$ is defined by

The expression $P(X\leq x| y \leq Y \leq y+h)$ is well defined since $P(y \leq Y \leq y+h) \neq 0$ for some values of $y$.

We can calculate this limit explicitly since

where in the last line we multiplied and divided by $\frac{1}{h}$. Using the fact that

as $h\to 0$, completes the proof.

QED

Let $X$ and $Y$ be two continuous random variables with joint pdf $f(x,y)$, then the conditional pdf of $X$ given $Y$ is

Example 11

Lets consider pdf from Example 6.

Find $f_{Y_1|Y_2}(y_1|y_2)$ and use it to calculate $P(Y_1\leq 3/4|Y_2 = 1/2)$.

From example 9, we found that \[ f_{Y_2}(y_2) = \frac{3}{2}(1-y_2^2) \quad 0\leq y_2\leq 1. \] Therefore

To calculate $P(Y_1\leq 1/2|Y_2 = 1/2)$ we note that

Therefore

Two random variables $X$ and $Y$ with joint cdf $F_{XY}(x,y)$ and marginal cdfs $F_X(x)$ and $F_Y(y)$ are independent if and only if

Otherwise $X$ and $Y$ are said to be dependent.

Two discrete random variables $X$ and $Y$ with joint distribution function $P_{XY}(x,y)$ and marginal distributions $P_X(x)$ and $P_Y(y)$ are independent if and only if

Otherwise $X$ and $Y$ are said to be dependent.

Example 12

Lets consider the die rolling experiment from Example 1. Of course we know that $X$ and $Y$ are independent, but lets check that out notion of independence is correct.

The probability table is

Since each marginal has probability $\frac{1}{6}$ and each cell has probability $\frac{1}{36}$ (the produce of the marginals), we can see that $X$ and $Y$ are independent.

On the otherhand, lets consider $X$ and $Z = 7 -X$. The probability table was given by

In this table, we can see that many of the probabilities are not the product of two marginal probabilities and so $X$ and $Z$ are dependent. Indeed, since none of the marginal probabilities are zero, then none of the cells with zero probability can be a product of marginals.

Two continuous random variables $X$ and $Y$ with joint pdf $f_{XY}(x,y)$ and marginal pdfs $f_X(x)$ and $f_Y(y)$ are independent if and only if

Otherwise $X$ and $Y$ are said to be dependent.

Example 13

In this case, since $f_X(x) = 2x$ for $0\leq x \leq 1$ and $f_Y(y) = 2y$ and $0\leq y\leq 1$, we can easily see that for $0\leq x,y\leq 1$, \[ f_X(x)f_Y(y) = 4xy = f_{XY}(x,y). \]

Example 14

How about the pdf from Example 6? Are $Y_1$ and $Y_2$ independent?

In this case, we know that for $0\leq y_1\leq 1$ and $0\leq y_2 \leq 1$ we have

So that for all $0\leq y_1,y_2\leq 1$

This is no where close to $f_{Y_1 Y_2}(y_1,y_2)$ if not for the simple fact that $f_{Y_1}(y_1)f_{Y_2}(y_2) \neq 0$ when $0\leq y_1\leq y_2 \leq 1$ while $f_{Y_1 Y_2}(y_1,y_2) = 0$.