Joint Distributions & Random Vectors

Overview

This topic extends probability to multiple random variables. We cover joint and marginal densities, conditional distributions, independence, and operations on pairs of random variables. We then generalize to random vectors, introducing the covariance matrix, and give full treatment to Gaussian vectors — the most important multivariate distribution in probability and finance.

1. Joint Distributions (Couples)

1.1 Joint Density & Marginals

Definition— Joint Density

A pair $(X, Y)$ of continuous random variables has a joint density $f_{X,Y} : \mathbb{R}^2 \to \mathbb{R}_+$ such that for any Borel set $A \subseteq \mathbb{R}^2$ :

\boxed{P((X,Y) \in A) = \iint_A f_{X,Y}(x,y)\, dx\, dy}

satisfying $f_{X,Y}(x,y) \geq 0$ and $\displaystyle\iint_{\mathbb{R}^2} f_{X,Y}(x,y)\, dx\, dy = 1$ .

Definition— Marginal Densities

The marginal densities of $X$ and $Y$ are obtained by integrating out the other variable:

\boxed{f_X(x) = \int_{-\infty}^{+\infty} f_{X,Y}(x,y)\, dy, \qquad f_Y(y) = \int_{-\infty}^{+\infty} f_{X,Y}(x,y)\, dx}

Definition— Joint CDF

\boxed{F_{X,Y}(x,y) = P(X \leq x,\, Y \leq y) = \int_{-\infty}^x \int_{-\infty}^y f_{X,Y}(u,v)\, dv\, du}

Remarque

Knowing the marginals $f_X$ and $f_Y$ is not sufficient to recover the joint density $f_{X,Y}$ — you also need the dependence structure. The joint density contains strictly more information than the two marginals separately, unless and are independent.

1.2 Conditional Density

Definition

The conditional density of $X$ given $Y = y$ (with $f_Y(y) > 0$ ) is:

Formula— Conditional Expectation

\boxed{E[X|Y=y] = \int_{-\infty}^{+\infty} x\, f_{X|Y}(x|y)\, dx}

Formula— Total Probability (continuous version)

\boxed{f_X(x) = \int_{-\infty}^{+\infty} f_{X|Y}(x|y)\, f_Y(y)\, dy}

💡Tip

The conditional density is the continuous analog of Bayes: $f_{X|Y}(x|y) \propto f_{Y|X}(y|x)\, f_X(x)$ . This is the foundation of Bayesian inference for continuous parameters.

1.3 Independence

Definition

$X$ and $Y$ are independent if and only if their joint density factors:

\boxed{f_{X,Y}(x,y) = f_X(x)\cdot f_Y(y) \quad \text{for all } (x,y) \in \mathbb{R}^2}

Properties

$X \perp Y \Rightarrow E[g(X)h(Y)] = E[g(X)]\cdot E[h(Y)]$ for any measurable

Remarque

The converse $\text{Cov}(X,Y) = 0 \Rightarrow X \perp Y$ is false in general. The canonical counterexample: $X \sim \mathcal{N}(0,1)$ , — then but is completely determined by .

1.4 Covariance & Correlation

Definition

The covariance between $X$ and $Y$ measures linear dependence:

\boxed{\text{Cov}(X,Y) = E[XY] - E[X]\,E[Y] = E[(X-\mu_X)(Y-\mu_Y)]}

Properties

$\text{Cov}(X,X) = \text{Var}(X)$
$\text{Cov}(aX+b,\, cY+d) = ac\,\text{Cov}(X,Y)$

2. Operations on Random Variables

2.1 Sum: $Z = X + Y$

Theorem— Convolution Formula

If $X$ and $Y$ are independent continuous random variables, the density of $Z = X + Y$ is:

f_{Z} (z) = (f_{X} * f_{Y}) (

Properties

Normal + Normal: $\mathcal{N}(\mu_1,\sigma_1^2) + \mathcal{N}(\mu_2,\sigma_2^2) = \mathcal{N}(\mu_1+\mu_2,\, \sigma_1^2+\sigma_2^2)$

💡Tip

In practice, use MGFs to find the distribution of a sum: $M_{X+Y}(t) = M_X(t)\cdot M_Y(t)$ when , then identify the resulting MGF. Convolution integrals are a last resort.

2.2 Product: $Z = X \cdot Y$

Formula

If $X$ and $Y$ are independent with densities $f_X$ and $f_Y$ , the density of is:

Example

If $X, Y \overset{iid}{\sim} \mathcal{U}(0,1)$ , then $Z = XY$ has density for .

2.3 Ratio: $Z = X / Y$

Formula

If $X$ and $Y$ are independent with densities $f_X$ and $f_Y$ and , the density of is:

Example

If $X, Y \overset{iid}{\sim} \mathcal{N}(0,1)$ independent, then $Z = X/Y \sim \text{Cauchy}(0,1)$ .

💡Tip

The ratio of two independent standard normals is Cauchy — this is why the Cauchy has no mean (its tails are too heavy). More generally: $t(\nu) = \mathcal{N}(0,1) / \sqrt{\chi^2(\nu)/\nu}$ is a ratio distribution.

3. Random Vectors

3.1 Definition & Marginals

Definition

A random vector $\mathbf{X} = (X_1, \ldots, X_n)^T$ is a measurable function from to . Its distribution is characterized by the satisfying:

3.2 Covariance Matrix

Definition

The covariance matrix of $\mathbf{X} = (X_1, \ldots, X_n)^T$ is the matrix:

Properties

$\Sigma$ is symmetric: $\Sigma = \Sigma^T$
$\Sigma$ is positive semi-definite: $\mathbf{v}^T \Sigma \mathbf{v} \geq 0$ for all

💡Tip

The covariance matrix is the fundamental object in portfolio optimization (Markowitz), PCA, and multivariate regression. In interviews, know that $\mathbf{a}^T\Sigma\mathbf{a}$ is the variance of the portfolio $\mathbf{a}^T\mathbf{X}$ .

3.3 Change of Variable (Multivariate)

Theorem

Let $\mathbf{X}$ have joint density $f_{\mathbf{X}}$ and $\mathbf{Y} = \varphi(\mathbf{X})$ where $\varphi : \mathbb{R}^n \to \mathbb{R}^n$ is a diffeomorphism. Then:

Example

Polar coordinates: $(X,Y) \to (R, \Theta)$ with $x = r\cos\theta$ , $y = r\sin\theta$ . The Jacobian gives , so .

4. Gaussian Vectors

4.1 Definition

Definition

A random vector $\mathbf{X} = (X_1, \ldots, X_n)^T$ is a (multivariate normal) if every linear combination follows a normal distribution. We write .

Remarque

The definition via linear combinations is more general than the density formula — it covers degenerate cases where $\Sigma$ is only positive semi-definite (e.g. $X_1 = X_2$ ). A vector $(X_1, \ldots, X_n)$ with all is Gaussian — the components must be jointly Gaussian.

4.2 Properties

Properties

$E[\mathbf{X}] = \boldsymbol{\mu}$ , $\text{Cov}(\mathbf{X}) = \Sigma$
Linear stability: $A\mathbf{X} + \mathbf{b} \sim \mathcal{N}(A\boldsymbol{\mu} + \mathbf{b},\, A\Sigma A^T)$

Formula— Conditional Distribution

Partition $\mathbf{X} = (\mathbf{X}_1, \mathbf{X}_2)$ with $\boldsymbol{\mu} = (\boldsymbol{\mu}_1, \boldsymbol{\mu}_2)$ and . Then:

💡Tip

The conditional mean $\boldsymbol{\mu}_1 + \Sigma_{12}\Sigma_{22}^{-1}(\mathbf{x}_2 - \boldsymbol{\mu}_2)$ is linear in — this is the multivariate version of linear regression. The conditional variance (Schur complement) does not depend on .

4.3 Covariance Zero $\Rightarrow$ Independence (Gaussian case)

Theorem

Let $(X, Y)$ be a Gaussian vector. Then:

\boxed{\text{Cov}(X,Y) = 0 \iff X \text{ and } Y \text{ are independent}}

Remarque

This result is specific to Gaussian vectors. For general random variables, uncorrelated does not imply independent — the Gaussian structure is essential. In interviews, always verify that $(X,Y)$ is jointly Gaussian before concluding independence from zero covariance.

5. Quick Reference

Key Formulas

Concept	Formula
Joint density normalization	$\iint f_{X,Y}(x,y)\,dx\,dy = 1$
Marginal of $X$

Key Distinctions

Statement	True or False
Marginals determine joint distribution	❌ False in general
$X \perp Y \Rightarrow \text{Cov}(X,Y)=0$	✅ Always
$\text{Cov}(X,Y)=0 \Rightarrow X \perp Y$

Joint Distributions & Random Vectors

1. Joint Distributions (Couples)

1.1 Joint Density & Marginals

1.2 Conditional Density

1.3 Independence

1.4 Covariance & Correlation

2. Operations on Random Variables

2.1 Sum: Z=X+YZ = X + YZ=X+Y

2.2 Product: Z=X⋅YZ = X \cdot YZ=X⋅Y

2.3 Ratio: Z=X/YZ = X / YZ=X/Y

3. Random Vectors

3.1 Definition & Marginals

3.2 Covariance Matrix

3.3 Change of Variable (Multivariate)

4. Gaussian Vectors

4.1 Definition

4.2 Properties

4.3 Covariance Zero ⇒\Rightarrow⇒ Independence (Gaussian case)

5. Quick Reference

Key Formulas

Key Distinctions

2.1 Sum: $Z = X + Y$

2.2 Product: $Z = X \cdot Y$

2.3 Ratio: $Z = X / Y$

4.3 Covariance Zero $\Rightarrow$ Independence (Gaussian case)