TCSLib

107 Learning Theory — Rademacher

107.1 Overview

This module establishes the Johnson–Lindenstrauss concentration bound for the Rademacher (random \(\pm 1/\sqrt{k}\)) matrix. It proceeds in two layers: a distribution-agnostic sub-Gaussian concentration theorem (jl_concentration_single_subgaussian) is proved first, and then the explicit Rademacher construction is shown to satisfy its hypotheses, yielding the concrete bound \(\Pr [\varepsilon \lVert x\rVert ^2 \lt \left\lvert \lVert Ax\rVert ^2 - \lVert x\rVert ^2\right\rvert ] \leq 2\exp (-k\varepsilon ^2/8)\).

107.2 Declarations

Theorem 107.1 Distribution-agnostic JL single-vector concentration

Let \(A : \Omega \to \mathbb {R}^{k \times d}\) be a random matrix on a probability space, and let \(x \in \mathbb {R}^d\) be a nonzero vector. Suppose the \(k\) row projections \((Ax)_i\) are mutually independent, each integrable against \(e^{t(\cdot )}\) for every \(t \in \mathbb {R}\), each has sub-Gaussian MGF bounded by \(\exp \! \bigl((\lVert x\rVert ^2/k)\, t^2/2\bigr)\), and each satisfies \(\int (Ax)_i^2 \, d\mu = \lVert x\rVert ^2/k\). Then for every \(0 \lt \varepsilon \lt 1\),

\[ \Pr \! \bigl[\varepsilon \lVert x\rVert ^2 \lt \bigl|\lVert Ax\rVert ^2 - \lVert x\rVert ^2\bigr|\bigr] \; \leq \; 2\exp \! \Bigl(-\tfrac {k\varepsilon ^2}{8}\Bigr). \]
Definition 107.2 Rademacher distribution on \(\mathbb {R}\)
#

The Rademacher distribution \(\mathrm{Rad}\) on \(\mathbb {R}\) is the measure \(\tfrac {1}{2}\delta _{-1} + \tfrac {1}{2}\delta _{+1}\), placing equal mass \(1/2\) at \(-1\) and at \(+1\).

Lemma 107.3 \(1/2\) is finite as an extended non-negative real
#

The value \(1/2\), viewed as an element of \(\mathbb {R}_{\geq 0}^\infty \), is not equal to \(\top \).

Lemma 107.4 Rademacher support in \([-1,1]\)
#

Almost every sample from the Rademacher distribution lies in the interval \([-1, 1]\): \(\mathrm{Rad}\)-a.e. \(y \in [-1, 1]\).

Lemma 107.5 Integrability under a scaled Dirac measure
#

For any \(a \in \mathbb {R}\) and any measurable function \(f : \mathbb {R} \to \mathbb {R}\), the function \(f\) is integrable with respect to the measure \(\tfrac {1}{2}\, \delta _a\).

Lemma 107.6 Every function is integrable under the Rademacher measure
#

For any \(f : \mathbb {R} \to \mathbb {R}\), the function \(f\) is integrable with respect to the Rademacher measure \(\mathrm{Rad}\).

Lemma 107.7 Mean of the Rademacher distribution is zero
#

The expected value of the identity under the Rademacher distribution is zero: \(\int y \, d\mathrm{Rad}(y) = 0\).

Lemma 107.8 Second moment of the Rademacher distribution is one
#

The second moment of the Rademacher distribution equals one: \(\int y^2 \, d\mathrm{Rad}(y) = 1\).

Lemma 107.9 Identity is sub-Gaussian with parameter \(1\) under \(\mathrm{Rad}\)

The identity function \(\mathrm{id} : \mathbb {R} \to \mathbb {R}\) satisfies \(\mathrm{HasSubgaussianMGF}(\mathrm{id}, 1, \mathrm{Rad})\), i.e. its MGF is bounded by \(e^{t^2/2}\) for all \(t \in \mathbb {R}\). This follows from Hoeffding’s lemma applied to the centred random variable supported in \([-1, 1]\), whose sub-Gaussian parameter is \(\bigl((1-(-1))/2\bigr)^2 = 1\).

Definition 107.10 Rademacher matrix sample space
#

The sample space for the Rademacher matrix is \(\mathrm{Fin}\, k \to \mathrm{Fin}\, d \to \mathbb {R}\), i.e. the type of \(k \times d\) arrays of real numbers, indexed by \((i,j) \in \mathrm{Fin}\, k \times \mathrm{Fin}\, d\).

Definition 107.11 Joint Rademacher product measure
#

The joint measure on \(\mathrm{Rad\Omega }(k,d)\) is the product measure \(\bigotimes _{(i,j)} \mathrm{Rad}\), making all \(k \cdot d\) entries i.i.d. Rademacher.

Definition 107.12 Rademacher matrix
#

The Rademacher matrix \(A : \mathrm{Rad\Omega }(k,d) \to \mathbb {R}^{k\times d}\) maps a sample \(\omega \) to the matrix with entries \(A(\omega )_{ij} = \omega _{ij}/\sqrt{k}\), scaling the raw \(\pm 1\) entries by \(1/\sqrt{k}\).

Lemma 107.13 Measurability of the Rademacher matrix
#

The map \(\omega \mapsto \mathrm{radMatrix}(k,d,\omega )\) is measurable with respect to the product \(\sigma \)-algebra on \(\mathrm{Rad\Omega }(k,d)\).

Lemma 107.14 Row marginal of the joint measure

The pushforward of the joint measure along the \(i\)-th row projection \(\omega \mapsto \omega _i\) equals the product measure \(\bigotimes _{j \in \mathrm{Fin}\, d} \mathrm{Rad}\).

Lemma 107.15 Entry marginal of the joint measure

The pushforward of the joint measure along the \((i,j)\)-entry projection \(\omega \mapsto \omega _{ij}\) equals \(\mathrm{Rad}\).

Lemma 107.16 Within-row entries are i.i.d. Rademacher

For each fixed row \(i \in \mathrm{Fin}\, k\), the \(d\) functions \(j \mapsto \omega _{ij}\) are mutually independent under the joint measure.

Lemma 107.17 Rows are mutually independent

The \(k\) row vectors \(i \mapsto (\omega _{ij})_{j}\), viewed as \(\mathrm{Fin}\, d \to \mathbb {R}\)-valued random variables, are mutually independent under the joint measure.

Lemma 107.18 Each entry is sub-Gaussian with parameter \(1\)

For each \((i,j) \in \mathrm{Fin}\, k \times \mathrm{Fin}\, d\), the function \(\omega \mapsto \omega _{ij}\) satisfies \(\mathrm{HasSubgaussianMGF}(\cdot , 1, \mathrm{radJointMeasure})\) under the joint product measure.

Lemma 107.19 Scaled entry is sub-Gaussian with parameter \(x_j^2/k\)

For \(k \gt 0\), \(x \in \mathbb {R}^d\), and indices \(i,j\), the function \(\omega \mapsto (x_j / \sqrt{k})\, \omega _{ij}\) satisfies \(\mathrm{HasSubgaussianMGF}(\cdot , x_j^2/k, \mathrm{radJointMeasure})\).

Lemma 107.20 Row projection is sub-Gaussian with parameter \(\lVert x\rVert ^2/k\)

For \(k \gt 0\), \(x \in \mathbb {R}^d\), and \(i \in \mathrm{Fin}\, k\), the \(i\)-th row projection \(\omega \mapsto (A(\omega )\, x)_i = (1/\sqrt{k})\sum _j \omega _{ij} x_j\) satisfies \(\mathrm{HasSubgaussianMGF}(\cdot , \lVert x\rVert ^2/k, \mathrm{radJointMeasure})\). This follows by combining per-summand Hoeffding bounds with independence of the within-row entries.

Lemma 107.21 Variance of \(\mathrm{id}\) under \(\mathrm{Rad}\) is \(1\)

The variance of the identity function under the Rademacher distribution is \(1\): \(\mathrm{Var}[\mathrm{id}; \mathrm{Rad}] = 1\).

Lemma 107.22 Variance of a single entry is \(1\)

For any \((i,j)\), the variance of the entry \(\omega \mapsto \omega _{ij}\) under the joint measure is \(1\): \(\mathrm{Var}[\omega \mapsto \omega _{ij}; \mathrm{radJointMeasure}] = 1\).

Lemma 107.23 Mean of a single entry is \(0\)

For any \((i,j)\), the expected value of \(\omega _{ij}\) under the joint measure is \(0\): \(\int \omega _{ij} \, d\mathrm{radJointMeasure} = 0\).

Lemma 107.24 Scaled entry belongs to \(L^2\)

For \(k \gt 0\), \(x \in \mathbb {R}^d\), and indices \(i,j\), the function \(\omega \mapsto (x_j/\sqrt{k})\, \omega _{ij}\) belongs to \(L^2(\mathrm{radJointMeasure})\).

Lemma 107.25 Variance of a scaled entry is \(x_j^2/k\)

For \(k \gt 0\), \(x \in \mathbb {R}^d\), and indices \(i,j\), \(\mathrm{Var}[\omega \mapsto (x_j/\sqrt{k})\, \omega _{ij}; \mathrm{radJointMeasure}] = x_j^2/k\).

Lemma 107.26 Variance of the row projection is \(\lVert x\rVert ^2/k\)

For \(k \gt 0\), \(x \in \mathbb {R}^d\), and \(i \in \mathrm{Fin}\, k\),

\[ \mathrm{Var}\! \bigl[\omega \mapsto (A(\omega )\, x)_i;\, \mathrm{radJointMeasure}\bigr] = \frac{\lVert x\rVert ^2}{k}. \]

This follows from per-summand variance \(x_j^2/k\) together with within-row independence.

Lemma 107.27 Mean of the row projection is \(0\)

For \(k \gt 0\), \(x \in \mathbb {R}^d\), and \(i \in \mathrm{Fin}\, k\), \(\int (A(\omega )\, x)_i \, d\mathrm{radJointMeasure} = 0\).

Lemma 107.28 Second moment of the row projection equals \(\lVert x\rVert ^2/k\)

For \(k \gt 0\), \(x \in \mathbb {R}^d\), and \(i \in \mathrm{Fin}\, k\),

\[ \int \bigl((A(\omega )\, x)_i\bigr)^2 \, d\mathrm{radJointMeasure} = \frac{\lVert x\rVert ^2}{k}. \]

This follows from \(\int Y^2 = \mathrm{Var}[Y] + (\int Y)^2 = \lVert x\rVert ^2/k + 0\).

Lemma 107.29 Row projections are mutually independent

For any \(x \in \mathbb {R}^d\), the \(k\) row projections \(i \mapsto (A(\omega )\, x)_i\) are mutually independent random variables under the joint Rademacher measure.

Lemma 107.30 Row projections are measurable
#

For any \(x \in \mathbb {R}^d\) and \(i \in \mathrm{Fin}\, k\), the map \(\omega \mapsto (A(\omega )\, x)_i\) is measurable.

Theorem 107.31 JL concentration for the Rademacher matrix

Let \(k, d \geq 1\) and let \(A\) be the Rademacher matrix \(\mathrm{radMatrix}(k,d)\) on the joint \(\pm 1\) sample space. For every nonzero \(x \in \mathbb {R}^d\) and every \(0 \lt \varepsilon \lt 1\),

\[ \Pr _{\mathrm{radJointMeasure}}\! \bigl[ \varepsilon \lVert x\rVert ^2 \lt \bigl|\lVert A\omega \, x\rVert ^2 - \lVert x\rVert ^2\bigr| \bigr] \; \leq \; 2\exp \! \Bigl(-\frac{k\varepsilon ^2}{8}\Bigr). \]

This recovers Achlioptas’s \(\pm 1\)-entries variant of Johnson–Lindenstrauss with the Dasgupta–Gupta exponent.