107 Learning Theory — Rademacher
107.1 Overview
This module establishes the Johnson–Lindenstrauss concentration bound for the Rademacher (random \(\pm 1/\sqrt{k}\)) matrix. It proceeds in two layers: a distribution-agnostic sub-Gaussian concentration theorem (jl_concentration_single_subgaussian) is proved first, and then the explicit Rademacher construction is shown to satisfy its hypotheses, yielding the concrete bound \(\Pr [\varepsilon \lVert x\rVert ^2 \lt \left\lvert \lVert Ax\rVert ^2 - \lVert x\rVert ^2\right\rvert ] \leq 2\exp (-k\varepsilon ^2/8)\).
107.2 Declarations
Let \(A : \Omega \to \mathbb {R}^{k \times d}\) be a random matrix on a probability space, and let \(x \in \mathbb {R}^d\) be a nonzero vector. Suppose the \(k\) row projections \((Ax)_i\) are mutually independent, each integrable against \(e^{t(\cdot )}\) for every \(t \in \mathbb {R}\), each has sub-Gaussian MGF bounded by \(\exp \! \bigl((\lVert x\rVert ^2/k)\, t^2/2\bigr)\), and each satisfies \(\int (Ax)_i^2 \, d\mu = \lVert x\rVert ^2/k\). Then for every \(0 \lt \varepsilon \lt 1\),
The Rademacher distribution \(\mathrm{Rad}\) on \(\mathbb {R}\) is the measure \(\tfrac {1}{2}\delta _{-1} + \tfrac {1}{2}\delta _{+1}\), placing equal mass \(1/2\) at \(-1\) and at \(+1\).
The value \(1/2\), viewed as an element of \(\mathbb {R}_{\geq 0}^\infty \), is not equal to \(\top \).
Almost every sample from the Rademacher distribution lies in the interval \([-1, 1]\): \(\mathrm{Rad}\)-a.e. \(y \in [-1, 1]\).
For any \(a \in \mathbb {R}\) and any measurable function \(f : \mathbb {R} \to \mathbb {R}\), the function \(f\) is integrable with respect to the measure \(\tfrac {1}{2}\, \delta _a\).
For any \(f : \mathbb {R} \to \mathbb {R}\), the function \(f\) is integrable with respect to the Rademacher measure \(\mathrm{Rad}\).
The expected value of the identity under the Rademacher distribution is zero: \(\int y \, d\mathrm{Rad}(y) = 0\).
The second moment of the Rademacher distribution equals one: \(\int y^2 \, d\mathrm{Rad}(y) = 1\).
The identity function \(\mathrm{id} : \mathbb {R} \to \mathbb {R}\) satisfies \(\mathrm{HasSubgaussianMGF}(\mathrm{id}, 1, \mathrm{Rad})\), i.e. its MGF is bounded by \(e^{t^2/2}\) for all \(t \in \mathbb {R}\). This follows from Hoeffding’s lemma applied to the centred random variable supported in \([-1, 1]\), whose sub-Gaussian parameter is \(\bigl((1-(-1))/2\bigr)^2 = 1\).
The sample space for the Rademacher matrix is \(\mathrm{Fin}\, k \to \mathrm{Fin}\, d \to \mathbb {R}\), i.e. the type of \(k \times d\) arrays of real numbers, indexed by \((i,j) \in \mathrm{Fin}\, k \times \mathrm{Fin}\, d\).
The joint measure on \(\mathrm{Rad\Omega }(k,d)\) is the product measure \(\bigotimes _{(i,j)} \mathrm{Rad}\), making all \(k \cdot d\) entries i.i.d. Rademacher.
The Rademacher matrix \(A : \mathrm{Rad\Omega }(k,d) \to \mathbb {R}^{k\times d}\) maps a sample \(\omega \) to the matrix with entries \(A(\omega )_{ij} = \omega _{ij}/\sqrt{k}\), scaling the raw \(\pm 1\) entries by \(1/\sqrt{k}\).
The map \(\omega \mapsto \mathrm{radMatrix}(k,d,\omega )\) is measurable with respect to the product \(\sigma \)-algebra on \(\mathrm{Rad\Omega }(k,d)\).
The pushforward of the joint measure along the \(i\)-th row projection \(\omega \mapsto \omega _i\) equals the product measure \(\bigotimes _{j \in \mathrm{Fin}\, d} \mathrm{Rad}\).
The pushforward of the joint measure along the \((i,j)\)-entry projection \(\omega \mapsto \omega _{ij}\) equals \(\mathrm{Rad}\).
For each fixed row \(i \in \mathrm{Fin}\, k\), the \(d\) functions \(j \mapsto \omega _{ij}\) are mutually independent under the joint measure.
The \(k\) row vectors \(i \mapsto (\omega _{ij})_{j}\), viewed as \(\mathrm{Fin}\, d \to \mathbb {R}\)-valued random variables, are mutually independent under the joint measure.
For each \((i,j) \in \mathrm{Fin}\, k \times \mathrm{Fin}\, d\), the function \(\omega \mapsto \omega _{ij}\) satisfies \(\mathrm{HasSubgaussianMGF}(\cdot , 1, \mathrm{radJointMeasure})\) under the joint product measure.
For \(k \gt 0\), \(x \in \mathbb {R}^d\), and indices \(i,j\), the function \(\omega \mapsto (x_j / \sqrt{k})\, \omega _{ij}\) satisfies \(\mathrm{HasSubgaussianMGF}(\cdot , x_j^2/k, \mathrm{radJointMeasure})\).
For \(k \gt 0\), \(x \in \mathbb {R}^d\), and \(i \in \mathrm{Fin}\, k\), the \(i\)-th row projection \(\omega \mapsto (A(\omega )\, x)_i = (1/\sqrt{k})\sum _j \omega _{ij} x_j\) satisfies \(\mathrm{HasSubgaussianMGF}(\cdot , \lVert x\rVert ^2/k, \mathrm{radJointMeasure})\). This follows by combining per-summand Hoeffding bounds with independence of the within-row entries.
The variance of the identity function under the Rademacher distribution is \(1\): \(\mathrm{Var}[\mathrm{id}; \mathrm{Rad}] = 1\).
For any \((i,j)\), the variance of the entry \(\omega \mapsto \omega _{ij}\) under the joint measure is \(1\): \(\mathrm{Var}[\omega \mapsto \omega _{ij}; \mathrm{radJointMeasure}] = 1\).
For any \((i,j)\), the expected value of \(\omega _{ij}\) under the joint measure is \(0\): \(\int \omega _{ij} \, d\mathrm{radJointMeasure} = 0\).
For \(k \gt 0\), \(x \in \mathbb {R}^d\), and indices \(i,j\), the function \(\omega \mapsto (x_j/\sqrt{k})\, \omega _{ij}\) belongs to \(L^2(\mathrm{radJointMeasure})\).
For \(k \gt 0\), \(x \in \mathbb {R}^d\), and indices \(i,j\), \(\mathrm{Var}[\omega \mapsto (x_j/\sqrt{k})\, \omega _{ij}; \mathrm{radJointMeasure}] = x_j^2/k\).
For \(k \gt 0\), \(x \in \mathbb {R}^d\), and \(i \in \mathrm{Fin}\, k\),
This follows from per-summand variance \(x_j^2/k\) together with within-row independence.
For \(k \gt 0\), \(x \in \mathbb {R}^d\), and \(i \in \mathrm{Fin}\, k\), \(\int (A(\omega )\, x)_i \, d\mathrm{radJointMeasure} = 0\).
For \(k \gt 0\), \(x \in \mathbb {R}^d\), and \(i \in \mathrm{Fin}\, k\),
This follows from \(\int Y^2 = \mathrm{Var}[Y] + (\int Y)^2 = \lVert x\rVert ^2/k + 0\).
For any \(x \in \mathbb {R}^d\), the \(k\) row projections \(i \mapsto (A(\omega )\, x)_i\) are mutually independent random variables under the joint Rademacher measure.
For any \(x \in \mathbb {R}^d\) and \(i \in \mathrm{Fin}\, k\), the map \(\omega \mapsto (A(\omega )\, x)_i\) is measurable.
Let \(k, d \geq 1\) and let \(A\) be the Rademacher matrix \(\mathrm{radMatrix}(k,d)\) on the joint \(\pm 1\) sample space. For every nonzero \(x \in \mathbb {R}^d\) and every \(0 \lt \varepsilon \lt 1\),
This recovers Achlioptas’s \(\pm 1\)-entries variant of Johnson–Lindenstrauss with the Dasgupta–Gupta exponent.