TCSLib

105 Learning Theory — Concentration Bound

105.1 Overview

This module proves the single-vector concentration bound for the Johnson–Lindenstrauss random projection. A random matrix \(A\) with i.i.d. \(N(0,1/k)\) entries is shown to preserve the squared norm of any fixed vector \(x\) up to a multiplicative \((1\pm \varepsilon )\) factor, with failure probability at most \(2\exp (-k\varepsilon ^2/8)\). The proof reduces to a centered chi-squared moment-generating-function (MGF) bound obtained via a classical Taylor inequality, then packages the result into the abstract Bernstein-MGF framework so that the same tail machinery applies to other sub-Gaussian random matrix families.

105.2 Declarations

Definition 105.1 Bad distortion event for a single vector
#

For a parameter \(\varepsilon \in \mathbb {R}\), a matrix \(A\), and a vector \(x\in \mathbb {R}^d\), the predicate \(\texttt{BadSingle}\, \varepsilon \, A\, x\) holds when the squared-norm distortion is strictly too large:

\[ \varepsilon \| x\| ^2 \lt \bigl|\, \| Ax\| ^2 - \| x\| ^2\, \bigr|. \]

This is the “bad event” whose probability must be controlled by the JL concentration argument.

Lemma 105.2 Zero vector has probability zero of being bad
#

For any probability measure \(\mu \) on \(\Omega \), any random matrix \(A\), and any \(\varepsilon \in \mathbb {R}\),

\[ \mu \bigl\{ \omega \mid \texttt{BadSingle}\, \varepsilon \, (A\, \omega )\, 0\bigr\} = 0 \; \le \; 2\exp \! \Bigl(-\frac{k\, \varepsilon ^2}{8}\Bigr). \]

The bad event at \(x=0\) is always empty because \(\varepsilon \cdot 0 \lt |0-0|\) simplifies to \(0 \lt 0\), which is false.

Lemma 105.3 Scalar multiple of a centered Gaussian
#

Let \(X : \Omega \to \mathbb {R}\) be measurable with \(X_*\mu = N(0,\sigma _0)\). Then for any constant \(c\in \mathbb {R}\),

\[ (c\, X)_*\mu \; =\; N\! \bigl(0,\, c^2\cdot \sigma _0\bigr). \]

This is an auxiliary restatement of gaussianReal_map_const_mul phrased for easy application in the inductive step of sum_scaled_iid_gaussian_map.

Lemma 105.4 Row-projection of i.i.d. Gaussians is Gaussian
#

Let \(Y_j : \Omega \to \mathbb {R}\) be i.i.d. with law \(N(0,\sigma _0)\), and let \(x : \iota \to \mathbb {R}\) be a fixed coefficient vector. For any finite set \(s\),

\[ \Bigl(\textstyle \sum _{j\in s} x_j Y_j\Bigr)_*\mu \; =\; N\! \Bigl(0,\; \sigma _0 \cdot \textstyle \sum _{j\in s} x_j^2\Bigr). \]

The proof proceeds by induction on the finset \(s\), using independence and map_const_mul_gaussian at each step.

Lemma 105.5 Row projections are independent
#

If the row vectors of \(A\) (as \((\mathrm{Fin}\, d\to \mathbb {R})\)-valued random variables) are mutually independent under \(\mu \), then for any fixed \(x\in \mathbb {R}^d\) the scalar row-projections \((A\, \omega ).\texttt{toEuclideanLin}\, x\, i = \sum _j A_{\omega ,i,j}\, x_j\) are independent across \(i\in \mathrm{Fin}\, k\), since each is a measurable function of a distinct row of \(A\).

Lemma 105.6 Coordinate-sum form of a row projection
#

For a matrix \(A\in \mathbb {R}^{k\times d}\), a vector \(x\in \mathbb {R}^d\), and an index \(i\in \mathrm{Fin}\, k\),

\[ (A.\texttt{toEuclideanLin}\, x)\, i \; =\; \sum _{j} A_{i,j}\, x_j. \]

This definitional equality unfolds the linear map into its explicit sum form.

Lemma 105.7 Squared norm as sum of squared row projections
#

For a matrix \(A\) and vector \(x\),

\[ \| A.\texttt{toEuclideanLin}\, x\| ^2 \; =\; \sum _{i\in \mathrm{Fin}\, k} \bigl((A.\texttt{toEuclideanLin}\, x)\, i\bigr)^2. \]

This expresses the squared Euclidean norm of \(Ax\) as the sum of the squares of its coordinates.

Lemma 105.8 Derivative of Taylor auxiliary function
#

The function \(h(u) = u^2 + u + \log (1-u)\) has derivative \(h'(u) = u(1-2u)/(1-u)\) at every \(u \lt 1\). This derivative identity is the key ingredient in the monotonicity argument used to bound the centered chi-squared log-MGF.

Lemma 105.9 Taylor auxiliary function vanishes at zero
#

The evaluation \(0^2 + 0 + \log (1-0) = 0\) establishes the base case for the Taylor bound, confirming that \(h(0)=0\) in the derivative argument for neg_log_one_sub_two_mul_le_two_sq.

Lemma 105.10 Taylor bound for the centered chi-squared log-MGF

For every \(s\in \mathbb {R}\) with \(|s|\le 1/4\),

\[ -s - \tfrac {1}{2}\log (1 - 2s) \; \le \; 2s^2. \]

This inequality is the Taylor-based analytic core of the centered chi-squared MGF bound; it is proved by showing that the auxiliary function \(h(u)=u^2+u+\log (1-u)\) satisfies \(h(0)=0\) and has non-negative derivative on \([0,1/2]\) and non-positive derivative on \([-1/2,0]\).

Lemma 105.11 Quadratic MGF of the standard Gaussian
#

For \(Z\sim N(0,1)\) and \(2s\lt 1\),

\[ \int z.\, e^{s z^2}\, \mathrm{d}(N(0,1)) \; =\; \frac{1}{\sqrt{1-2s}}. \]

This is the closed-form moment-generating function of the standard chi-squared distribution.

Lemma 105.12 Quadratic MGF of a general centered Gaussian
#

For \(Y\sim N(0,v)\) with \(v\ne 0\) and \(2tv \lt 1\),

\[ \int y.\, e^{t y^2}\, \mathrm{d}(N(0,v)) \; =\; \frac{1}{\sqrt{1-2tv}}. \]

This generalizes integral_exp_mul_sq_standardGaussian from unit variance to an arbitrary non-zero NNReal variance \(v\).

Lemma 105.13 Integrability of \(e^{ty^2}\) under \(N(0,v)\)
#

For \(v\ne 0\) and \(2tv \lt 1\), the function \(y\mapsto e^{t y^2}\) is integrable with respect to \(N(0,v)\). This integrability statement accompanies integral_exp_mul_sq_gaussianReal_zero and is a prerequisite for the Fubini and interchange-of-expectation arguments in centered_chi_squared_step.

Theorem 105.14 Centered chi-squared MGF bound

Let \((\Omega ,\mu )\) be a probability space, \(k\gt 0\), and let \(Y:\Omega \to \mathbb {R}\) be measurable with \(Y_*\mu = N(0,1/k)\). For every \(t\in \mathbb {R}\) with \(|t|\le k/4\), the map \(\omega \mapsto \exp \bigl(t(Y(\omega )^2-1/k)\bigr)\) is \(\mu \)-integrable and

\[ \mathbb {E}_\mu \! \left[e^{t(Y^2 - 1/k)}\right] \; \le \; \exp \! \Bigl(\frac{2t^2}{k^2}\Bigr). \]

The proof combines the Gaussian quadratic MGF closed form (integral_exp_mul_sq_gaussianReal_zero) with the Taylor inequality (neg_log_one_sub_two_mul_le_two_sq) applied at \(s := t/k\).

Theorem 105.15 Bernstein MGF instance for centered chi-squared

Under the same hypotheses as centered_chi_squared_step, the centered summand \(Y^2 - 1/k\) satisfies \(\text{\texttt{ProbabilityTheory.HasBernsteinMGF}}\bigl(2/k^2,\; k/4\bigr)\) under \(\mu \). This packages the chi-squared MGF bound into the abstract Bernstein form, so that the i.i.d.-sum tail-bound machinery of bernstein.lean can be applied uniformly to Gaussian and other sub-Gaussian matrix families.

Lemma 105.16 Chi-squared tail bound for i.i.d. \(N(0,1/k)\) variables

Let \(Y_1,\dots ,Y_k:\Omega \to \mathbb {R}\) be i.i.d. with law \(N(0,1/k)\) under a probability measure \(\mu \), and let \(0\lt \varepsilon \lt 1\). Then

\[ \mu \! \left\{ \omega \; \middle |\; \varepsilon \lt \Bigl|\textstyle \sum _{i=1}^k Y_i(\omega )^2 - 1\Bigr|\right\} \; \le \; 2\exp \! \Bigl(-\frac{k\, \varepsilon ^2}{8}\Bigr). \]

The proof applies the Bernstein tail bound to the i.i.d. sum of centered chi-squared summands, using hasBernsteinMGF_centered_chi_squared and the generic sum and tail-bound lemmas from the Bernstein framework.

Theorem 105.17 JL single-vector concentration via chi-squared reduction

Let \(A:\Omega \to \mathbb {R}^{k\times d}\) be a random matrix under a probability measure \(\mu \) whose entries \(A_{ij}\) are each \(N(0,1/k)\) and are jointly independent (row entries and row vectors alike). For any fixed \(x\in \mathbb {R}^d\) and \(0\lt \varepsilon \lt 1\),

\[ \mu \bigl\{ \omega \mid \texttt{BadSingle}\, \varepsilon \, (A\, \omega )\, x\bigr\} \; \le \; 2\exp \! \Bigl(-\frac{k\, \varepsilon ^2}{8}\Bigr). \]

The proof handles \(x=0\) via concentration_zero and the general case by showing each row projection \(\sum _j A_{ij}\, x_j\) is Gaussian with variance \(\| x\| ^2/k\), establishing independence of row projections via rows_indep, and then applying chi_squared_tail.

Theorem 105.18 Distribution-agnostic JL concentration via Bernstein tails

Let \(A:\Omega \to \mathbb {R}^{k\times d}\) be a random matrix whose row projections \((A\, \omega ).\texttt{toEuclideanLin}\, x\, i\) are measurable, independent across \(i\), and such that each centered squared summand \(((A\, \omega ).\texttt{toEuclideanLin}\, x\, i)^2 - \| x\| ^2/k\) satisfies \(\text{\texttt{ProbabilityTheory.HasBernsteinMGF}}(c, t_{\max })\) for some \(c\gt 0\). For any \(s\ge 0\) with \(s\le 2k\, c\, t_{\max }\),

\[ \mu \! \left\{ \omega \; \middle |\; s \lt \bigl|\, \| (A\, \omega ).\texttt{toEuclideanLin}\, x\| ^2 - \| x\| ^2\, \bigr| \right\} \; \le \; 2\exp \! \Bigl(-\frac{s^2}{4k\, c}\Bigr). \]

This theorem makes no assumption on the distribution of the matrix entries beyond the Bernstein MGF condition, giving a general tail bound for the squared-norm distortion.