TCSLib

103 Learning Theory — Johnson–Lindenstrauss Main

103.1 Overview

This module assembles the complete Johnson–Lindenstrauss flattening lemma. Starting from the distortion predicate and the bad-event definitions, it applies a Gaussian (or sub-Gaussian Rademacher) concentration bound and a union bound to show, via the probabilistic method, that for any finite set \(V \subseteq \mathbb {R}^d\) of at most \(n\) points and any \(0 \lt \varepsilon \lt 1\), a random \(k \times d\) matrix with \(k \ge 32 \log n / \varepsilon ^2\) rows is, with positive probability, an \(\varepsilon \)-JL embedding of \(V\); it then extracts the deterministic witness and provides both squared-distance and distance-form corollaries together with explicit logarithmic dimension bounds.

103.2 Declarations

Definition 103.1 Single-pair distortion bound
✓
#

For \(\varepsilon \in \mathbb {R}\), vectors \(u, v \in \mathbb {R}^d\), and their images \(u', v' \in \mathbb {R}^k\), JLDistortion asserts the two-sided squared-distance inequality

\[ (1 - \varepsilon )\, \| u - v\| ^2 \; \le \; \| u' - v'\| ^2 \; \le \; (1 + \varepsilon )\, \| u - v\| ^2. \]
Definition 103.2 \(\varepsilon \)-JL embedding
✓
#

A linear map \(f : \mathbb {R}^d \to _L[\mathbb {R}] \mathbb {R}^k\) is an \(\varepsilon \)-JL embedding of a finite set \(V\) if, for every ordered pair \((u, v) \in V \times V\), the pair \((f(u), f(v))\) satisfies JLDistortion \(\varepsilon \; u\; v\).

Definition 103.3 Bad-pair event
✓
#

Given a \(k \times d\) matrix \(A\) and a finite set \(V \subseteq \mathbb {R}^d\), BadPair \(\varepsilon \; V\; A\) holds when there exists some ordered pair \((u, v) \in V \times V\) whose images under \(A\) fail the distortion predicate, i.e. the projection distorts that pair by more than a factor of \(\varepsilon \).

Theorem 103.4 Johnson–Lindenstrauss concentration for a single vector
✓
#

Let \(k \ge 1\) and let \(A\) be a random \(k \times d\) matrix whose entries \(A_{ij}\) are independent Gaussian random variables, each distributed as \(\mathcal{N}(0, 1/k)\). Then for every fixed vector \(x \in \mathbb {R}^d\) and every \(\varepsilon \) with \(0 \lt \varepsilon \lt 1\), the probability that \(A\) distorts the squared norm of \(x\) by more than a factor \(\varepsilon \) satisfies

\[ \mathbb {P}\! \left[\, \bigl|\, \| Ax\| ^2 - \| x\| ^2\, \bigr| \gt \varepsilon \, \| x\| ^2 \, \right] \; \le \; 2\, \exp \! \left(-\frac{k\, \varepsilon ^2}{8}\right). \]
Lemma 103.5 Distortion bound from a controlled single-vector event
✓

Let \(\varepsilon \in \mathbb {R}\), let \(u, v \in \mathbb {R}^d\), and let \(A\) be a real \(k \times d\) matrix, acting as a linear map \(\mathbb {R}^d \to \mathbb {R}^k\). Suppose the difference \(u - v\) does not trigger the bad distortion event for \(A\); that is, the squared-norm distortion of \(u-v\) stays within the allowed band,

\[ \bigl|\, \lVert A(u-v)\rVert ^2 - \lVert u-v\rVert ^2\, \bigr| \; \le \; \varepsilon \, \lVert u-v\rVert ^2 . \]

Then the images \(Au\) and \(Av\) satisfy the two-sided single-pair distortion bound

\[ (1 - \varepsilon )\, \lVert u - v\rVert ^2 \; \le \; \lVert Au - Av\rVert ^2 \; \le \; (1 + \varepsilon )\, \lVert u - v\rVert ^2 . \]
Theorem 103.6 Union bound for Johnson–Lindenstrauss distortion
✓

Let \((\Omega ,\mu )\) be a probability space, and let \(A\colon \Omega \to \mathbb {R}^{k\times d}\) be a measurable random \(k\times d\) matrix. Fix a finite set \(V\subseteq \mathbb {R}^d\) and a parameter \(\varepsilon \in \mathbb {R}\), and suppose that for every pair \(u,v\in V\) the single-vector bad event for the difference \(u-v\) has probability at most \(2\exp (-k\varepsilon ^2/8)\), that is,

\[ \mu \bigl\{ \, \omega : \varepsilon \, \| u-v\| ^2 \lt \bigl|\, \| A(\omega )(u-v)\| ^2 - \| u-v\| ^2\, \bigr|\, \bigr\} \le 2\exp \! \bigl(-k\varepsilon ^2/8\bigr). \]

Then the probability of the bad-pair event — that some ordered pair \((u,v)\in V\times V\) fails the two-sided distortion bound \((1-\varepsilon )\| u-v\| ^2 \le \| A(\omega )(u-v)\| ^2 \le (1+\varepsilon )\| u-v\| ^2\) — is at most

\[ |V|^2 \cdot 2\exp \! \bigl(-k\varepsilon ^2/8\bigr). \]
Theorem 103.7 Johnson–Lindenstrauss embedding from a Gaussian matrix
✓

Fix dimensions \(d\) and \(k\) with \(k \ge 1\), a distortion parameter \(\varepsilon \in (0,1)\), and a finite set of points \(V \subseteq \mathbb {R}^d\). Let \(A\) be a random \(k \times d\) matrix whose entries \(A_{ij}\) are independent Gaussian random variables, each with mean \(0\) and variance \(1/k\). If the union-bound quantity satisfies

\[ |V|^2 \cdot 2\exp \! \left(-\, k\varepsilon ^2/8\right) \lt 1, \]

then there exists a linear map \(f : \mathbb {R}^d \to \mathbb {R}^k\) that is an \(\varepsilon \)-JL embedding of \(V\): for every pair \(u, v \in V\),

\[ (1-\varepsilon )\, \| u-v\| ^2 \; \le \; \| f(u)-f(v)\| ^2 \; \le \; (1+\varepsilon )\, \| u-v\| ^2. \]
Theorem 103.8 Sub-Gaussian random projections give a Johnson–Lindenstrauss embedding
✓

Let \(k \ge 1\), and let \(A\) be a random \(k \times d\) real matrix defined on a probability space \((\Omega ,\mu )\), so that for each \(x \in \mathbb {R}^d\) the image \(Ax \in \mathbb {R}^k\) has coordinates \((Ax)_1,\dots ,(Ax)_k\). Suppose that for every \(x \in \mathbb {R}^d\) these \(k\) coordinates are independent, and that each coordinate \((Ax)_i\) is sub-Gaussian with variance proxy \(\| x\| ^2/k\), in the sense that \(t \mapsto e^{t(Ax)_i}\) is integrable and its moment generating function obeys \(\mathbb {E}\! \left[e^{t(Ax)_i}\right] \le \exp \! \bigl(\tfrac {\| x\| ^2}{k}\cdot \tfrac {t^2}{2}\bigr)\) for all \(t \in \mathbb {R}\), and moreover has second moment \(\mathbb {E}\! \left[(Ax)_i^2\right] = \| x\| ^2/k\). Fix \(0 \lt \varepsilon \lt 1\) and a finite set \(V \subseteq \mathbb {R}^d\) satisfying the union bound \(|V|^2 \cdot 2\exp (-k\varepsilon ^2/8) \lt 1\). Then there exists a linear map \(f : \mathbb {R}^d \to \mathbb {R}^k\) that is an \(\varepsilon \)-JL embedding of \(V\): for all \(u,v \in V\),

\[ (1-\varepsilon )\, \| u-v\| ^2 \; \le \; \| f(u)-f(v)\| ^2 \; \le \; (1+\varepsilon )\, \| u-v\| ^2. \]
Lemma 103.9 Measurability of the single-vector distortion event
✓
#

Fix a vector \(x\in \mathbb {R}^d\) and a real number \(\varepsilon \), and let \((\Omega ,\mathcal{F})\) be a measurable space. Suppose \(A\colon \Omega \to \mathbb {R}^{k\times d}\) is a measurable map assigning to each \(\omega \) a \(k\times d\) real matrix \(A(\omega )\). Then the set of outcomes on which \(x\) is badly distorted,

\[ \bigl\{ \, \omega \in \Omega \; :\; \varepsilon \, \| x\| ^2 \lt \bigl|\, \| A(\omega )\, x\| ^2 - \| x\| ^2\, \bigr|\, \bigr\} , \]

is measurable.

Lemma 103.10 Existence of a Gaussian matrix with independent entries
✓
#

Let \(k \ge 1\) and \(d \ge 0\) be integers. Then there is a probability space \((\Omega , \mathcal{F}, \mu )\) carrying a measurable map \(\omega \mapsto A(\omega )\), valued in the \(k \times d\) real matrices, with the following properties: each entry \(A_{ij}\), for \(1 \le i \le k\) and \(1 \le j \le d\), has law \(\mathcal{N}(0, 1/k)\) (mean zero and variance \(1/k\)); for each row index \(i\) the family of entries \((A_{ij})_{j}\) is jointly independent; and the family of rows \(\bigl((A_{ij})_{j}\bigr)_{i}\) is jointly independent.

Lemma 103.11 Numerical failure-probability bound for Johnson–Lindenstrauss
✓
#

Let \(\varepsilon \gt 0\) and let \(n \ge 2\) be an integer, and suppose the target dimension \(k\) satisfies \(k \ge 32 \log n / \varepsilon ^2\). Then for any finite set \(V\) of points in a Euclidean space with \(\left\lvert V\right\rvert \le n\), one has \(k \gt 0\) and

\[ \left\lvert V\right\rvert ^2 \cdot 2\exp \! \left(-\tfrac {k\varepsilon ^2}{8}\right) \lt 1. \]
Theorem 103.12 Johnson–Lindenstrauss flattening lemma
✓

Let \(0 \lt \varepsilon \lt 1\) and let \(n \ge 2\) be an integer. Suppose the target dimension \(k\) satisfies \(k \ge 32\, \log n / \varepsilon ^2\). Then for every finite set \(V \subseteq \mathbb {R}^d\) of at most \(n\) points there exists a linear map \(f : \mathbb {R}^d \to \mathbb {R}^k\) that is an \(\varepsilon \)-JL embedding of \(V\); that is,

\[ (1 - \varepsilon )\, \lVert u - v\rVert ^2 \; \le \; \lVert f(u) - f(v)\rVert ^2 \; \le \; (1 + \varepsilon )\, \lVert u - v\rVert ^2 \]

for all \(u, v \in V\).

Let \(0 \lt \varepsilon \lt 1\), let \(n \ge 2\) be an integer, and let \(k\) be a target dimension satisfying \(k \ge 32\, \log n / \varepsilon ^2\). Then for every finite set \(V\) of at most \(n\) points in \(\mathbb {R}^d\) there exists a linear map \(f \colon \mathbb {R}^d \to \mathbb {R}^k\) such that every pair \(u, v \in V\) has its squared distance preserved up to a factor \(\varepsilon \):

\[ (1 - \varepsilon )\, \| u - v\| ^2 \; \le \; \| f(u) - f(v)\| ^2 \; \le \; (1 + \varepsilon )\, \| u - v\| ^2. \]
Lemma 103.14 Distance form of a JL embedding
✓
#

Let \(0 \lt \varepsilon \lt 1\), let \(V\) be a finite set of points in \(\mathbb {R}^d\), and let \(f : \mathbb {R}^d \to \mathbb {R}^k\) be a linear map that is an \(\varepsilon \)-JL embedding of \(V\), so that for every pair \(u, v \in V\) one has \((1-\varepsilon )\, \| u-v\| ^2 \le \| f(u)-f(v)\| ^2 \le (1+\varepsilon )\, \| u-v\| ^2\). Then for all \(u, v \in V\),

\[ \sqrt{1-\varepsilon }\, \| u - v\| \; \le \; \| f(u) - f(v)\| \; \le \; \sqrt{1+\varepsilon }\, \| u - v\| . \]
Theorem 103.15 Johnson–Lindenstrauss flattening in distance form
✓
#

Let \(\varepsilon \in (0,1)\) and let \(n \ge 2\) be an integer, and suppose the target dimension \(k\) satisfies \(k \ge 32\, \log n / \varepsilon ^2\). Then for every finite set \(V \subseteq \mathbb {R}^d\) of at most \(n\) points there exists a linear map \(f : \mathbb {R}^d \to \mathbb {R}^k\) such that, for all \(u, v \in V\),

\[ \sqrt{1-\varepsilon }\, \lVert u - v\rVert \; \le \; \lVert f(u) - f(v)\rVert \; \le \; \sqrt{1+\varepsilon }\, \lVert u - v\rVert , \]

where \(\lVert \cdot \rVert \) denotes the Euclidean norm.

Theorem 103.16 Johnson–Lindenstrauss flattening: distance form
✓

Let \(\varepsilon \in (0,1)\) and let \(n \ge 2\) be an integer. Suppose \(k\) is a natural number with \(k \ge 32\, (\log n)/\varepsilon ^2\), and let \(V\) be a finite set of at most \(n\) points in \(\mathbb {R}^d\). Then there exists a linear map \(f : \mathbb {R}^d \to \mathbb {R}^k\) such that for all \(u, v \in V\),

\[ \sqrt{1-\varepsilon }\, \lVert u-v\rVert \; \le \; \lVert f(u)-f(v)\rVert \; \le \; \sqrt{1+\varepsilon }\, \lVert u-v\rVert . \]
Theorem 103.17 Existence of a Johnson–Lindenstrauss dimension threshold
✓

Let \(\varepsilon \) be a real number with \(0 \lt \varepsilon \lt 1\), and let \(n \ge 2\) be an integer. Then there is a threshold \(k_0 \in \mathbb {N}\) such that for every target dimension \(k \ge k_0\), every ambient dimension \(d\), and every finite set \(V \subseteq \mathbb {R}^d\) of at most \(n\) points, there exists a linear map \(f : \mathbb {R}^d \to \mathbb {R}^k\) that is an \(\varepsilon \)-JL embedding of \(V\); that is, for all \(u, v \in V\),

\[ (1 - \varepsilon )\, \lVert u - v\rVert ^2 \; \le \; \lVert f(u) - f(v)\rVert ^2 \; \le \; (1 + \varepsilon )\, \lVert u - v\rVert ^2. \]
Theorem 103.18 Dimension threshold for Johnson–Lindenstrauss embeddings
✓

Fix a distortion parameter \(\varepsilon \) with \(0 \lt \varepsilon \lt 1\) and an integer \(n \ge 2\). Then there is a threshold \(k_0 \in \mathbb {N}\) such that for every target dimension \(k \ge k_0\), every source dimension \(d\), and every finite set \(V \subseteq \mathbb {R}^d\) of at most \(n\) points, there exists a linear map \(f : \mathbb {R}^d \to \mathbb {R}^k\) that is an \(\varepsilon \)-JL embedding of \(V\): that is, \((1-\varepsilon )\, \| u - v\| ^2 \le \| f(u) - f(v)\| ^2 \le (1+\varepsilon )\, \| u - v\| ^2\) for all \(u, v \in V\).