RKHS Intuition

At first RKHS looked scary, but after I realized that it’s just a space where every point in the space is a linear combination of (positive-definite) kernels, which allows us to replace the inner product calculation in this space with the kernel evaluation, I feel like things are much simpler now.

After all, the following two facts lead to the reproducing property (which can be verified by simple algebra):

  • the RKHS is spanned by kernels

  • the “regular” inner product definition

For example, take two functions $f(\cdot) = \sum_{i=1}^m \alpha_i k(\cdot, x_i)$ and $g(\cdot) = \sum_{j=1}^{m'} \beta_j k(\cdot, x'_j)$.
Define the inner product:
$<f, g>_{\mathcal{H}_k} = \sum_{i=1}^m \sum_{j=1}^{m'} \alpha_i \beta_j k(x_i, x'_j)$

Then,
$<f, k(\cdot, x'_k)> = \sum_{i=1}^m \sum_{j=1}^{m'} \alpha_i \beta_j k(x_i, x'_j) = \sum_{i=1}^m \alpha_i k(x_i,x'_k) = f(x'_k)$
where I set $\beta_j = 0$ when $j \neq k$ and $\beta_j = 1$ when $j=k$ to represent $k(\cdot, x_k)$ as a linear combinations of kernels in the Hilbert space.

Often times, the reproducing property is introduced to define RKHS (because it’s simpler?), but at least to me, viewing the reproducing property as a natural consequence from these two facts above was easier to grasp the concept of RKHS than thinking it as a condition required to construct RKHS, although this is more like the chicken and egg thing.