Existence of an Unbiased Entropy Estimator for the Special Bernoulli Measure

Abstract

Let \(\Omega = A^{N}\)  be a space of right-sided infinite sequences drawn from a finite alphabet \(A = \{0,1\}\),  \(N = \{1,2,\dots \}\),
\[\label{rho}
  \rho(\boldsymbol{x},\boldsymbol{y}) =
\sum_{k=1}^{\infty}|x_{k} - y_{k}|2^{-k}
\]
a metric on \(\Omega = A^{N}\),
and \(\mu\) is a probability measure on \(\Omega\). Let \(\boldsymbol{\xi_0}, \boldsymbol{\xi_1}, \dots, \boldsymbol{\xi_n}\) be independent identically distributed points on \(\Omega\). We study the estimator \(\eta_n^{(k)}(\gamma)\) of the reciprocal of the entropy \(1/h\) that are defined as

\[ \label{etan}
\eta_n^{(k)}(\gamma) = k \left(r_{n}^{(k)}(\gamma) - r_{n}^{(k+1)}(\gamma)\right),\]
where
\[\label{def_r}
r_n^{(k)}(\gamma) =
\frac{1}{n+1}\sum_{j=0}^{n} \gamma\left(\min_{i:i \neq j} {^{(k)}}
\rho(\boldsymbol{\xi_{i}}, \boldsymbol{\xi_{j}})\right),
\]
 \(\min ^{(k)}\{X_1,\dots,X_N\}=  X_k\), if  \(X_1\leq X_2\leq \dots\leq X_N\). The number \(k\) and the function \(\gamma(t)\) are auxiliary parameters.

The main result of this paper is

Theorem. Let \(\mu\) be the Bernoulli measure  with probabilities \(p_0,p_1>0\), \(p_0+p_1=1\), \(p_0=p_1^2\). There exists a function \(\gamma(t)\) such that
\[E\eta_n^{(k)}(\gamma) =  \frac1h.\]