Existence of an Unbiased Consistent Entropy Estimator for the Special Bernoulli Measure

Abstract

Let \(\Omega = A^N\) - be a space of right-sided infinite sequences drawn from a finite alphabet \(A = \{0,1\}\), \(N = {1,2,\dots} \), \[\rho(\boldsymbol{x},\boldsymbol{y}) = \sum_{k=1}^{\infty}|x_{k} - y_{k}|2^{-k} \] - a metric on \(\Omega\), and \(\mu\) - a probability measure on \(\Omega\). Let \(\boldsymbol{\xi_0}, \boldsymbol{\xi_1}, \dots, \boldsymbol{\xi_n}\) - be independent identically distributed points on \(\Omega\). We study the estimator \(\eta_n^{(k)}(\gamma)\) - of the reciprocal of the entropy \(1/h\), that are defined as. \[\eta_n^{(k)}(\gamma) = k \left(r_{n}^{(k)}(\gamma) - r_{n}^{(k+1)}(\gamma)\right),\] where \[r_n^{(k)}(\gamma) =\frac{1}{n+1}\sum_{j=0}^{n} \gamma\left(\min_{i:i \neq j} {^{(k)}} \rho(\boldsymbol{\xi_{i}}, \boldsymbol{\xi_{j}})\right),\] \(\min ^{(k)}\{X_1,\dots,X_N\}= X_k\), если \(X_1\leq X_2\leq \dots\leq X_N\). Number \(k\) and a function \(\gamma(t)\) - are auxiliary parameters. The main result of this paper is

Theorem. Let \(m\) - be the Bernoulli measure with probabilities \(p_0,p_1>0\), \(p_0+p_1=1\), \(p_0=p_1^2\), then \(\forall eps>0\) some continuous function \(\gamma(t)\) such that \[
\left|E\eta_n^{(k)}(\gamma) - \frac1h\right| <eps,\quad DD\eta_n^{(k)}(\gamma)\to 0,n\to \infty. \]