Optimal low-dimensional approximations of random vector functions ^{∗}

J. vom Scheidt  H.-J. Starkloff  R. Wunderlich

Abstract

The paper considers approximations of first- and second-order moments of random functions with values in a high-dimensional Euclidean space using projections onto suitable low-dimensional linear submanifolds. To quantify the goodness of the approximation a criterion based on the mean squared Euclidean distance is introduced. In case of wide-sense stationary random functions optimal low-dimensional linear submanifolds are given in terms of the mean vector and eigenvectors of the variance matrix.

1 Introduction

Let (Ω, A, P) be a probability space, where A denotes the σ-algebra of subsets of Ω on which is defined a probability measure P and D ⊂ R be an interval. Further, let z(t, ω) be a random function defined on D × Ω with values in a n-dimensional Euclidean space Z = C^{n}, n ≥ 1, where C denotes the space of complex numbers. It is assumed, that there exist moments of z(t, ω) up to second order. The mean of z(t, ω) is denoted by

m_{z}(t) = E {z(t)}

and the central second-order moment, i.e. the correlation function by

K_{zz}(t, s) = E {(z(t) − m_{z}(t))(z(s) − m_{z}(s))^{∗}}

where E {.} denotes expectation with respect to P and (.)^{∗} the adjoint matrix.

The paper considers approximations of the first- and second-order moments of z by the corresponding moments of a projection of z onto a linear submanifold of Z. This problem arises in case of a high-dimensional space Z, where the computation of the moments of z can be very difficult or impossible because of the high numerical effort (see [4, 5]).

∗This work was supported by Deutsche Forschungsgemeinschaft

If the dynamics and the randomness of z(t, ω) can be expressed - up to a certain degree of accuracy - by its projection onto a low-dimensional linear submanifold practicable approximations of the moments can be found. This approximation idea can be applied to the computation of moments of the response of discrete vibration systems with a random excitation (see [4]).

Section 2 provides some notations to investigate projections of z(t, ω) onto a linear submanifold of Z. To quantify the ”degree of accuracy” of the approximation of firstand second-order moments of z(t, ω) by the corresponding moments of the projection of z onto a submanifold we introduce some approximation criterions in Section 3. First, the pointwise squared Euclidean distance between z(t, ω) and its projection is used. By averaging with respect to t and ω a non-negative quantity – the approximation risk – is obtained. The approximation risk can be related with the mean and the trace of the variance matrix of z in the case of orthogonal projections. Further, bounds for the error of the approximation of the characteristics m_{z}(t) and K_{zz}(t, s) in terms of the approximation risk can be given.

In order to find optimal low-dimensional linear submanifolds Section 4 considers the minimization of the approximation risk with respect to the defining elements of the submanifold. As the main result we find, that in case of a wide-sense stationary function z the eigenvectors of the variance matrix K_{zz}(0) define optimal submanifolds and the mean m_{z} leads to the corresponding translation from the origin.

Applying these optimal projections to the approximation of moments of z the mean and the variance matrix of z have to be computed. This partial information on the characteristics can be computed much easier and more effective than the complete characteristics.

Similar problems of approximation of (not necessarily vector valued) random functions by projections onto finite dimensional (function) spaces are studied e.g. in [1] and [3].

2 Projections

In this section we consider an affin linear transformation of the coordinate system of Z and projections onto m-dimensional linear submanifolds of Z, 1 ≤ m ≤ n. Afterwards these transformations and projections will be applied to the function z(t, ω) pointwise for all t and ω to derive an approximation of z. For abbrevation the dependence of z on t and ω is omitted and we write z instead z(t, ω).

Let (e_{k})_{1≤k≤n} be the canonical basis of Z , i.e. z = (z_{1}, …, z_{n})^{τ} ∈ Z can be represented by

z = ∑_{k=1}^{n} z_{k}e_{k}

with coordinates z_{k} ∈ C, k = 1, …, n, with respect to (e_{k})_{1≤k≤n}. Denote the scalar product and the norm of Z by

(x, y) = ∑_{k=1}^{n} x_{k}\overline{y_{k}}  and  | ; |x| ; | = (x, x)^{\frac{1}{2}} = (∑_{k=1}^{n} |x_{k}|^{2})^{\frac{1}{2}} ,

for x, y ∈ Z.

Let (v_{k})_{1≤k≤n} be a set of linear independent elements of Z with | ; |v_{k}| ; | = 1 and v_{0} ∈ Z. Denote by x_{1}, …, x_{n} ∈ C the new coordinates of z ∈ Z with respect to the basis (v_{k})_{1≤k≤n} and the translation element v_{0} (see Figure 1), i.e.

z = ∑_{k=1}^{n} x_{k}v_{k} + v_{0}.  (1)

Defining the vector x = (x_{1}, …, x_{n})^{τ} and the non-singular matrix V = (v_{1}, …, v_{n}) (v_{1}, …, v_{n} are linear independent) the transformation (1) and its inverse transformation can be written as

z = Vx + v_{0}  and  x = V^{−1}(z − v_{0}).

.^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}  ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}e1 ....... z_{2} e_{2}  z1z  x_{1} x_{2}  v_{1} v_{2}  v_{0}

Figure 1: Transformation of coordinate system for n = 2

Now we consider projections of elements z ∈ Z onto m-dimensional linear submanifolds S_{m} defined by

S_{m} := v_{0} + span(v_{1}, …, v_{m}), 1 ≤ m ≤ n,
and S_{0} := v_{0}.

Let (P_{m})_{0≤m≤n} be the sequence of projectors P_{m} : Z → S_{m} defined by

P_{m}z := v_{0} + ∑_{k=1}^{m} x_{k}v_{k}, 1 ≤ m ≤ n,

(2) and P_{0}z := v_{0}.

Defining real and symmetric matrices P_{m} by

P_{m} := ( \begin{matrix}I_{m} & O_{m,n−m} \\ O_{n−m,m} & O_{n−m,n−m} \end{matrix} ) 1 ≤ m ≤ n,
and P_{0} := O_{n,n},

where I_{m} denotes the m × m identity matrix and O_{..} denote null matrices of the corresponding dimensions. Then relation (2) can be written as

P_{m}z = v_{0} + VP_{m}x
= v_{0} + VP_{m}V^{−1}(z − v_{0}).  (3)

For the sake of shorter notation complementary projection operators Q_{m} are introduced by

Q_{m}z := (I − P_{m})z = z − P_{m}z  (4)
= VQ_{m}x = VQ_{m}V^{−1}(z − v_{0}),

where the real and symmetric matrix Q_{m} is defined by Q_{m} := I_{n} − P_{m}, i.e.

Q_{m} = ( \begin{matrix}O_{m,m} & O_{m,n−m} \\ O_{n−m,m} & I_{n−m} \end{matrix} ) 1 ≤ m ≤ n,

(5) and Q_{0} = I_{n},

and I denotes the identity operator I : Z → Z with Iz = z.

3 Approximation criterions

In this section the projection operator P_{m} is applied to z(t, ω) pointwise for all t and ω and P_{m}z is used as an approximation of z. In order to quantify and to assess the accuracy of this approximation several criterions are studied. We start with the definition of a non-negative loss function, which measures pointwise the distance between z(t, ω) and its approximation P_{m}z(t, ω). Averaging with respect to t and ω leads to a non-negative quantity – the approximation risk – which characterizes the ”goodness” of the projector P_{m}. If the squared Euclidean distance is used as loss function it will be proved that orthogonal projections are optimal with respect to the approximation risk.

Finally, we deal with the approximation of first- and second-order moments of z by the corresponding moments of P_{m}z and estimate the accuracy of these approximations in terms of the approximation risk.

Definition 1 A (B(Z) ⊗ B(Z), B(R_{+}))-measurable non-negative function l(., .) : Z × Z → R_{+} is called \underline{loss function} and

β = β(P_{m}, t, ω) := l(z(t, ω), P_{m}z(t, ω))

is called \underline{approximation error} of the approximation P_{m}z of z (with respect to l(., .) and P_{m}). Here B(.) denotes the corresponding σ-algebra of Borel-sets.

The approximation error β = β(P_{m}, t, ω) with β(P_{m}, ., .) : D × Ω → R_{+}, is a nonnegative scalar random function. By averaging with respect to t and ω a non-random and non-time-dependent quantity is obtained which will be used as objective in the the optimization afterwards.

Definition 2 Let ψ(.) : R → R_{+} be an integrable non-negative weighting function with ∫_{D} ψ(t)dt = 1.

Then, the quantity

γ = γ(P_{m}) := ∫_{D} E {β(P_{m}, t)} ψ(t)dt

is called \underline{approximation risk} of P_{m} with respect to the loss function l and the weighting function ψ.

Denote by M the set of all sequences (P_{m})_{0≤m≤n} of projection operators defined in (2). Each sequence can be identified by n linear independent basis elements v_{1}, …, v_{n} ∈ Z, | ; |v_{k}| ; | = 1, k = 1, …, n, and a translation element v_{0} ∈ Z.

Definition 3 A sequence (P_{m}^{∗})_{0≤m≤n} ∈ M is said to be \underline{γ-optimal} with respect to the set M if

γ(P_{m}^{∗}) ≤ γ(P_{m}), m = 0, …, n,

holds for all sequences (P_{m})_{0≤m≤n} ∈ M.

Now some possible choices of the loss function l(., .) are listed.

i) l_{1}(x, y) := | ; |x − y| ; |^{2} with β(P_{m}, t, ω) = | ; |z(t, ω) − P_{m}z(t, ω)| ; |^{2} = | ; |Q_{m}z(t, ω)| ; |^{2}

This function is an appropriate choice if the approximation error can be controlled by the squared Euclidean distance between z(t, ω) and P_{m}z(t, ω).

ii) l_{2}(x, y) := | ; |W(x − y)| ; |^{2} with β(P_{m}, t, ω) = | ; |WQ_{m}z(t, ω)| ; |^{2}

where W denotes a n × n diagonal matrix with diagonal elements w_{k} ≥ 0, k = 1, …, n. The goodness of approximation of several components of z can be weighted by this function. Less important components z_{k} can be neglected by choosing of weights w_{k} = 0.

iii) l_{3}(x, y) := | ; |C(x − y)| ; |2_{Z′} with β(P_{m}, t, ω) = | ; |CQ_{m}z(t, ω)| ; |2_{Z′}

where C denotes a rectangular n′ × n-matrix with 1 ≤ n′ ≤ n, and Z′ is a n′- dimensional Euclidean space with the norm | ; |.| ; |2_{Z′}. This function can be applied if an output equation

y = Cz

is given and the goodness of the approximation ỹ = CP_{m}z of the ”output” has to be assessed .

Further loss functions and the corresponding approximation errors and risks can be found in [3] p. 93 ff.

In mechanical applications the random dynamics of multibody systems is investigated using a vector function z(t, ω) which contains positions and velocities of the bodies of the system. In this case not only the approximation goodness with respect to the moments of z(t, ω) but also of its derivative ż(t, ω) (which contains the accelerations) has to be assessed. This can be done with a modified approximation error and risk

β̆(P_{m}, t, ω) := l(ż(t, ω), P_{m}ż(t, ω))
γ̆(P_{m}) := ∫_{D} E {β̆(P_{m}, t, ω)} ψ(t)dt.

For this paper the loss function l = l_{1} with the approximation error

β(P_{m}, t, ω) = | ; |Q_{m}z(t, ω)| ; |^{2}

is considered, only. Then the corresponding approximation risk can be written as

γ(P_{m}) = ∫_{D} E {| ; |Q_{m}z(t)| ; |^{2}} ψ(t)dt.  (6)

Now we deal with the case of orthogonal projectors. Denote by N the set of all sequences (P_{m})_{0≤m≤n} of projection operators with respect to a translation element v_{0} ∈ Z and orthonormal elements v_{1}, …, v_{n} ∈ Z, i.e. (v_{k}, v_{l}) = δ_{kl}, for 1 ≤ k, l ≤ n. Then N is a subset of M. Subsequently, we consider γ-optimal sequences (P_{m}^{∗})_{0≤m≤n} with repect to N and compare these sequences with the corresponding γ-optimal sequences for the larger set M.

Lemma 1 If a sequence (P_{m}^{∗})_{0≤m≤n} ∈ N is γ-optimal with respect to the set N ⊂ M then it is also γ-optimal with respect to the set M.

Proof: If (P_{m}^{∗})_{0≤m≤n} ∈ N is γ-optimal with respect to the set N then

γ(P_{m}^{∗}) ≤ γ(P_{m}), m = 0, …, n

holds ∀ (P_{m})_{0≤m≤n} ∈ N ⊂ M . We have to prove that the relations

γ(P_{m}^{∗}) ≤ γ(P_{m}), m = 0, …, n  (7)

are fulfilled ∀ (P_{m})_{0≤m≤n} ∈ M \ N.

Now it is assumed that relations (7) are not fulfilled, i.e. there exists a sequence (R_{m})_{0≤m≤n} ∈ M \ N and an index r, 1 ≤ r ≤ n, with

γ(P_{r}^{∗}) > γ(R_{r}).  (8)

The sequence (R_{m})_{0≤m≤n} is assumed to be defined by the translation element v_{0} and the linear independent elements v_{1}, …, v_{n}. A set of pairwise orthonormal elements w_{1}, …, w_{n} with respect to v_{1}, …, v_{n} can be found (e.g. by means of Schmidtorthogonalization) which fulfill the conditions

span(v_{1}, …, v_{m}) = span(w_{1}, …, w_{m}),  m = 1, …, n,
(w_{k}, w_{l}) = δ_{kl}, 1 ≤ k, l ≤ n,  (9)
(w_{k}, v_{l}) = 0, 1 ≤ l < k ≤ n.

[figure: .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; e_{1}; z; v_{1} = w_{1}; v_{2}; e_{2}; w_{2}; P_{r}^{w}z; R_{r}z]

Figure 2: Orthogonal and non-orthogonal projections of z for n = 2, r = 1 Denote by (P_{m}^{w})_{0≤m≤n} the sequence of projectors defined by v_{0}, w_{1}, …, w_{n}, where (P_{m}^{w})_{0≤m≤n} ∈ N. Now we consider the case m = r and the identity

z − R_{r}z = (z − P_{r}^{w}z) + (P_{r}^{w}z − R_{r}z), ∀ t ∈ [0, T] and ∀ ω ∈ Ω.

Let the coordinates of z with respect to the orthonormal basis (w_{k})_{0≤k≤n} and the translation element v_{0} be denoted by y_{1}, …, y_{n}. By means of notations in (1) and (2) it follows

z = v_{0} + ∑_{k=1}^{n} x_{k}v_{k} = v_{0} + ∑_{k=1}^{n} y_{k}w_{k},
R_{r}z = v_{0} + ∑_{k=1}^{r} x_{k}v_{k},
P_{r}^{w}z = v_{0} + ∑_{k=1}^{r} y_{k}w_{k}.

Applying relations (9) we derive

(z − P_{r}^{w}z, P_{r}^{w}z − R_{r}z) = ( ∑_{k=r+1}^{n} y_{k}w_{k}, ∑_{l=1}^{r}(y_{l}w_{l} − x_{l}v_{l}))
= ∑_{k=r+1}^{n} ∑_{l=1}^{r} (y_{k}y_{l} (w_{k}, w_{l}) − y_{k}x_{l} (w_{k}, v_{l}))
= 0.

Hence, z − P_{r}^{w}z and P_{r}^{w}z − R_{r}z are orthogonal and it follows

| ; |z − R_{r}z| ; |^{2} = ||z − P_{r}^{w}z||^{2} + ||P_{r}^{w}z − R_{r}z||^{2} ≥ ||z − P_{r}^{w}z||^{2} .

Since

β(R_{r}, t, ω) = | ; |z(t, ω) − R_{r}z(t, ω)| ; |^{2}
β(P_{r}^{w}, t, ω) = ||z(t, ω) − P_{r}^{w}z(t, ω)||^{2}

the relation β(R_{r}, t, ω) ≥ β(P_{r}^{w}, t, ω) pointwise ∀ t ∈ [0, T] and ∀ ω ∈ Ω is obvious and it holds

γ(R_{r}) = ∫_{D} E {β(R_{r}, t)} ψ(t)dt ≥ ∫_{D} E {β(P_{r}^{w}, t)} ψ(t)dt = γ(P_{r}^{w}).

Since (P_{m}^{w})_{0≤m≤n} ∈ N we have γ(R_{r}) ≥ γ(P_{r}^{w}) ≥ γ(P_{r}^{∗}) in contradiction to the assumption (8) and the proof is complete.

By means of Lemma 1 the search of γ-optimal projections can be restricted to the case of orthonormal elements v_{1}, …, v_{n}. In this case the transformation matrix V = (v_{1}, …, v_{n}) is an unitary matrix, i.e. V^{−1} = V^{∗}.

Another result which needs not to be true in the non-orthogonal case is given in the following lemma. This result leads to a more convenient measure for the goodness of approximation

Lemma 2 Let (P_{m})_{0≤m≤n} ∈ N be a sequence of orthogonal projectors defined by a translation element v_{0} ∈ Z and orthonormal elements v_{1}, …, v_{n} ∈ Z. Then for the approximation error β(P_{m}, t, ω) with respect to the loss function l_{1}(x, y) = | ; |x − y| ; |^{2} , x, y ∈ Z, it holds for 0 < m_{1} < m_{2} < n and ∀ t ∈ [0, T], ∀ ω ∈ Ω

0 = β(P_{n}, t, ω) ≤ β(P_{m_{2}}, t, ω) ≤ β(P_{m_{1}}, t, ω) ≤ β(P_{0}, t, ω),

i.e. β(P_{m}, t, ω) is monotonically decreasing with respect to m.

Proof: In case of 0 ≤ m < n we obtain

β(P_{m}, t, ω) = | ; |Q_{m}z(t, ω)| ; |^{2} = | ; | ∑_{k=m+1}^{n} x_{k}(t, ω)v_{k}| ; |^{2}

= ∑_{k,l=m+1}^{n} x_{k}(t, ω)\overline{x_{l}}(t, ω) (v_{k}, v_{l})
= ∑_{k=m+1}^{n} |x_{k}(t, ω)|^{2},  (10)

by means of Eq. (4), where x_{k}, k = 1, …, n, denote coordinates of z with respect to the orthonormal elements v_{1}, …, v_{n} and the translation element v_{0}, i.e. z = v_{0} + ∑_{k=1}^{n} x_{k}v_{k}. Then using m_{1} < m_{2} it follows

0 = | ; |Q_{n}z(t, ω)| ; |^{2} = β(P_{n}, t, ω)
≤ ∑_{k=m_{2}+1}^{n} |x_{k}(t, ω)|^{2} = | ; |Q_{m_{2}}z(t, ω)| ; |^{2} = β(P_{m_{2}}, t, ω)
≤ ∑_{k=m_{1}+1}^{n} |x_{k}(t, ω)|^{2} = | ; |Q_{m_{1}}z(t, ω)| ; |^{2} = β(P_{m_{1}}, t, ω)
≤ ∑_{k=1}^{n} |x_{k}(t, ω)|^{2} = | ; |z(t, ω) − v_{0}| ; |^{2} = β(P_{0}, t, ω).

Remark 1 The assumption of orthogonal elements v_{1}, …, v_{n} in lemma 2 is essential. In case of non-orthogonal elements the inequalities

β(P_{m_{2}}, t, ω) ≤ β(P_{m_{1}}, t, ω) ≤ β(P_{0}, t, ω),

for 0 < m_{1} < m_{2} < n need not to be correct.

Definition 4 Let γ be the approximation risk of P_{m} with respect to the loss function l_{1}(x, y) = | ; |x − y| ; |^{2} and a weighting function ψ.

i) For a sequence (P_{m})_{0≤m≤n} ∈ N of orthogonal projection operators with γ(P_{0}) > 0

ρ = ρ(P_{m}) = 1 − \frac{γ(P_{m})}{γ(P_{0})}, m = 0, …, n,

is called \underline{approximation efficiency} of P_{m}.

ii) A sequence (P_{m}^{∗})_{0≤m≤n} ∈ N is said to be \underline{ρ-optimal} with respect to N if

ρ(P_{m}^{∗}) ≥ ρ(P_{m}), m = 0, …, n

holds for all sequences (P_{m})_{0≤m≤n} ∈ N.

Remark 2 The condition γ(P_{0}) > 0 is not restrictive, since

0 = γ(P_{0}) = ∫_{D} E {β(P_{0}, t)} ψ(t)dt = ∫_{D} E {| ; |z(t) − v_{0}| ; |^{2}} ψ(t)dt

only holds in the degenerate case z(t, ω) ≡ v_{0} ∀ t ∈ {s ∈ D : ψ(s) ≠ 0} a.s. .

Corollary 1

i) Applying Lemma 2 a sequence (P_{m})_{0≤m≤n} ∈ N fulfills the inequality

0 = ρ(P_{0}) ≤ ρ(P_{m_{1}}) ≤ ρ(P_{m_{2}}) ≤ ρ(P_{n}) = 1

for 0 < m_{1} < m_{2} < n.

ii) Assuming a fixed translation element v_{0} a sequence (P_{m}^{∗})_{0≤m≤n} ∈ N is ρoptimal with respect to N if and only if (P_{m}^{∗})_{0≤m≤n} is γ-optimal with respect to N.

Now several representations of the approximation risk and efficiency are derived using sequences (P_{m})_{0≤m≤n} ∈ N defined by orthonormal elements v_{1}, …, v_{n}. Using representation (10) of the approximation error we obtain for (P_{m})_{0≤m≤n} ∈ N

β(P_{m}, t, ω) = | ; |Q_{m}z(t, ω)| ; |^{2} = \{ \begin{matrix}∑_{k=m+1}^{n} |x_{k}(t, ω)|^{2} & 0 ≤ m < n \\ 0 & m = n \end{matrix}
= tr [Q_{m}x(t, ω)x^{∗}(t, ω)Q_{m}] ,

where tr [.] denotes the trace of a matrix and Q_{m} is the complementary projection matrix defined in (5). Then the approximation risk can be written as

γ(P_{m}) = ∫_{D} E {β(P_{m}, t)} ψ(t)dt = ∫_{D} E {tr [Q_{m}x(t)x^{∗}(t)Q_{m}]} ψ(t)dt
= ∫_{D} tr [Q_{m}E {x(t)x^{∗}(t)} Q_{m}] ψ(t)dt
= ∫_{D} tr [Q_{m} (K_{xx}(t, t) + m_{x}(t)m_{x}^{∗}(t)) Q_{m}] ψ(t)dt
= γ_{1}(P_{m}) + γ_{2}(P_{m}),

where γ_{1}(P_{m}) and γ_{2}(P_{m}) are defined by

γ_{1}(P_{m}) := ∫_{D} tr [Q_{m}m_{x}(t)m_{x}^{∗}(t)Q_{m}] ψ(t)dt = ∫_{D} | ; |Q_{m}m_{x}(t)| ; |^{2} ψ(t)dt

(11)

γ_{2}(P_{m}) := ∫_{D} tr [Q_{m}K_{xx}(t, t)Q_{m}] ψ(t)dt.

Representations of γ_{1}(P_{m}) and γ_{2}(P_{m}) in terms of the moments of z(t, ω) can be derived using the relations

z(t, ω) = v_{0} + Vx(t, ω) and x(t, ω) = V^{∗}(z(t, ω) − v_{0})

with an unitary matrix V. They imply the following identities for the mean and the correlation function

m_{x}(t) = V^{∗}(m_{z}(t) − v_{0})

(12)

K_{xx}(t, s) = V^{∗}K_{zz}(t, s)V

and it follows

γ_{1}(P_{m}) = ∫_{D} ||Q_{m}V^{∗}(m_{z}(t) − v_{0})||^{2} ψ(t)dt

(13)

γ_{2}(P_{m}) = ∫_{D} tr [Q_{m}V^{∗}K_{zz}(t, t)VQ_{m}] ψ(t)dt.

For square matrices A and C where C^{−1} exists it holds the identity

tr [CAC^{−1}] = tr [A] .  (14)

Substituting C = V, C^{−1} = V^{∗} and applying Eq. (4)

Q_{m}z(t, ω) = VQ_{m}V^{∗}(z(t, ω) − v_{0}) = VQ_{m}x(t, ω)

we derive by means of Eqs. (11) and (13) a representation in terms of the complementary projection Q_{m}z(t, ω)

γ_{1}(P_{m}) = ∫_{D} | ; |m_{Q_{m}z}(t)| ; |^{2} ψ(t)dt

(15)

γ_{2}(P_{m}) = ∫_{D} tr [K_{Q_{m}zQ_{m}z}(t, t)] ψ(t)dt.

Eqs. (15) show that γ_{1}(P_{m}) is based on the mean of the complementary projection Q_{m}z and γ_{2}(P_{m}) is based on the correlation function of Q_{m}z, only.

Similar representations can be found for the approximation efficiency ρ(P_{m}) which is introduced in Definition 4.

In the following considerations the goodness of the approximation of first- and secondorder moments of z(t, ω) by the corresponding moments of the projection P_{m}z(t, ω) is studied. Again, let us start with the definition of a non-negative loss function which gives a pointwise measure for the approximation error. In order to derive a non-negative real number which characterizes the goodness of the projection with respect to the approximation of the moments this pointwise measure is averaged with respect to t.

Definition 5 The non-negative functions

l^{m}(., .) : Z × Z → R_{+}
and l^{K}(., .) : Z^{2} × Z^{2} → R_{+}

are called \underline{loss functions} with respect to the statistical characteristics m(.) and K(., .), respectively.

To a given approximation z̃ of z the quantities

δ^{m}(z, z̃) := ∫_{D} l^{m}(m_{z}(t), mz̃(t))ψ(t)dt
δ^{K}(z, z̃) := ∫_{D} ∫_{D} l^{K}(K_{zz}(t, s), K_{z̃z̃}(t, s))ψ(t)ψ(s)dtds

are called \underline{approximation error} of the approximation z̃ of z with respect to m(.) and K(., .), the loss function l^{m} and l^{K} and the weighting function ψ, respectively.

For the approximation of z by its projection, i.e. z̃ = P_{m}z, the next lemma gives representations and estimates of the introduced measures of the approximation error in terms of the approximation risk γ(P_{m}). These estimates motivate the choice of γ as objective for the optimization in Section 4.

Lemma 3 Assuming a sequence (P_{m})_{0≤m≤n} ∈ N of orthogonal projection operators it holds

i) for the approximation error with respect to m(.) and the loss function l^{m}(x, y) = | ; |x − y| ; |^{2} , x, y ∈ Z,

δ^{m}(z, P_{m}z) = γ_{1}(P_{m}), m = 0, …, n,

ii) for the approximation error with respect to K(., .) and the loss function l^{K}(X, Y) = |tr [X] − tr [Y] |, X, Y ∈ Z^{2},

δ^{K}(z, P_{m}z) ≤ γ_{2}(P_{m}), m = 0, …, n,

iii) for the approximation error with respect to K(., .) and the loss function

l^{K}(X, Y) = | ; |X − Y| ; |2_{Z^{2}} = ∑_{i,j=1}^{n} |X_{ij} − Y_{ij}|^{2}, X, Y ∈ Z^{2},

δ^{K}(z, P_{m}z) ≤ 3γ_{2}(P_{m})(2γ_{2}(P_{0}) − γ_{2}(P_{m})),  m = 0, …, n.

The quantities γ_{1}(P_{m}) and γ_{2}(P_{m}) are given by Eqs. (11).

Proof:

i) By means of the chosen loss function and Eqs. (4) and (15) it follows from the definition of δ^{m}

δ^{m}(z, P_{m}z) = ∫_{D} | ; |m_{z}(t) − m_{P_{m}z}(t)| ; |^{2} ψ(t)dt = ∫_{D} | ; |m_{Q_{m}z}(t)| ; |^{2} ψ(t)dt = γ_{1}(P_{m}).

ii) In case of the chosen loss function the approximation error δ^{K} can be written as

δ^{K}(z, P_{m}z) = ∫_{D} ∫_{D} |tr [K_{zz}(t, s)] − tr [K_{P_{m}zP_{m}z}(t, s)]| ψ(t)ψ(s)dtds.

Using z(t, ω) = v_{0} + Vx(t, ω) and P_{m}z(t, ω) = v_{0} + VP_{m}x(t, ω) (see Eq. (3)) we obtain

K_{zz}(t, s) = VK_{xx}(t, s)V^{∗} and K_{P_{m}zP_{m}z}(t, s) = VP_{m}K_{xx}(t, s)P_{m}V^{∗}

and by means of Eq. (14) and Cauchy-Schwarz inequality |tr [K_{zz}(t, s)] − tr [K_{P_{m}zP_{m}z}(t, s)]| = |tr [K_{xx}(t, s)] − tr [P_{m}K_{xx}(t, s)P_{m}]|

= |tr [Q_{m}K_{xx}(t, s)Q_{m}]| = | ∑_{k=m+1}^{n} R_{x_{k}x_{k}}(t, s)|  (16)
≤ ∑_{k=m+1}^{n} |R_{x_{k}x_{k}}(t, s)| ≤ ∑_{k=m+1}^{n} \sqrt{R_{x_{k}x_{k}}(t, t)R_{x_{k}x_{k}}(s, s)}.

Then it follows

δ^{K}(z, P_{m}z) ≤ ∫_{D} ∫_{D} ∑_{k=m+1}^{n} \sqrt{R_{x_{k}x_{k}}(t, t)R_{x_{k}x_{k}}(s, s)}ψ(t)ψ(s)dtds
= ∑_{k=m+1}^{n} (∫_{D} \sqrt{R_{x_{k}x_{k}}(t, t)}ψ(t)dt)^{2}
≤ ∑_{k=m+1}^{n} ∫_{D} R_{x_{k}x_{k}}(t, t)ψ(t)dt · ∫_{D} ψ(t)dt
= ∫_{D} tr [Q_{m}K_{xx}(t, t)Q_{m}] ψ(t)dt = γ_{2}(P_{m})

(see (11)). iii) In the third case the definition of δ^{K} leads to

δ^{K}(z, P_{m}z) = ∫_{D} ∫_{D} ∑_{i,j=1}^{n} |R_{z_{i}z_{j}}(t, s) − R_{[P_{m}z]_{i}[P_{m}z]_{j}}(t, s)|^{2} ψ(t)ψ(s)dtds,

where [.]_{i} denotes the i-th component of a vector. Because of z = P_{m}z + Q_{m}z it follows K_{zz}(t, s) = K_{P_{m}zP_{m}z}(t, s) + K_{P_{m}zQ_{m}z}(t, s) + K_{Q_{m}zP_{m}z}(t, s) + K_{Q_{m}zQ_{m}z}(t, s). and using again Cauchy-Schwarz inequality it holds

|R_{z_{i}z_{j}}(t, s) − R_{[P_{m}z]_{i}[P_{m}z]_{j}}(t, s)|^{2}
= |R_{[P_{m}z]_{i}[Q_{m}z]_{j}}(t, s) + R_{[Q_{m}z]_{i}[P_{m}z]_{j}}(t, s) + R_{[Q_{m}z]_{i}[Q_{m}z]_{j}}(t, s)|^{2}
≤ 3 (|R_{[P_{m}z]_{i}[Q_{m}z]_{j}}(t, s)|^{2} + |R_{[Q_{m}z]_{i}[P_{m}z]_{j}}(t, s)|^{2} + |R_{[Q_{m}z]_{i}[Q_{m}z]_{j}}(t, s)|^{2})
≤ 3 (R_{[P_{m}z]_{i}[P_{m}z]_{i}}(t, t) R_{[Q_{m}z]_{j}[Q_{m}z]_{j}}(s, s)
+R_{[Q_{m}z]_{i}[Q_{m}z]_{i}}(t, t) R_{[P_{m}z]_{j}[P_{m}z]_{j}}(s, s)
+ R_{[Q_{m}z]_{i}[Q_{m}z]_{i}}(t, t) R_{[Q_{m}z]_{j}[Q_{m}z]_{j}}(s, s)) .

The substitution of this inequality into the representation of δ^{K}(z, P_{m}z) leads to

δ^{K}(z, P_{m}z) ≤ 3 ∫_{D} ∫_{D} [(∑_{i=1}^{n} R_{[P_{m}z]_{i}[P_{m}z]_{i}}(t, t)) (∑_{j=1}^{n} R_{[Q_{m}z]_{j}[Q_{m}z]_{j}}(s, s))

+ (∑_{i=1}^{n} R_{[Q_{m}z]_{i}[Q_{m}z]_{i}}(t, t)) (∑_{j=1}^{n} R_{[P_{m}z]_{j}[P_{m}z]_{j}}(s, s))

+ (∑_{i=1}^{n} R_{[Q_{m}z]_{i}[Q_{m}z]_{i}}(t, t)) (∑_{j=1}^{n} R_{[Q_{m}z]_{j}[Q_{m}z]_{j}}(s, s))] ψ(t)ψ(s)dtds

= 3 [∫_{D} tr [K_{P_{m}zP_{m}z}(t, t)] ψ(t)dt · ∫_{D} tr [K_{Q_{m}zQ_{m}z}(s, s)] ψ(s)ds

+ ∫_{D} tr [K_{Q_{m}zQ_{m}z}(t, t)] ψ(t)dt · ∫_{D} tr [K_{P_{m}zP_{m}z}(s, s)] ψ(s)ds

+ ∫_{D} tr [K_{Q_{m}zQ_{m}z}(t, t)] ψ(t)dt · ∫_{D} tr [K_{Q_{m}zQ_{m}z}(s, s)] ψ(s)ds] .

From Eq. (15), ∫_{D} tr [K_{Q_{m}zQ_{m}z}(t, t)] ψ(t)dt = γ_{2}(P_{m}), and (16) we can deduce

∫_{D} tr [K_{P_{m}zP_{m}z}(t, t)] ψ(t)dt = ∫_{D} (tr [K_{zz}(t, t)] − tr [K_{Q_{m}zQ_{m}z}(t, t)])ψ(t)dt
= γ_{2}(P_{0}) − γ_{2}(P_{m}),

hence

δ^{K}(z, P_{m}z) ≤ 3γ_{2}(P_{m}) (2(γ_{2}(P_{0}) − γ_{2}(P_{m})) + γ_{2}(P_{m}))
= 3γ_{2}(P_{m}) (2γ_{2}(P_{0}) − γ_{2}(P_{m})) .

Finally, the case of wide-sense stationary vector functions is considered, i.e. it holds

m_{z}(t) ≡ m_{z} and K_{zz}(t, s) =: R_{zz}(s − t).

Evaluating the integrals in Eqs. (13) it follows

γ(P_{m}) = γ_{1}(P_{m}) + γ_{2}(P_{m}) = ||Q_{m}V^{∗}(m_{z} − v_{0})||^{2} + tr [Q_{m}V^{∗}R_{zz}(0)VQ_{m}] .

Then, the approximation error with respect to m(.) can be written as

δ^{m}(z, P_{m}z) = l^{m}(m_{z}, m_{P_{m}z}).

Furthermore, for the approximation error with respect to K(., .) hold the following relations

δ^{K}(z, z̃) = ∫_{D} ∫_{D} l^{K}(K_{zz}(t, s), K_{z̃z̃}(t, s))ψ(t)ψ(s)dtds

= ∫_{D} ∫_{{τ∈R:t+τ∈D}} l^{K}(K_{zz}(t, t + τ), K_{z̃z̃}(t, t + τ))ψ(t)ψ(t + τ)dtdτ

= ∫_{{τ∈R:τ=s−t,s,t∈D}} ∫_{{t∈D:t+τ∈D}} l^{K}(K_{zz}(t, t + τ), K_{z̃z̃}(t, t + τ))ψ(t)ψ(t + τ)dtdτ

= ∫_{T} l^{K}(R_{zz}(τ), R_{z̃z̃}(τ))ψ̆(τ)dτ

where the notations

T := {τ ∈ R : τ = s − t, s, t ∈ D} and ψ̆(τ) := ∫_{{t∈D:t+τ∈D}} ψ(t)ψ(t + τ)dt, τ ∈ T

are used. Obviously, it holds ∫_{T} ψ̆(τ)dτ = 1. These relations motivate the introduction

of a slightly modified approximation error with repect to R(.) using an arbitrary weighting function θ(.) : T → R_{+} with ∫_{T} θ(τ)dτ = 1.

Definition 6 To a given wide-sense stationary approximation z̃ of wide-sense stationary vector function z the quantity

δ^{R}(z, z̃) := ∫_{T} l^{K}(R_{zz}(τ), R_{z̃z̃}(τ)) θ(τ)dτ

is called \underline{approximation error} of the approximation z̃ of z with respect to R(τ), the loss function l^{K} and the weighting function θ.

The next Lemma provides estimates for δ^{R} in terms of the approximation risk which are the corresponding counterparts of the estimates for δ^{K} given in Lemma 3 for the general case.

Lemma 4 Assuming a wide-sense stationary vector function z and a sequence of orthogonal projection operators (P_{m})_{0≤m≤n} ∈ N then it holds for the approximation error with respect to R_{zz}(τ) and

i) a loss function l^{K}(X, Y) = |tr [X] − tr [Y] |, X, Y ∈ Z^{2}

δ^{R}(z, P_{m}z) ≤ γ_{2}(P_{m}) = tr [Q_{m}V^{∗}R_{zz}(0)VQ_{m}] , m = 0, …, n, ii) a loss function

l^{K}(X, Y) = | ; |X − Y| ; |2_{Z^{2}} = ∑_{i,j=1}^{n} |X_{ij} − Y_{ij}|^{2}, X, Y ∈ Z^{2},

δ^{R}(z, P_{m}z) ≤ 3γ_{2}(P_{m})(2γ_{2}(P_{0}) − γ_{2}(P_{m})), m = 0, …, n.

Proof: The proof can be obtained using the techniques of the proofs of Lemma 3 (ii) and (iii).

Remark 3 While the approximation error δ^{R}(z, P_{m}z) depends on the weighting function θ the upper bounds for δ^{R}(z, P_{m}z) derived in the lemma above do not depend on the selection of θ.

In a similar way the goodness of the approximation of the spectral density

S_{zz}(α) := \frac{1}{2π} ∫_{−∞}^{∞} e^{−iατ}R_{zz}(τ)dτ, α ∈ R,

can be treated.

Definition 7 The non-negative function

l^{S}(., .) : Z^{2} × Z^{2} → R_{+}

is called \underline{loss function} with respect to the spectral density S(α).

For a wide-sense stationary approximation z̃ of the wide-sense stationary vector function z the quantity

δ^{S}(z, z̃) := ∫_{−∞}^{∞} l^{S}(S_{zz}(α), S_{z̃z̃}(α)) dα

is called \underline{approximation error} of the approximation z̃ of z with respect to S(.) and the loss function l^{S}.

Lemma 5 Assuming a wide-sense stationary vector function z, a sequence (P_{m})_{0≤m≤n} ∈ N of orthogonal projection operators and a loss function

l^{S}(X, Y) = |tr [X] − tr [Y] |, X, Y ∈ Z^{2}

then it holds for the approximation error with respect to S(.)

δ^{S}(z, z̃) = γ_{2}(P_{m}) = tr [Q_{m}V^{∗}R_{zz}(0)VQ_{m}] , m = 0, …, n.

Proof: Using the identities

R_{zz}(τ) = VR_{xx}(τ)V^{∗} and R_{P_{m}zP_{m}z}(τ) = VP_{m}R_{xx}(τ)P_{m}V^{∗}

it follows

S_{zz}(α) = \frac{1}{2π} ∫_{−∞}^{∞} e^{−iατ}VR_{xx}(τ)V^{∗}dτ = VS_{xx}(α)V^{∗}, α ∈ R,

and analogously S_{P_{m}zP_{m}z}(α) = VP_{m}S_{xx}(α)P_{m}V^{∗}. Applying Eq. (14) the approximation error δ^{S} can be written as

δ^{S}(z, P_{m}z) = ∫_{−∞}^{∞} |tr [S_{zz}(α)] − tr [S_{P_{m}zP_{m}z}(α)] | dα

= ∫_{−∞}^{∞} |tr [S_{xx}(α)] − tr [P_{m}S_{xx}(α)P_{m}] | dα = ∫_{−∞}^{∞} ∑_{k=m+1}^{n} S_{x_{k}x_{k}}(α) dα

= ∑_{k=m+1}^{n} R_{x_{k}x_{k}}(0) = tr [Q_{m}R_{xx}(0)Q_{m}] = tr [Q_{m}V^{∗}R_{zz}(0)VQ_{m}]
= γ_{2}(P_{m}).

4 Optimal projection

This section considers the problem of finding γ-optimal sequences (P_{m}^{∗})_{0≤m≤n} with respect to the set N of orthogonal projectors, which are also γ-optimal with respect to the set M of all sequences of projectors (see Lemma 1). The optimization problem will be solved for the case of wide-sense stationary vector functions z(t, ω).

First we note that the relations

γ_{1}(P_{m}) ≥ 0 and γ_{1}(P_{m}) = 0 for m_{z} = v_{0}  (17)

hold for (P_{m})_{0≤m≤n} ∈ M. To find a lower bound for γ_{2}(P_{m}) the eigenvalue decomposition of the variance matrix R_{zz}(0) is considered. R_{zz}(0) is assumed to be non-singular, then it is a hermitian (R_{zz}^{∗}(0) = R_{zz}(0) ) and (strictly) positive definite matrix. As a standard result of algebraic eigenvalue theory it is known that R_{zz}(0) possesses n real and (strictly) positive eigenvalues σ_{1}^{2} ≥ … ≥ σ_{n}^{2} > 0 and that n pairwise orthonormal eigenvectors u_{1}, …, u_{n} ∈ Z exist which form an unitary matrix U = (u_{1}, …, u_{n}) (i.e. U^{−1} = U^{∗}). The decomposition

R_{zz}(0) = UΣU^{∗}

with Σ = diag(σ_{1}^{2}, …, σ_{n}^{2}) can be derived.

Lemma 6 Let z(t, ω) be a wide-sense stationary vector function with a non-singular variance matrix R_{zz}(0) which possesses the eigenvalues σ_{1}^{2} ≥ … ≥ σ_{n}^{2} > 0. Then it holds for all sequences (P_{m})_{0≤m≤n} ∈ N (of orthogonal projection operators) the inequality

γ_{2}(P_{m}) ≥ ∑_{k=m+1}^{n} σ_{k}^{2}.

In case of V = U the equality is fulfilled where V denotes the matrix belonging to (P_{m})_{0≤m≤n} (see Eq. (3)).

Proof: First the assertion concerning the equality for V = U will be proved. Applying the decomposition R_{zz}(0) = UΣU^{∗}, the relation U^{∗}U = I_{n} and the representation (5) of the matrix Q_{m} it can be found that

γ_{2}(P_{m}) = tr [Q_{m}V^{∗}R_{zz}(0)VQ_{m}]
= tr [Q_{m}U^{∗}(UΣU^{∗})UQ_{m}] = tr [Q_{m}ΣQ_{m}]
= tr [( \begin{matrix}O_{m,m} & O_{m,n−m} \\ O_{n−m,m} & Σ_{n−m}^{c} \end{matrix} )] = ∑_{k=m+1}^{n} σ_{k}^{2}

where Σ_{n−m}^{c} is a n − m × n − m submatrix of the matrix Σ, which is defined by Σ_{n−m}^{c} := diag(σ_{m+1}^{2}, …, σ_{n}^{2}).

Now we deal with the general assertion. Since in case of (P_{m})_{0≤m≤n} ∈ N the matrix V is unitary it follows that the matrix B := V^{∗}R_{zz}(0)V is the result of a similarity transformation of the variance matrix R_{zz}(0). Therefore B and R_{zz}(0) possess the same positive eigenvalues σ_{1}^{2}, …, σ_{n}^{2}. Further it holds

P_{m}BP_{m} = ( \begin{matrix}B_{m} & O_{m,n−m} \\ O_{n−m,m} & O_{n−m,n−m} \end{matrix} )

where B_{m} denotes the m × m principal submatrix of B possessing the eigenvalues μ_{1} ≥ … ≥ μ_{m}. The eigenvalues of a principal submatrix of the hermitian matrix B fulfill the inequalities

μ_{k} ≤ σ_{k}^{2}, k = 1, …, m (see [2], p. 186)

and it follows

tr [B_{m}] = ∑_{k=1}^{m} μ_{k} ≤ ∑_{k=1}^{m} σ_{k}^{2}.

This inequality and the identity Q_{m} = I_{n} − P_{m} lead to

γ_{2}(P_{m}) = tr [Q_{m}V^{∗}R_{zz}(0)VQ_{m}] = tr [(I_{n} − P_{m})B(I_{n} − P_{m})]
= tr [B] − tr [P_{m}B] − tr [BP_{m}] + tr [P_{m}BP_{m}]
= ∑_{k=1}^{n} σ_{k}^{2} − ∑_{k=1}^{m} μ_{k} − ∑_{k=1}^{m} μ_{k} + ∑_{k=1}^{m} μ_{k}
≥ ∑_{k=1}^{n} σ_{k}^{2} − ∑_{k=1}^{m} σ_{k}^{2} = ∑_{k=m+1}^{n} σ_{k}^{2}.

The proof is complete.

Applying the preceding results for γ_{1}(P_{m}) and γ_{2}(P_{m}) we are now able to find γoptimal projections.

Theorem 1 Let z(t, ω) be a wide-sense stationary vector function with

• mean m_{z} = E {z(t)}

• and a non-singular variance matrix R_{zz}(0) = E {(z(t) − m_{z})(z(t) − m_{z})^{∗}} with eigenvalues σ_{1}^{2} ≥ … ≥ σ_{n}^{2} > 0 and corresponding eigenvectors u_{1}, …, u_{n}.

A sequence \binom{}{P̃_{m}}_{0≤m≤n} of projection operators defined by

v_{0} = m_{z} and v_{k} = u_{k}, k = 1, …, n,

is γ-optimal with respect to M and it holds

γ(P̃_{m}) = ∑_{k=m+1}^{n} σ_{k}^{2}.

Proof: Using the orthonormality of u_{1}, …, u_{n} it follows \binom{}{P̃_{m}}_{0≤m≤n} ∈ N ⊂ M.

First the γ-optimality of \binom{}{P̃_{m}}_{0≤m≤n} with respect to the subset N ⊂ M will be proved. We write

γ(P_{m}) = γ(m, v_{0}, V)
γ_{j}(P_{m}) = γ_{j}(m, v_{0}, V), j = 1, 2,

in order to represent the dependence on v_{0} and V = (v_{1}, …, v_{n}) of the sequence (P_{m})_{0≤m≤n} ∈ N. Now it has to be proved that the inequality

γ(P̃_{m}) = γ(m, m_{z}, U) ≤ γ(m, v_{0}, V)

is true for m = 0, …, n, ∀v_{0} ∈ Z, ∀ V = (v_{1}, …, v_{n}), v_{k} ∈ Z, k = 1, …, n, with (v_{k}, v_{l}) = δ_{kl}, 1 ≤ k, l ≤ n and the eigenvectors U = (u_{1}, …, u_{n}) of R_{zz}(0).

We have seen that γ_{2}(m, v_{0}, V) does not depend on v_{0}, i.e. γ_{2}(m, v_{0}, V) = γ_{2}(m, V).

By means of relation (17) and Lemma 6 it can be deduced

γ(m, v_{0}, V) = γ_{1}(m, v_{0}, V) + γ_{2}(m, V)
≥ γ_{2}(m, V) ≥ γ_{2}(m, U)
= γ_{2}(m, U) + \underbrace{γ_{1}(m, m_{z}, U)}_{=0} = γ(m, m_{z}, U).

Then \binom{}{P̃_{m}}_{0≤m≤n} is γ-optimal with respect to N ⊂ M. Applying Lemma 1 the γ-optimality with respect to the set M can be obtained.

Finally, the equation

γ(m, m_{z}, U) = γ(P̃_{m}) = γ_{2}(P̃_{m}) = ∑_{k=m+1}^{n} σ_{k}^{2}

follows from Lemma 6 and γ_{1}(P̃_{m}) = 0.

Corollary 1 ii) leads to the following assertion.

Corollary 2 Assuming the same conditions as in Theorem 1 the sequence \binom{}{P̃_{m}}_{0≤m≤n} is ρ-optimal with respect to N and ρ(P̃_{m}) is given by

ρ(P̃_{m}) = 1 − \frac{γ(P̃_{m})}{γ(P̃_{0})} = 1 − \frac{∑_{k=m+1}^{n} σ_{k}^{2}}{∑_{k=1}^{n} σ_{k}^{2}} = \frac{∑_{k=1}^{m} σ_{k}^{2}}{∑_{k=1}^{n} σ_{k}^{2}} .

Remark 4 In case of Z = R^{n}, we have real mean m_{z} and real variance matrix R_{zz}(0), which has real eigenvectors. Hence, the transformation matrix V = U and the resulting γ- or ρ-optimal projection P_{m}z is real, too.

Remark 5 The projection used in [5] which is based on the modal decomposition is not γ-optimal, since the transformation matrix used there is the eigenvector matrix of the system matrix of a linear vibration system, which is non-unitary in general.

In contrast to the γ-optimal projection the projection considered in [5] does not demand the knowledge of the variance matrix R_{zz}(0) (as well as its eigenvalues and eigenvectors) of the original system.

Remark 6 The assertion of Theorem 1 corresponds to a result of principal component analysis of multivariate statistics. There, a random vector y is considered which has values in a n-dimensional Euclidean space and a given mean m_{y} and variance matrix R_{yy} (e.g. empirical). A m-dimensional linear submanifold (m < n) which contains the ”maximum of the variance” is found to be spaned by the eigenvectors belonging to the largest eigenvalues of R_{yy} and shifted by the mean m_{y}.

References

[1] A.V. Gnedin. On mean-square epsilon dimension. Math. Jap., 37(4):623–627, 1992.

[2] B.N. Parlett. The symmetric eigenvalue problem. Prentice Hall, Englewood Cliffs, 1980.

[3] M. Richter. Approximation of Gaussian random elements and statistics. Teubner, Leipzig, 1992.

[4] J. vom Scheidt, H.-J. Starkloff, and R. Wunderlich. Low-dimensional approximations of linear vibration systems. to appear.

[5] R. Wunderlich, J. Gruner, and J. vom Scheidt. Model reduction of random vibration systems. In M. Brøns, M.P. Bendsøe, and M.P. Sørensen, editors, Progress in Industrial Mathematics at ECMI 96, pages 382 – 389, Stuttgart, 1997. Teubner.