Efficient Estimation of Conditional Covariance Matrices For Dimension Reduction - Solís, Loubes, Marteau - JDS, Bruxelles 2012

Ecient estimation of conditional covariance matrices
for dimension reduction

Maikol Sols Chacon Jean-Michel Loubes Clement Marteau
Institut de Mathematiques de Toulouse
Universite Paul Sabatier
44
e
Journees de Statistiques
Bruxelles
22 August 2012
Sols Chacon, Loubes, Marteau (IMT) Bruxelles, 22 August 2012 1 / 23
Outline
1
Introduction
2
Methodology
3
Estimation of quadratic functionals
4
Main results
5
Summary
Introduction
Outline
1
Introduction
2
Methodology
3
4
Main results
5
Summary
Introduction
Background: Sliced Inverse Regression [Li, 1991]
Consider the problem
Y = (X) + ,
where X R
p
, Y R and E[] = 0.
Find s such that
Y = (
1
X, . . . ,
K
X, ),
where the s are unknown vectors in R
p
, is independent of X and
is an arbitrary function in R
K+1
. K p.
The eigenvectors of Cov
_
E[X|Y]
_
spans the same subspace that the
s (eective dimension reduction directions or e.d.r.ds).
The objective is to estimate Cov
_
E[X|Y]
_
.
Introduction
Y = (X) + ,
where X R
p
, Y R and E[] = 0.
Find s such that
Y = (
1
X, . . . ,
K
X, ),
p
K+1
. K p.
_
E[X|Y]
_
_
E[X|Y]
_
.
Introduction
Y = (X) + ,
where X R
p
, Y R and E[] = 0.
Find s such that
Y = (
1
X, . . . ,
K
X, ),
p
K+1
. K p.
_
E[X|Y]
_
_
E[X|Y]
_
.
Introduction
Y = (X) + ,
where X R
p
, Y R and E[] = 0.
Find s such that
Y = (
1
X, . . . ,
K
X, ),
p
K+1
. K p.
_
E[X|Y]
_
_
E[X|Y]
_
.
Introduction
Previous work
Estimators for Cov
_
E[X|Y]
_
and the e.d.r.ds can be found using
Kernel estimators, [Zhu & Fang, 1996] and
[Ferre & Yao, 2003,2005].
A combination of the nearest neighbor and SIR, [Hsing 1999].
Assumption that E[X|Y] has some parametric form,
[Bura & Cook, 2001].
K-means, [Setodji & Cook,2004].
Transformation of SIR to least square form,
[Cook & Ni, 2005].
etc.
Introduction
Our idea to estimate Cov
_
E[X|Y]
_
The objective is to estimate directly Cov
_
E[X|Y]
_
, using a plug-in
method in a semiparametric framework.
We aim to estimate E
_
E[X|Y] E[X|Y]
_
as a quadratic functional
(studied by [Da Veiga & Gamboa, 2012] and [Laurent, 1996]).
We will also focus on the ecient estimation of
E
_
E[X|Y] E[X|Y]
_
.
Introduction
_
E[X|Y]
_
_
E[X|Y]
_
, using a plug-in
_
E[X|Y] E[X|Y]
_
E
_
E[X|Y] E[X|Y]
_
.
Introduction
_
E[X|Y]
_
_
E[X|Y]
_
, using a plug-in
_
E[X|Y] E[X|Y]
_
E
_
E[X|Y] E[X|Y]
_
.
Methodology
Outline
1
Introduction
2
Methodology
3
4
Main results
5
Summary
Methodology
Preliminaries
1
Let f (x
i
, x
j
, y) be the joint distribution of (X
i
, X
j
, Y), then dene
T
ij
(f ) = E
_
E[X
i
|Y] E[X
j
|Y]
_
.
2
In general, for f L
2
(dx
i
, dx
j
, dy), dene the non-linear functional
f T
ij
(f ) with T
ij
(f ) having the form
_ _
_
x
i
f (x
i
, x
j
, y)dx
i
dx
j
_
f (x
i
, x
j
, y)dx
i
dx
j
__
_
x
j
f (x
i
, x
j
, y)dx
i
dx
j
_
f (x
i
, x
j
, y)dx
i
dx
j
_
f (x
i
, x
j
, y)dx
i
dx
j
dy.
3
For an i.i.d sample (X
(k)
i
, X
(k)
j
, Y
(k)
), k = 1, . . . , n, we can build a
preliminary estimator

f of f with a subsample n
1
< n.
Methodology
Preliminaries
1
Let f (x
i
, x
j
i
, X
j
, Y), then dene
T
ij
(f ) = E
_
E[X
i
|Y] E[X
j
|Y]
_
.
2
In general, for f L
2
(dx
i
, dx
j
f T
ij
(f ) with T
ij
_ _
_
x
i
f (x
i
, x
j
, y)dx
i
dx
j
_
f (x
i
, x
j
, y)dx
i
dx
j
__
_
x
j
f (x
i
, x
j
, y)dx
i
dx
j
_
f (x
i
, x
j
, y)dx
i
dx
j
_
f (x
i
, x
j
, y)dx
i
dx
j
dy.
3
(k)
i
, X
(k)
j
, Y
(k)
), k = 1, . . . , n, we can build a

1
< n.
Methodology
Preliminaries
1
Let f (x
i
, x
j
i
, X
j
, Y), then dene
T
ij
(f ) = E
_
E[X
i
|Y] E[X
j
|Y]
_
.
2
In general, for f L
2
(dx
i
, dx
j
f T
ij
(f ) with T
ij
_ _
_
x
i
f (x
i
, x
j
, y)dx
i
dx
j
_
f (x
i
, x
j
, y)dx
i
dx
j
__
_
x
j
f (x
i
, x
j
, y)dx
i
dx
j
_
f (x
i
, x
j
, y)dx
i
dx
j
_
f (x
i
, x
j
, y)dx
i
dx
j
dy.
3
(k)
i
, X
(k)
j
, Y
(k)
), k = 1, . . . , n, we can build a

1
< n.
Methodology
Taylors development
Dene the auxiliar function F : [0, 1] R as
F(u) = T
ij
(uf + (1 u)
f ) with u [0, 1].

We have F(1) = F(0) + F
(0) +
1
2
F
(0) +
1
6
F
()(1 )
3
for some
[0, 1].
Notice that F(1) = T
ij
(f ) and F(0) = T
ij
(
f ).
Methodology
Taylors development
F(u) = T
ij
(uf + (1 u)
f ) with u [0, 1].

We have F(1) = F(0) + F
(0) +
1
2
F
(0) +
1
6
F
()(1 )
3
for some
[0, 1].
ij
(f ) and F(0) = T
ij
(
f ).
Methodology
Taylors development
F(u) = T
ij
(uf + (1 u)
f ) with u [0, 1].

We have F(1) = F(0) + F
(0) +
1
2
F
(0) +
1
6
F
()(1 )
3
for some
[0, 1].
ij
(f ) and F(0) = T
ij
(
f ).
Methodology
Taylors development
We obtain,
T
ij
(f ) =
_
H
1
(
f , x
i
, x
j
, y)f (x
i
, x
j
, y)dx
i
dx
j
dy
. .
Linear Functional (LF)
+
_
H
2
(
f , x
i 1
, x
j 2
, y)f (x
i 1
, x
j 1
, y)f (x
i 2
, x
j 2
, y)dx
i 1
dx
j 1
dx
i 2
dx
j 2
dy
. .
Quadratic Functional (QF)
+
n
..
Error
Methodology
Estimating (LF) and (QF)
The linear part is direct.
Dene n
2
= n n
1
. We estimate (LF) by
1
n
2
n
2
k=1
H
1
_
f , X
(k)
i
, X
(k)
j
, Y
(k)
_
.
Core of the work: Main issue.
To estimate (QF), we will build an estimator of
(f ) =
_
(x
i 1
, x
j 2
, y)f (x
i 1
, x
j 1
, y)f (x
i 2
, x
j 2
, y)dx
i 1
dx
j 1
dx
i 2
dx
j 2
dy
where : R
3
R is a bounded function.
Methodology
Estimating (LF) and (QF)
The linear part is direct.
Dene n
2
= n n
1
. We estimate (LF) by
1
n
2
n
2
k=1
H
1
_
f , X
(k)
i
, X
(k)
j
, Y
(k)
_
.
Core of the work: Main issue.
To estimate (QF), we will build an estimator of
(f ) =
_
(x
i 1
, x
j 2
, y)f (x
i 1
, x
j 1
, y)f (x
i 2
, x
j 2
, y)dx
i 1
dx
j 1
dx
i 2
dx
j 2
dy
where : R
3
R is a bounded function.
Outline
1
Introduction
2
Methodology
3
4
Main results
5
Summary
Estimation of (f ): Projection method.
Assumption
S
M
n
f f
2
2
0 where S
M
n
f =
l M
n
a
l
p
l
(p
l
is a basis of
L
2
(dx
i
, dx
j
, dy)).
_
sup
l / M
n
|c
l
|
2
_
2
|M
n
| /n
2
for c
l
a xed sequence.
We build an estimator of using a projection scheme with Bias equal
to
_
(S
M
n
f (x
i 1
, x
j 1
, y)f (x
i 1
, x
j 1
, y))(S
M
n
f (x
i 2
, x
j 2
, y)f (x
i 2
, x
j 2
, y))
(x
i 1
, x
j 2
, y)dx
i 1
dx
j 1
dx
i 2
dx
j 2
dy
(AMSE) E
_
_
2
= O
_
1
n
_
Assumption
S
M
n
f f
2
2
0 where S
M
n
f =
l M
n
a
l
p
l
(p
l
is a basis of
L
2
(dx
i
, dx
j
, dy)).
_
sup
l / M
n
|c
l
|
2
_
2
|M
n
| /n
2
for c
l
a xed sequence.
to
_
(S
M
n
f (x
i 1
, x
j 1
, y)f (x
i 1
, x
j 1
, y))(S
M
n
f (x
i 2
, x
j 2
, y)f (x
i 2
, x
j 2
, y))
(x
i 1
, x
j 2
, y)dx
i 1
dx
j 1
dx
i 2
dx
j 2
dy
(AMSE) E
_
_
2
= O
_
1
n
_
Assumption
S
M
n
f f
2
2
0 where S
M
n
f =
l M
n
a
l
p
l
(p
l
is a basis of
L
2
(dx
i
, dx
j
, dy)).
_
sup
l / M
n
|c
l
|
2
_
2
|M
n
| /n
2
for c
l
a xed sequence.
to
_
(S
M
n
f (x
i 1
, x
j 1
, y)f (x
i 1
, x
j 1
, y))(S
M
n
f (x
i 2
, x
j 2
, y)f (x
i 2
, x
j 2
, y))
(x
i 1
, x
j 2
, y)dx
i 1
dx
j 1
dx
i 2
dx
j 2
dy
(AMSE) E
_
_
2
= O
_
1
n
_
Main results
Outline
1
Introduction
2
Methodology
3
4
Main results
5
Summary
Main results
Putting (LF) and (QF) together
T
(n)
ij
=
1
n
2
n
2
k=1
H
1
(
f , X
(k)
i
, X
(k)
j
, Y
(k)
)+Estimator of (QF)
The estimator of (QF) has two sums which depends on the basis p
l
chosen for the projection.
The term
n
is negligible.
Main results
Asymptotic normality of

T
(n)
ij
Theorem
Under some technical assumptions and |M
n
| /n 0 when n . We
have:
n
_
T
(n)
ij
T
ij
(f )
_
D
N (0, C
ij
(f )) ,
and
lim
n
nE
_
T
(n)
ij
T
ij
(f )
_
2
= C
ij
(f ),
where
C
ij
(f ) = Var
_
H
1
(f , X
i
, X
j
, Y)
_
Conclusion: There is asymptotic normality.
Main results
Semi-parametric Cramer-Rao bound
Theorem
Under the same conditions as before, for any estimator

T
(n)
ij
of T
ij
(f ) and
every family {V
r
(f )}
r >0
of neighborhoods of f we have
inf
{V
r
(f )}
r >0
liminf
n
sup
f V
r
(f )
nE
_
T
(n)
ij
T
ij
(f )
_
2
C
ij
(f ).
Conclusion: The estimator is ecient.
Generalization via half-vectorization to matrices: T(f ) = (T
ij
(f ))
pp
and
H
1
(f ) = (H
1
(f , x
i
, x
j
, y))
pp
.
n vech
_
T
(n)
T(f )
_
D
N (0, C(f )) ,
C(f ) = Cov
_
vech(H
1
(f ))
_
Main results
Semi-parametric Cramer-Rao bound
Theorem
Under the same conditions as before, for any estimator

T
(n)
ij
of T
ij
(f ) and
every family {V
r
(f )}
r >0
of neighborhoods of f we have
inf
{V
r
(f )}
r >0
liminf
n
sup
f V
r
(f )
nE
_
T
(n)
ij
T
ij
(f )
_
2
C
ij
(f ).
Conclusion: The estimator is ecient.
Generalization via half-vectorization to matrices: T(f ) = (T
ij
(f ))
pp
and
H
1
(f ) = (H
1
(f , x
i
, x
j
, y))
pp
.
n vech
_
T
(n)
T(f )
_
D
N (0, C(f )) ,
C(f ) = Cov
_
vech(H
1
(f ))
_
Summary
Outline
1
Introduction
2
Methodology
3
4
Main results
5
Summary
Summary
Summary
We use a plug-in method used by
[Laurent, 1996, Da Veiga & Gamboa, 2012] to nd Cov
_
E[X|Y]
_
.
Taylors development on E
_
E[X|Y] E[X|Y]
_
(semi-parametric
framework)
Projection method to estimate the non-linear part.
Asymptotically normal and ecient.
Generalization of the asymptotic normality to the whole matrix.
Non-adaptative: the method depends directly on the regularity of f .
Future work
Numerical results for this method (In progress).
Analyze rates of convergence exploring other techniques like kernel
estimators.
Summary
Summary
_
E[X|Y]
_
.
_
E[X|Y] E[X|Y]
_
(semi-parametric
framework)
Future work
estimators.
Summary
Summary
_
E[X|Y]
_
.
_
E[X|Y] E[X|Y]
_
(semi-parametric
framework)
Future work
estimators.
Summary
Summary
_
E[X|Y]
_
.
_
E[X|Y] E[X|Y]
_
(semi-parametric
framework)
Future work
estimators.
Summary
Summary
_
E[X|Y]
_
.
_
E[X|Y] E[X|Y]
_
(semi-parametric
framework)
Future work
estimators.
Summary
Summary
_
E[X|Y]
_
.
_
E[X|Y] E[X|Y]
_
(semi-parametric
framework)
Future work
estimators.
Summary
Summary
_
E[X|Y]
_
.
_
E[X|Y] E[X|Y]
_
(semi-parametric
framework)
Future work
estimators.
Summary
Summary
_
E[X|Y]
_
.
_
E[X|Y] E[X|Y]
_
(semi-parametric
framework)
Future work
estimators.
Summary
Summary
_
E[X|Y]
_
.
_
E[X|Y] E[X|Y]
_
(semi-parametric
framework)
Future work
estimators.
Thank you
References
Da Veiga, S. & Gamboa, F. (2012).
Ecient estimation of sensitivity indices.
Arxiv Preprint ArXiv:1203.2899.
Laurent, B. (1996).
Ecient estimation of integral functionals of a density.
The Annals of Statistics, 24(2), 659681.
Li, K. C. (1991).
Sliced inverse regression for dimension reduction.
Journal of the American Statistical Association, 86(414), 316327.
Sols Chacon, M., Loubes, J. M., Marteau, C., & Da Veiga, S. (2011).
Ecient estimation of conditional covariance matrices for dimension
reduction.
Summary
Hypothesis
Let(p
l
(x
i
, x
j
, y))
l D
an orthonormal basis of L
2
(dx
i
dx
j
dy) (D
countable).
E =
_
l D
a
l
p
l
: (a
l
)
l D
such that
l D
a
l
c
l
2
< 1
_
L
2
(dx
i
dx
j
dy)
where (c
l
)
l D
is a xed sequence.
f E.
Let (M
n
)
n1
a subset sequence of D. For any n there is M
n
such that
M
n
D. Denote by |M
n
| the cardinal of M
n
.
Summary
Assumptions
1
For any n 1cmmere is M
n
D such that
_
sup
l / M
n
|c
l
|
2
_
2
|M
n
| /n
2
. Moreover, f L
2
(dxdydz),
_
(S
M
n
f f )
2
dxdydz 0 when n 0, where S
M
n
f =
l M
n
a
l
p
l
.
2
supp f [d
1
, b
1
] [d
2
, b
2
] [d
3
, b
3
] and (x, y, z) supp f ,
0 < f (x, y, z) with , R.
3
We can build

f , such that > 0,
(x, y, z) supp f , 0 <
f (x, y, z) +
y 2 q +, l N
, E
f
_
_
f f
_
_
l
q
C(q, l )n
l
1
for > 1/6 and a constant C(q, l ) no depending of f E.
Summary
Appendix: Estimator (LF) and (QF)
T
(n)
ij
=
1
n
2
n
2
k=1
H
1
(
f , X
(k)
i
, X
(k)
j
, Y
(k)
)
+
1
n
2
(n
2
1)
l M
n
2
k=k
=1
p
l
_
X
(k)
i
, X
(k)
j
, Y
(k)
_
_
p
l
_
x
i
, x
j
, Y
(k
)
_
H
3
_
f , x
i
, x
j
, X
(k
)
i
, X
(k
)
j
, Y
(k
)
_
dx
i
dx
j
1
n
2
(n
2
1)
l ,l
M
n
2
k=k
=1
p
l
_
X
(k)
i
, X
(k)
j
, Y
(k)
_
p
l
_
X
(k
)
i
, X
(k
)
j
, Y
(k
)
_
_
p
l
(x
i 1
, x
j 1
, y)p
l
(x
i 2
, x
j 2
, y)H
2
_
f , x
i 1
, x
j 2
, y)dx
i 1
dx
j 1
dx
i 2
dx
j 2
dy.
where H
3
(f , x
i 1
, x
j 1
, x
i 2
, x
j 2
, y) = H
2
(f , x
i 1
, x
j 2
, y) + H
2
(f , x
i 2
, x
j 1
, y) and
n
2
= n n
1
.

Efficient Estimation of Conditional Covariance Matrices For Dimension Reduction - Solís, Loubes, Marteau - JDS, Bruxelles 2012

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Efficient Estimation of Conditional Covariance Matrices For Dimension Reduction - Solís, Loubes, Marteau - JDS, Bruxelles 2012

Uploaded by

Copyright:

Available Formats

Ecient estimation of conditional covariance matrices

for dimension reduction

f ) with u [0, 1].

f ) with u [0, 1].

f ) with u [0, 1].

You might also like