RACSAM (2013) 107:79–89 DOI 10.1007/s13398-012-0089-z ORIGINAL PAPER

Hausdorff limits of Rolle leaves Jean-Marie Lion · Patrick Speissegger To Professor Heisuke Hironaka on the occasion of his 80th birthday

Received: 23 November 2011 / Accepted: 24 July 2012 / Published online: 14 August 2012 © Springer-Verlag 2012

Abstract Let R be an o-minimal expansion of the real field. We introduce a class of Hausdorff limits, the T ∞ -limits over R, that do not in general fall under the scope of Marker and Steinhorn’s definability-of-types theorem. We prove that if R admits analytic cell decomposition, then every T ∞ -limit over R is definable in the pfaffian closure of R. Keywords limits

O-minimal structures · Pfaffian systems · Analytic stratification · Hausdorff

Mathematics Subject Classification (2000)

Primary 14P10 · 58A17; Secondary 03C99

Introduction We fix an o-minimal expansion R of the real field. In this paper, we study T ∞ -limits over R as defined in Sect. 1 below; they generalize the pfaffian limits over R introduced in [5, Section 4]. Pfaffian limits over R are definable in the pfaffian closure P (R) of R [7], by the variant of Marker and Steinhorn’s definability-of-types theorem [6] found in van den Dries [1, Theorem 3.1] and our paper [4, Theorem 1]. The T ∞ -limits over R considered here do not seem to fall under the scope of these theorems, as explained in Sect. 1 below. Nevertheless, T ∞ -limits were used by Lion and Rolin [3] to establish the o-minimality of the expansion of Ran by all Rolle leaves over Ran of codimension one. To state our results, we work in the setting of [5, Introduction]; in particular, recall that a set W ⊆ Rn is a Rolle leaf over R if there exists a nested Rolle leaf (W0 , . . . , Wk ) over R

J.-M. Lion IRMAR, Université de Rennes I, Campus de Beaulieu, 35042 Rennes Cedex, France e-mail: [email protected] P. Speissegger (B) Department of Mathematics and Statistics, McMaster University, 1280 Main Street West, Hamilton, ON L8S 4K1, Canada e-mail: [email protected]

80

J.-M. Lion, P. Speissegger

such that W = Wk (again, we refer the reader to Sect. 1 for more details). First, we obtain the following generalization of [3, Théorème 1]. Theorem A. Let N (R) be the expansion of R by all Rolle leaves over R. (1) (2)

There is an o-minimal expansion T ∞ (R) of N (R) in which every T ∞ -limit over R is definable. Let M ⊆ Rn be a bounded, definable C 2 -manifold and d be a definable and integrable nested distribution on M. Let K ⊆ Rn be a T ∞ -limit obtained from d. Then dim K ≤ dim d.

The question then arises how T ∞ (R) relates to the pfaffian closure P (R) of R. Indeed, we do not know in general if T ∞ (R) is interdefinable with N (R) or P (R), or if T ∞ (T ∞ (R)) is interdefinable with T ∞ (R). Based on [5], we can answer such questions under an additional hypothesis: Theorem B. Assume that R admits analytic cell decomposition. (1) (2)

Every T ∞ -limit over P (R) is definable in P (R). The structures T ∞ (R) and P (R) are interdefinable; in particular, T ∞ (R) and T ∞ (T ∞ (R)) are interdefinable.

We view the combination of Theorems A(2) and B(1) as a non-first order extension of [1, Theorem 3.1] and [4, Theorem 1]. Our proofs of these theorems rely heavily on terminology and notation introduced in [5, Introduction and Section 2]. We prove Theorem A in Sect. 3 below using the approach of [7], but based on an adaptation of some results of [5, Section 4] to T ∞ -limits carried out in Sect. 2 below. Theorem B then follows by adapting [5, Proposition 7.1] to T ∞ -limits and using [5, Proposition 10.4]; the details are given in Sect. 4. Conventions. For a set S ⊆ Rn , we denote by cl S the topological closure of S and by int S the topological interior of S. We define the frontier of S as fr S := cl S\S and the boundary of S as bd S := cl S \int S.

1 The definitions Let M ⊆ Rn be a bounded, definable C 2 -manifold of dimension m. We adopt the terminology and results found in [5]; for a more detailed discussion, we refer the reader to [5]. A distribution on M is a map d on M with values in the Grassmannian G n of all linear subspaces of Rn such that d(x) ⊆ g M (x), where g M : M −→ G m n is the Gaussian map defined by g M (x) := Tx M. j

We say that d has dimension if there exists j ≤ m such that d : M −→ G n ; in this case, we call d integrable if the leaves of d form a partition of M. Given distributions d and e on M, we write d ⊆ e if d(x) ⊆ e(x) for all x, and we denote by d ∩ e the distribution defined by (d ∩ e)(x) := d(x) ∩ e(x). Moreover, if N is a C 2 -submanifold of M and d is a distribution on M, we define the pullback d N of d to N as d N := g N ∩ d N ,

Hausdorff limits of Rolle leaves

81

where d  N denotes the restriction of d to N . A set D of distributions on M is compatible  N has dimension, for every E ⊆ D. with a C 2 -submanifold N of M if the pullback d∈E d We now let d = (d0 , . . . , dk ) be an integrable nested distribution on M; that is, dk ⊆ · · · ⊆ d0 = g M and each d j is integrable. In this situation, a tuple V = (V0 , . . . , Vk ) is a nested integral manifold (nested leaf) of d if each V j is an integral manifold (leaf) of d j and Vk ⊆ · · · ⊆ V0 = M; if, in addition, each V j for j > 0 is a Rolle leaf of d j  V j−1 , we call the tuple V a nested Rolle leaf of d and, for convenience, each V j a Rolle leaf of d. Finally, a (nested) Rolle leaf V of d is called a (nested) Rolle leaf over R whenever d is definable. One of the key definitions in [5] is that of a core distribution of d, which extracts the “non-definable content” of d in the following sense: by integrability, M is partitioned by the leaves of each d j , represented by an equivalence relation ∼d j on M given by d

d

x ∼d j y if and only if L x j = L y j , d

where L x j denotes the unique leaf of d j containing x. Note that, if ∼d j is definable, then so is every leaf of d j and, by o-minimality, there is a definable set of representatives of ∼d j . Thus, we define dim d := m − k and   deg d :=  j ∈ {0, . . . , k} : ∼d j is not definable  ≤ k, and an integrable, definable nested distribution e = (e0 , . . . , dl ) on M, with l ≤ k, is a core distribution of d if (i) (ii)

∼d j is definable for j = 1, . . . , k − l; d j = dk−l ∩ e j−k+l for j = k − l + 1, . . . , k.

Note that, in this situation, we have deg d ≤ deg e. Finally, a nonempty integral manifold V of dk is an admissible integral manifold of d if d has a core distribution e = (e0 , . . . , el ) and there are a definable, closed integral manifold B of dk−l and a Rolle leaf W of e such that V = W ∩ B. In this situation, we call W the core of V corresponding to e and B a definable part of V corresponding to W . We are now ready to generalize [5, Definitions 4.3 and 4.4]. Definition 1.1 A sequence (Vι )ι∈N of integral manifolds of dk is called a T ∞ -sequence of integral manifolds of d if there are a core distribution e = (e0 , . . . , el ) of d, a sequence (Wι ) of Rolle leaves of e and a definable family B of closed integral manifolds of dk−l such that each Vι is an admissible integral manifold of d with core Wι corresponding to e and definable part in B corresponding to Wι . In this situation, we call (Wι ) the core sequence of the sequence (Vι ) corresponding to e and B a definable part of the sequence (Vι ) corresponding to the sequence (Wι ). Remark (1) We think of the core sequence of (Vι ) as extracting the “non-definable content” of (Vι ). If Wι = W1 for all ι, then (Vι ) is an admissible sequence of integral manifolds of d as defined in [5, Definition 4.3]. (2) Let (Vι ) be a T ∞ -sequence of integral manifolds of d. Then there is a T ∞ -sequence (Uι ) of integral manifolds of (d0 , . . . , dk−1 ) such that Vι ⊆ Uι for ι ∈ N. Definition 1.2 Let (Vι ) be a T ∞ -sequence of integral manifolds of d. If (Vι ) converges to K in Kn (which, since M is bounded, means by definition that the sequence (cl Vι ) converges to K in the space Kn of all compact subsets of Rn equipped with the Hausdorff metric), we call K a T ∞ -limit over R. In this situation, we say that K is obtained from d, and we put deg K := min {deg f : K is obtained from f } .

82

J.-M. Lion, P. Speissegger

Remark (1) It is unknown1 whether the family of all Rolle leaves of e is definable in P (R). As a consequence, contrary to the situation described by [5, Lemma 4.5] for pfaffian limits over R, the variant of Marker and Steinhorn’s definability-of-types theorem [6] found in [1, Theorem 3.1] and [4, Theorem 1] does not apply; in particular, we do not know in general whether a T ∞ -limit over R is definable in P (R). (2) If Wι = W1 for all ι, then K is a pfaffian limit over R as introduced in [5, Definition 4.4]. 2 Towards the proof of Theorem A Let M ⊆ Rn be a definable C 2 -manifold of dimension m. Pfaffian fiber cutting. We fix a finite family  = {d 1 , . . . , d q } of definable nested distrip p butions on M; we write d p = (d0 , . . . , dk( p) ) for p = 1, . . . , q. As in [5, Section 3], we associate to  the following set of distributions on M:  p−1 p 1 D := d00 ∩ dk(1) ∩ · · · ∩ dk( p−1) ∩ d j : p = 1, . . . , q and j = 0, . . . , k( p) , where we put d00 := g M . If N is a C 2 -submanifold of M compatible with D , we let ,N d ,N = d0,N , . . . , dk(,N ) be the nested distribution on N obtained by listing the set  N  g : g ∈ D in order of decreasing dimension. In this situation, if V p is an integral manp ifold of dk( p) , for p = 1, . . . , q, then the set N ∩ V1 ∩ · · · ∩ Vq is an integral manifold of ,N dk(,N ). Let A ⊆ M be definable. For I ⊆ {1, . . . , q} we put (I ) := {d p : p ∈ I }.

Lemma 2.1 Let I ⊆ {1, . . . , q}. Then there is a finite partition P of definable C 2 -cells contained in A such that P is compatible with D(J ) for every J ⊆ {1, . . . , q} and (i) (ii)

(I ),N

dim dk((I ),N ) = 0 for every N ∈ P ;  whenever V p is a Rolle leaf of d p for p ∈ I , every component of A ∩ p∈I V p intersects some cell in P .

Proof By induction on dim A; if dim A = 0, there is nothing to do, so we assume dim A > 0 and the corollary is true for lower values of dim A. By [5, Proposition 2.2] and the inductive hypothesis, we may assume that A is a C 2 -cell compatible with D(J ) for J ⊆ {1, . . . , q}. (I ),A Thus, if dim dk((I ),A) = 0, we are done; otherwise, we let φ and B be as in [5, Lemma 3.1] with (I ) in place of . Let V p be a Rolle leaf of d p for each p; it suffices to show that every component of  (I ),A X := A ∩ p∈I V p intersects B. However, since dk((I ),A) has dimension, X is a closed, embedded submanifold of A. Thus, φ attains a maximum on every component of X , and any point in X where φ attains a local maximum belongs to B.

Corollary 2.2 Let d be a definable nested distribution on M and ν ≤ n. Then there is a finite partition P of C 2 -cells contained in A such that for every Rolle leaf V of d, we have ν (A ∩ V ) = ν (N ∩ V ) N ∈P 1 For instance, a positive answer to this question for all e definable in P(R) would imply the second part of

Hilbert’s 16th problem.

Hausdorff limits of Rolle leaves

83

and for every N ∈ P , the set N ∩ V is a submanifold of N , ν (N ∩V ) is an immersion and for every n  ≤ n and every strictly increasing λ : {1, . . . , n  } −→ {1, . . . , n}, the projection λ(N ∩V ) has constant rank. Proof Apply Lemma 2.1 with q := n + 1, d p := ker d x p for p = 1, . . . , n, d q := d and I := {1, . . . , k, n + 1}.

T ∞ -limits We assume that M has a definable C 2 -carpeting function φ (that is, a C 2 function φ : M −→ (0, ∞) that is proper and satisfies lim x→y φ(x) = 0 whenever y ∈ fr(M)), and we let d = (d0 , . . . , dk ) be a definable distribution on M with core distribution e = (e0 , . . . , el ). First, we reformulate [5, Proposition 4.7]. Proposition 2.3 Let (Vι ) be a T ∞ -sequence of integral manifolds of d with core sequence (Wι ), and assume that K  := limι fr Vι exists. Then there exist q ∈ N, a definable open M  ⊆ M and a definable nested distribution d  on M  , all independent of ι, such that (1) (2) (3)



d  has core distribution d M , deg d  ≤ deg d and dim d  < dim d;    each Wι ∩ M  is the union of Rolle leaves (Wι )1M , . . . , (Wι )qM of d M ;  of T ∞ -limits obtained from d  with core sequences among K is Ma finite union   (Wι )1 ι , . . . , (Wι )qM ι .

Proof We adopt the notation introduced before [5, Proposition 4.6] (which, in particular,  defines M  and d  := dφM ) and note that the q in [5, Remark 4.2] can be chosen independent of W . The proof of the proposition now proceeds exactly as that of [5, Proposition 4.7],  except for replacing “core W ” by “core sequence (Wι )” and “core W pM ” by “core sequence   (Wι ) M p ”. Second, as we do not know yet whether T ∞ -limits are definable in an o-minimal structure, we work with the following notion of dimension (see also van den Dries and Speissegger [2, Section 8.2]): we call N ⊆ Rn a C 0 -manifold of dimension p if N = ∅ and each point of N has an open neighbourhood in N homeomorphic to R p ; in this case p is uniquely determined (by a theorem of Brouwer), and we write p = dim(N ). Correspondingly, a set S ⊆ Rn has dimension if S is a countable union of C 0 -manifolds, and in this case put

max{dim(N ) : N ⊆ S is a C 0 -manifold} if S = ∅ dim(S) := −∞ otherwise.  It follows (by a Baire category argument) that, if S = i∈N Si and each Si has dimension, then S has dimension and dim(S) = max{dim(Si ) : i ∈ N}. Thus, if N is a C 1 -manifold of dimension p, then N has dimension in the sense of this definition and the two dimensions of N agree. Hausdorff limits have dimension in the following situation: for η > 0, the manifold M is called η-bounded if, for x ∈ M, there is a matrix L = (li j ) ∈ Mn−m,m (R) such that |L| := supi, j |li j | ≤ η and g M (x) is the graph of the linear map u  → Lu : Rm −→ Rn−m . Lemma 2.4 Let (Vι ) be a sequence of submanifolds of M of dimension p ≤ m. Let η > 0, and assume that each Vι is η-bounded. Moreover, assume that both limι Vι and limι fr Vι exist and there exists ν ∈ N such that for every ι and every open box U ⊆ Rn , the set Vι ∩ U has at most ν components. Then the set limι Vι \ limι fr Vι is either empty or has dimension p.

84

J.-M. Lion, P. Speissegger

Proof The lemma follows directly from [5, Lemma 1.5].



Therefore, we replace [5, Lemma 4.5] by Proposition 2.5 Let K be a T ∞ -limit obtained from d. Then K has dimension and satisfies dim K ≤ dim d. Proof Let (Vι ) be a T ∞ -sequence of integral manifolds of d such that K = limι Vι . We proceed by induction on dim d. If dim d = 0, then [5, Corollary 3.3(2)] gives a uniform bound on the cardinality of Vι , so K is finite. So assume dim d > 0 and the corollary holds for lower values of dim d. By [5, Proposition 2.2 and Remark 4.2], we may assume that M is a definable C 2 -cell; in particular, there is a definable C 2 -carpeting function φ on M. For each σ ∈ 

n , let Mσ,2 be as before [5, Lemma 1.3] with dk in place of d. Then by that lemma, M = σ ∈ n Mσ,2 and each Mσ,2 is an open subset of M. Hence d is compatible with each Mσ,2 , and after passing to a subsequence if necessary,  we may assume that K σ = limι (Vι ∩ Mσ,2 ) exists for each σ . It follows that K = σ ∈ n K σ , so by [5, Lemma 1.3(2)], after replacing M with each σ −1 (Mσ,2 ), we may assume that dk is 2-bounded. Passing to a subsequence again, we may assume that K  := limι fr Vι exists as well. Then by Lemma 2.4, the set K \ K  is either empty or has dimension dim d. By Proposition 2.3, the set K  is a finite union of T ∞ -limits obtained from a definable nested distribution d  on a definable manifold M  that satisfies deg d  ≤ deg d and dim d  < dim d. So K  has dimension with dim K  < dim d by the inductive hypothesis, and the proposition is proved.

Definition 2.6 A T ∞ -limit K ⊆ Rn obtained from d is proper if dim K = dim d. Corollary 2.7 Let K ⊆ Rn be a T ∞ -limit obtained from d. Then K is a finite union of proper T ∞ -limits over R of degree at most deg d. Proof We proceed by induction on dim d; as in the previous proof, we assume dim d > 0 and the corollary holds for lower values of dim d. If dim K = dim d, we are done, so assume that dim K < dim d. Also as in the previous proof, we now reduce to the case where dk is 2-bounded and K  := limι fr Vι exists. Then Lemma 2.4 implies that K = K  , so the corollary follows from Proposition 2.3 and the inductive hypothesis.

Finally, T ∞ -limits over R are well behaved with respect to intersecting with closed definable sets. To see this, define M := M × (0, 1) and write (x, ) for the typical element of M with x ∈ M and  ∈ (0, 1). We consider the components of d as distributions on M in the obvious way, and we set d0 := gM , d1 := d M and d1+i := di ∩ d1 for i = 1, . . . , k and put d := (d0 , . . . , d1+k ). Moreover, whenever e is a core distribution of d, we similarly define a corresponding core distribution e = (e0 , . . . , e1+l ) of d. In this situation, for every Rolle leaf W of e and every  ∈ (0, 1), the set W := W × {} is a Rolle leaf of e. Proposition 2.8 Let K be a T ∞ -limit obtained from d, and let C ⊆ Rn be a definable closed set. Then there are a definable open subset N of M and q ∈ N, both independent of K , and there are T ∞ -limits K 1 , . . . , K q ⊆ Rn+1 obtained from dN such that K ∩ C = n (K 1 ) ∪ · · · ∪ n (K q ). n Proof For  > 0 put  T (C, ) := {x ∈ R : d(x, C)  < }. Note first that K ∩ C =  K ∩ T (C, ) , and the latter is equal to limκ K ∩ T (C, κ ) , for every sequence >0 (κ )κ∈N of positive real numbers such that limκ κ = 0. Next, let (Vι ) be a T ∞ -sequence of

Hausdorff limits of Rolle leaves

85

integral manifolds of d such that K = limι Vι . Then for every  > 0, there is a subsequence (ι(κ)) of (ι) such that the sequence (Vι(κ) ∩ T (C, )) converges to some compact set K  . Note that K  ∩ T (C, ) = K ∩ T (C, ), since T (C, ) is an open set. Fix a sequence (κ ) of positive real numbers approaching 0, and for each κ, choose ι(κ) such that d(Vι(κ) ∩ T (C, κ ), K  to a subsequence if necessary, we may  κ ) < κ . Passing assume that limκ K κ and limκ Vι(κ) ∩ T (C, κ ) exist; note that these limits are then equal. Hence by the above,     K ∩ C = limκ K ∩ T (C, κ ) = limκ K κ ∩ T (C, κ )   ⊆ limκ K κ = limκ Vι(κ) ∩ T (C, κ ) .   The reverse inclusion is obvious, so K ∩ C = limκ Vι(κ) ∩ T (C, κ ) . Therefore, put N := {(x, ) ∈ M : d(x, C) < }; then N is an open, definable subset of M and by the above K ∩ C = limκ (Vι(κ) ∩ Nκ ), where N := {x ∈ M : (x, ) ∈ N}. Hence K ∩ C =  limκ n (V × { }) ∩ N . Since lim  = 0, it follows that K ∩ C =  lim (Vι(κ) × κ κ κ n κ ι(κ)    {κ }) ∩ N . Since the sequence Vι(κ) × {κ } is a T ∞ -sequence of integral manifolds of d, the proposition now follows from [5, Remark 4.2].

Remark 2.9 Let B and C be two definable families of closed subsets of Rn . Then the T ∞ limits in the previous proposition depend uniformly on C ∈ C , for all T ∞ -limits obtained from d with definable part B. That is, there are μ, q ∈ N, a bounded, definable manifold M ⊆ Rn+μ+1 , a definable nested distribution d on M and a definable family B of subsets of Rn+μ+1 such that whenever K is a T ∞ -limit obtained from d with definable part B and C ∈ C , there are T ∞ -limits K 1 , . . . , K q ⊆ Rn+μ+1 obtained from d with definable part B such that K ∩ C = n (K 1 ) ∪ · · · ∪ n (K q ).

3 O-minimality and proof of Theorem A Similar to [3,7], we show that all sets definable in T ∞ (R) are of the following form: Definition 3.1 Let q ∈ N. A set X ⊆ Rm is a q-basic T ∞ -set if there exist n ≥ m, a definable, bounded C 2 -manifold M ⊆ Rn , a definable nested distribution d on M with core p distribution e and, for κ ∈ N and p ∈ {1, . . . , q}, a T ∞ -sequence (Vκ,ι )ι of integral manifolds p of d with core sequence (Wκ,ι )ι corresponding to e and definable part B independent of κ, such that: (1) (2)

p

p

for each κ and p, the  limit K κ :=qlim  ι Vκ,ι exists in Kn ; the sequence m K κ1 ∪ · · · ∪ K κ κ is increasing and has union X .

In this situation, we say that X is obtained from d with core distribution e and definable part B. For convenience, we call a set a basic T ∞ -set if it is a q-basic T ∞ -set for some q ∈ N. A T ∞ -set is a finite union of basic T ∞ -sets. Proposition 3.2 In the situation of Definition 3.1, there is an N ∈ N such that every q-basic T ∞ -set obtained from d with core distribution e and definable part B has at most q N components. In particular, if X ⊆ Rm is a T ∞ -set and l ≤ m, there is an N ∈ N such that for every a ∈ Rl the fiber X a has at most N components. Proof Let N be a bound on the number of components of the sets W ∩ B as W ranges over all Rolle leaves of e and B ranges over B. Let X be a q-basic T ∞ -set as in Definition 3.1. Then

86

J.-M. Lion, P. Speissegger p

p

each Vκ,ι has at most N components, so each K κ has at most N components, and hence X has at most q N components. Combining this observation with Remark 2.9 yields, for every T ∞ -set X ⊆ Rm , a uniform bound on the number of connected components of the fibers of X.

Proposition 3.3 (1) Any coordinate projection of a T ∞ -limit over R is a T ∞ -set. (2) Every bounded definable set is a T ∞ -set. (3) Let d be a definable nested distribution on M := (−1, 1)n and L be a Rolle leaf of d. Then L is a T ∞ -set. Proof (1) is obvious. For (2), let C ⊆ Rn be a bounded, definable cell. By cell decomposition, it suffices that C is aT ∞ -set. Let φ be a definable carpeting function on C. ∞ to show −1 Then C = κ=1 cl φ ((1/κ, ∞)) , so let C := {(x, r ) ∈ C × (0, 1) : φ(x) > r } and put d1 := ker dr C and d := (gC , d1 ). Then for r > 0, the set Cr = φ −1 ((r, ∞)) × {r } is an admissible integral manifold of d with core C and definable part Cr , so cl Cr = limι Cr  ∞ −1 is a T -limit obtained from d. Since cl φ ((r, ∞)) = n (cl Cr ), it follows that C is a 1-basic T ∞ -set. (3) Let φ be a carpeting function on M. Then L=

∞

  cl L ∩ φ −1 ((1/κ, ∞)) ,

κ=1

so we let M := {(x, r ) ∈ M × (0, 1) : φ(x) > r } and put d0 := gM , d1 := ker dr M , d1+i := d1 ∩ di for i = 1, . . . , k and d := (d0 , . . . , d1+k ). Let L 1 , . . . , L q be the components of (L × (0, 1)) ∩ M; note that each L p is a Rolle leaf of d. Thus for r > 0 and each p, the set L p ∩ φ −1 ((r, ∞)) is an admissible integral manifold of d with core L p and definable part Mr = φ −1 ((r, ∞)) × {r }. Arguing now as for part (2), it follows that L is a q-basic T ∞ -set.

Proposition 3.4 The topological closure of a bounded T ∞ -set is a finite union of projections of T ∞ -limits over R; in particular, it is a T ∞ -set. Proof Let X ⊆ Rm be a bounded, basic T ∞ -set with associated data as in Definition 3.1. Then ⎛ ⎞ q q q



 p p = K κp ⎠ = m lim lim Vκ,ι m lim Vκ,ι(κ) cl(X ) = lim m ⎝ κ

p=1

p=1

κ

ι

p=1

κ

 p for some subsequence (ι(κ))κ , and each m limκ Vκ,ι(κ) is a T ∞ -set by Proposition 3.3(1).

Proposition 3.5 The collection of all T ∞ -sets is closed under taking finite unions, finite intersections, coordinate projections, cartesian products and permutations of coordinates. Proof Closure under taking finite unions, coordinate projections and permutations of coordinates is obvious from the definition and the properties of nested pfaffian sets over R. For cartesian products, let X 1 ⊆ Rm 1 and X2 ⊆ Rm2 be basic T ∞ -sets, and let M i ⊆ p n i R , d i = (d0i , . . . , dki i ), ei = (e0i , . . . , elii ) and Vι,κ (i) be the data associated to X i as in Definition 3.1, for p = 1, . . . , q(i), i = 1, 2. We assume that both M 1 and M 2 are connected; the general case is easily reduced to this situation. Define   M := (x, y, u, v) : (x, u) ∈ M 1 and (y, v) ∈ M 2 ,

Hausdorff limits of Rolle leaves

87

where x ranges over Rm 1 , y over Rm 2 , u over Rn 1 −m 1 and v over Rn 2 −m 2 . We interpret d i and ei as sets of distributions on M correspondingly, for i = 1, 2, and we define d := (d01 , . . . , dk11 , dk11 ∩ d12 , . . . , dk11 ∩ dk22 ) and e := (e01 , . . . , el11 , el11 ∩ e12 , . . . , el11 ∩ el22 ). Since M 1 and M 2 are connected, each set   p,q p q := (x, y, u, v) : (x, u) ∈ Vκ,ι (1) and (y, v) ∈ Vκ,ι (2) Vκ,ι is an admissible integral manifold of d with core distribution e. It is now easy to see that p,q p,q for each κ, p and q, the

limit K κ := limι Vκ,ι exists in Kn 1 +n 2 , and that the sequence  p,q m 1 +m 2 is increasing and has union X 1 × X 2 . p,q K κ κ

For intersections, let X 1 , X 2 ⊆ Rm be basic T ∞ -sets. Then X 1 ∩X 2 = m ((X 1 ×X 2 )∩), where  := {(x, y) ∈ Rm × Rm : xi = yi for i = 1, . . . , m}. Therefore, we let X ⊆ Rm be a basic T ∞ -set and C ⊆ Rm be closed and definable, and we show that X ∩ C is a T ∞ -set. Let the data associated to X be as in Definition 3.1, and let M, d and e be associated to that data as before Proposition 2.8. Let also N be the open subset of M given by that proposition with C  := C × Rn−m in place of C. Then by that proposition, there is a q  ∈ N such that, for p p,1 p,q  every κ and p, the set K κ ∩ C  is the union of the projections of T ∞ -limits K κ , . . . , K κ  p, p obtained from dN . Note that each K κ is the limit of a T ∞ -sequence of integral manifolds of q N N d with core distribution e . Since the sequence m (K κ1 ∪ · · · ∪ K κ ) is

 and since 

increasing q q p, p  1  1 m ((K κ ∪ · · · ∪ K κ ) ∩ C ) = m (K κ ∪ · · · ∪ K κ ) ∩ C, the sequence m p, p  K κ

is increasing. Hence X ∩ C is a qq  -basic T ∞ -set.



Proposition 3.6 Let X ⊆ Rm be a bounded T ∞ -set. Then bd X is contained in a closed T ∞ -set with empty interior. Proof We may assume that X is basic; let the data associated to X be given as in Definition 3.1 and write d = (d0 , . . . , dk ). Since X is bounded, we have bd X ⊆

q p=1

  lim bd m (K κp ) ; κ

 p  so we fix p and show that limκ bd m (K κ ) is contained in a T ∞ -set of empty interior. To simplify notation, we omit the superscript p below. Fix an arbitrary κ; since m (K κ ) = limι m (Vκ,ι ) we may assume, by Corollary 2.2, [5, Remark 4.2] and after replacing M if necessary, that k  Vκ,ι is an immersion and has constant rank r ≤ m; in particular, dim(Vκ,ι ) ≤ m. If r < m, then each m (K κ ) has empty interior by Proposition 2.4, so lim bd(m (K κ )) = lim m (K κ ) = m (lim K κ ) = m (lim Vκ,ι(κ) ) κ

κ

κ

κ

for some subsequence (ι(κ)), and we conclude by Propositions 2.5 and 3.3(1) in this case. So assume that r = m; in particular, m (Vκ,ι ) is open for every κ and ι. In this case, since M is bounded, we have bd(m (K κ )) ⊆ m (limι fr Vκ,ι ) for each κ. Hence



 lim bd(m (K κ )) ⊆ m lim lim fr Vκ,ι = m lim fr Vκ,ι(κ) κ

κ

ι

κ

for some subsequence (ι(κ)). Now use Propositions 2.3 and 3.3(1).



Following [8] and [3], and proceeding exactly as in [7, Corollary 3.11 and Proposition 3.12] using the previous propositions, we obtain:

88

J.-M. Lion, P. Speissegger

m ∞ Proposition 3.7 (1) Let X ⊆ R   be a T -set, and let 1 ≤ l ≤ m. Then the set B := l a ∈ R : cl(X a ) = cl(X )a has empty interior. (2) Let X ⊆ [−1, 1]m be a T ∞ -set. Then [−1, 1]m \ X is also a T ∞ -set.

For m ∈ N, let Tm be the collection of all T ∞ -sets X ⊆ I m . Corollary 3.8 The collection T := (Tm )m∈N forms an o-minimal structure on I .



Proof of Theorem A For each m, let τm : Rm −→ (−1, 1)m be the (definable) homeomorphism given by   x1 xm ,..., τm (x1 , . . . , xm ) := , 1 + xm2 1 + x12 and let Sm be the collection of sets τm−1 (X ) with X ∈ Tm . By Corollary 3.8, the collection S := (Sm )m gives rise to an o-minimal expansion T ∞ (R) of R whose definable sets are exactly the members of S . By Proposition 3.3(2), every definable set is definable in T ∞ (R). But if L is a Rolle leaf of a definable nested distribution d on Rn , then τn (L) is a Rolle leaf of the pullback (τn−1 )∗ d. It follows from Proposition 3.3(3) that τn (L) ∈ Tn , so L is definable in T ∞ (R). Therefore, N (R) is a reduct of T ∞ (R) in the sense of definability.



4 An o-minimal observation and proof of Theorem B The proof of Theorem B uses the following observation in o-minimality; we are not aware of it having been published elsewhere. Proposition 4.1 Let S be an o-minimal expansion of R and assume that, for every bounded set X definable in S , there exists a set Y definable in R such that X ⊆ Y and dim X = dim Y . Then R and S are interdefinable. Proof Let X ⊆ Rn be definable in S ; we show that X is definable in R by induction on d := dim X . If d = 0, this is trivial, so we assume that d > 0 and the claim holds for lower values of d. Since the diffeomorphisms τm are definable in R, we may assume that X is bounded. By cell decomposition in S and the inductive hypothesis, after permuting coordinates if necessary, we may assume that d (X ) is open and X is the graph of a continuous map f : d (X ) −→ Rn−d . Since bd d (X ) has dimension less than d and d (X ) is open, the inductive hypothesis and o-minimality imply that d (X ) is a finite union of components of Rd \ bd d (X ) and hence definable in R. Let now Y ⊆ Rn be definable in R such that X ⊆ Y and dim Y = d; since d (X ) is definable in R, we may assume that d (Y ) = d (X ). By the properties of dimension, it follows that the set Y  := {z ∈ d (Y ) : the fiber Yz is infinite} has dimension less than d. Hence the set   X  := x ∈ X : d (x) ∈ cl Y  is definable in S and has dimension less than d; so by the inductive hypothesis, X  is definable in R. Thus, after replacing X by X \ X  , we may assume that Yz is finite for all z ∈ Rd as well. By cell decomposition (in R) and the inductive hypothesis, after possibly shrinking X again, we may therefore assume that X is connected and there are continuous maps

Hausdorff limits of Rolle leaves

89

g1 , . . . , g p : d (X ) −→ Rn−d , definable in R, such that Y = gr(g1 ) ∪ · · · ∪ gr(g p ) and the graphs gr(g j ) are pairwise disjoint. Since X ⊆ Y and each gr(g j ) is a component of Y , it follows that X = gr(g j ) for some j, so the claim is proved.

Proof of Theorem B First, we establish [5, Proposition 7.1] with “T ∞ -limit” and “T ∞ (R)” in place of “pfaffian limit” and “P (R)”. To do so, we proceed exactly as in [5], making the following additional changes. (B1) Replacing “admissible sequence” with “T ∞ -sequence”, we obtain corresponding versions of Lemma 4.8, Remark 4.9, Proposition 4.11, Corollary 4.13 and Proposition 5.3 in [5]. (B2) Using (B1), we obtain the corresponding version of [5, Proposition 7.1]. Second, assuming that R admits analytic cell decomposition, statement (B2), Proposition 3.4 and [5, Proposition 10.4] imply that, for every set X definable in T ∞ (R), there exists a set Y definable in N (R) such that X ⊆ Y and dim Y = dim X . It follows from Proposition 4.1 that T ∞ (R) and N (R) are interdefinable. Hence, by [5, Corollary 1], T ∞ (R) and P (R) are interdefinable. Replacing once more R by P (R), Theorem B is proved.



References 1. van den Dries, L.: Limit sets in o-minimal structures. In: Edmundo, M., Richardson, D., Wilkie, A.J. (eds) Proceedings of the RAAG Summer School Lisbon 2003: O-minimal structures, pp. 172–215 (2005) 2. van den Dries, L., Speissegger, P.: real field with convergent generalized power series. Trans. Am. Math. Soc. 350, 4377–4421 (1998) 3. Lion, J.-M., Rolin, J.-P.: Volumes, feuilles de Rolle de feuilletages analytiques et théème de Wilkie. Ann. Fac. Sci. Toulouse Math. 7(6), 93–112 (1998) 4. Lion, J.-M., Speissegger, P.A.: geometric proof of the definability of Hausdorff limits. Selecta Math. (N.S.) 10, 377–390 (2004) 5. Lion, J.-M., Speissegger, P.: The theorem of the complement for nested sub-pfaffian sets. Duke Math. J. 155, 35–90 (2010) 6. Marker, D., Steinhorn C., I.: Definable types in o-minimal theories. J. Symb. Logic 59, 185–198 (1994) 7. Speissegger, P.: The Pfaffian closure of an o-minimal structure. J. Reine Angew. Math. 508, 189–211 (1999) 8. Wilkie, A.J.: A theorem of the complement and some new o-minimal structures. Sel. Math. (N.S.) 5, 397–421 (1999)