Februry 10, 2005. revised version

Research supported in part by National Science Foundation.
<ph f="cmbx">Area-Preserving Surface Diffeomorphisms</ph>

Zhihong Xia

Department of Mathematics, Northwestern University, Evanston, Illinois 60208 E-mail address : xia@math.northwestern.edu

1 Introduction and statement of main results

Let Diff r μ ( M )   be the set of all C r   , r = 1 , 2 , … , ∞   diffeomorphisms of compact orientable surface M   that preserves a smooth area element μ   . A property for C r   area-preserving diffeomorphisms of M   is said to be generic, if there is a residual subset R ⊂ Diff r μ ( M )   such that the property holds for all f ∈ R   . The following are some examples of known generic properties:
  • G1 All the periodic points are either elliptic or hyperbolic. This is proved by Robinson [20] .
  • G2 All elliptic periodic points are Moser stable. Moser stable means that the normal form at the elliptic periodic points are non-degenerate in the sense of KAM theory (cf. Siegel & Moser [22] ). This implies that there are invariant curves surrounding each elliptic periodic point. This condition requires that the map is sufficiently smooth, say r ≥ 16   .
  • G3 For any two hyperbolic periodic points p   , q   , the intersections of the stable manifold W s ( p )   and the unstable manifold W u ( q )   are transversal.
    This is again proved by Robinson [20] . G1 and G3 together are often referred as the Kupka-Smale condition for area-preserving diffeomorphisms.
  • G4 For any hyperbolic periodic point p   , let Γ 1   and Γ 2   be any two branches of the stable manifold or unstable manifold of p   , then Γ 1 ¯ = Γ 2 ¯   . For S 2   , this is proved by Mather [13] . The general case is proved by Oliveira [14] .
  • G5 If M   is a two-sphere S 2   or a two torus T 2   , let Γ 1   be a branch of the stable manifold of a hyperbolic periodic point of p   and Γ 2   be a branch of the unstable manifold of p   , then Γ 1 ∩ Γ 2 ≠ ∅   . For S 2   , this is proved by Robinson [21] and Pixton [16] . For T 2   , it is proved by Oliveira [14] . For general surfaces, it is proved by Oliveira [15] for most homotopy classes of maps. For C 1   case and any compact manifold of arbitrary dimension, the result is proved by Takens [23] (see also Xia [26] for a stronger result).
Franks & Le Calvez [8] recently showed another remarkable C r   generic property for area-preserving diffeomorphisms on S 2   . Their results state that the stable and unstable manifolds of hyperbolic periodic points are dense in S 2   generically. In this paper, we extend this result to arbitrary orientable surfaces and to arbitrary homotopy classes. More precisely, we state our main result.
Theorem 1.1. Let M   be a compact orientable surface. There exists a residual subset R ⊂ Diff r μ ( M )   such that if f ∈ R   and P   is the set of all hyperbolic periodic points of f   , then both the sets ∪ p ∈ P W s ( p )   and ∪ p ∈ P W u ( p )   are dense in M   .
Furthermore, let U ⊂ M   be an open connected subset such that it contains no periodic point for f   and suppose that, for some hyperbolic periodic point p   , U ∩ W s ( p ) ≠ ∅   , then W s ( p )   is dense in U   .
The main tool of our proof is the theory of prime ends [4]  [13] and Lefschetz fixed point theorem. Also, Arnold's conjecture for symplectic fixed points, as proved by Conley & Zehnder [5] for the case of torus, turned out to be essential for our results on T 2   .
Our work was motivated in our effort to prove the so-called C r   closing lemma, as conjectured by Poincaré [17] . Our result provides a strong evidence supporting the C r   closing lemma. We will discuss this and other related problems in the end of the paper. I am grateful to Kamlesh Parwani for useful comments and corrections to the original manuscript.

2 Prime ends and a Theorem of Mather

Let U   be a simply connected domain of S 2   whose complement contains more than one point. One can define the prime end compactification of U   , introduced by Caratheodory [4] , by adding the circle S 1   . Each point in S 1   are defined by a sequence of nested arcs in U   . S 1   inherits a topology from U   . We refer to Mather [13] for definitions and general discussions. Here we only summarize what we will need in this paper. The compactification U ^ = U ⊔ S 1   of U   , with inherited topology on S 1   , is homeomorphic to a closed disk D   . If U ⊂ S 2   is an invariant subset for a homeomorphism on S 2   , then the function f   extends continuously to a homeomorphism f ^ : U ^ → U ^   . Moreover, f ^   restricted to the prime ends S 1   is a circle homeomorphism.
When the rotation number ρ   of f ^ | S 1   is rational, then f ^   has a periodic points on the prime ends. But it may happen in general that f   does not have any periodic points on ∂ U   . However, if f   is area-preserving, then this will never happen.
Lemma 2.1. Let f   be an area-preserving homeomorphism on S 2   and let U   be a simply connected, invariant set whose complement contains more than one point. Let f ^ : U ^ → U ^   be the extension of f   to the prime end compactification of U   . If f ^   has a periodic point in its prime ends then f   has a periodic point in ∂ U   .
The proof is simple, we refer to Mather [13] and Franks & Le Calvez [8] . The next theorem, which is due to Mather [13] states that in generic situations, there are no periodic points on the prime ends.
Theorem 2.2. Let f   be an area-preserving diffeomorphism on S 2   and let U   be a simply connected, invariant set whose complement contains more than one point. Further assume that f   satisfies the generic conditions G1, G2 and G3. Then there is no periodic point for f   on the boundary of U   and as a consequence, there is no periodic point for f ^   on the prime ends and the rotation number of f ^   on the prime ends is irrational.
The proof is based on the following ideas: by generic conditions G1 and G2, f   can not have any elliptic periodic points on ∂ U   . If there is any hyperbolic periodic points on ∂ U   , then the boundary of U   has to contain some branches of the stable and unstable manifolds of the hyperbolic periodic points and this will lead to a contradiction to the generic property G3. For details of the proof, see Mather [13] and Franks & Le Calvez [8] .
The theory of prime ends can be easily generalized to other orientable compact surfaces M   , possibly with boundaries. Let U ⊂ M   be a connected open domain in M   . Assume that the boundary of U   contains finite number of connected pieces and each piece contains more than one point. Then U   can be compactified by adding prime ends. In this case, the prime ends are homeomorphic to a union of finitely many circles S 1   . The number of circles S 1   added is the number of boundary pieces seeing from the inside of U   . It may be more than the number of connected components of ∂ U   . The compactification of U   , denoted as U ^   , is a compact surface with boundaries, each S 1   will be a hole for U ^   . If f : M → M   is a homeomorphism and U   is invariant for f   , then f   extends continuously to a homeomorphism on U ^   .
We denote this extension by f ^   .
Lemma  2.1 and Theorem  2.2 , with obvious modifications, are both true for U ^   . In case where U   has infinitely many connected boundary components, or where some of the boundary components are single points, one can do a prime end compactification to finitely many isolated connected boundary pieces that contain more than one point. Again, the number of boundary pieces are defined to be the number seeing from inside U  

3 Periodic-point-free regions

Let R   be a residual subset of Diff r μ ( M )   such that for any map in R   the generic properties G1, G2, G3 and G4 hold. Whenever G4 is assumed, we require the map to have sufficient smoothness, say r ≥ 16   . Our final result will be true for any r = 1 , 2 , … ,   .
Fix a map f ∈ R   . Let U ⊂ M   be a connected open set such that f   has no periodic point in U   . Our goal is to show that either U   is filled by stable and unstable manifolds of some hyperbolic periodic points, or by an arbitrarily small C r   perturbation of f   , we can create a periodic point in U   .
Let A U   be the set of all points whose orbit under f   passes through U   . i.e., A U = ∪ i = − ∞ ∞ f i ( U ) .   Since f   is area-preserving, almost every point is recurrent. Since M   has finite area, A U   has finite number of connected components. Each of these components are open and periodic. Without loss of generality, we may assume that A U   has only one component and this component certainly contains U   . The case with more than one components can be reduced to this case simply by considering powers of f   .
The set A U   is invariant under f   and it is periodic point free. We are interested in whether the closure of A U   contains any periodic point. The set A U   typically has infinitely many connected boundary components, so we can not apply Lemma  2.1 and Theorem  2.2 , or other results obtained by the prime ends directly. We have the following simple lemma.
Lemma 3.1. The closure of A U   , A U ¯   , is a compact invariant set which contains no elliptic periodic point for f   . Here we assume f   has sufficient smoothness so that the generic condition G2 holds.
Proof: Recall that the map f ∈ R   satisfies the generic property G2, thus all elliptic periodic points are Moser stable. This implies that surrounding each elliptic periodic point there are infinitely many invariant curves. If A U ¯   contains an elliptic periodic point, then A U   has to intersect an invariant curve with irrational rotation number close to the elliptic point. The set A U   is invariant, thus it completely covers that curve. By Birkhoff fixed point theorem, or more generally the Aubry-Mather theory, any neighborhood of the invariant curve contains infinitely many periodic orbits. This contradicts to the assumption that A U   is periodic point free.
This proves the lemma.
The closure of A U   may still contain hyperbolic periodic points. We show that this happens only if A U   contains stable or unstable manifolds of hyperbolic periodic points.
Lemma 3.2. If the closure of A U   , A U ¯   , contains a hyperbolic periodic point p   , then U ∩ W s ( p ) ≠ ∅   and U ∩ W u ( p ) ≠ ∅   .
Proof: Recall that for f ∈ R   , generic property G4 implies that W s ( p ) ¯ = W u ( p ) ¯   , it suffices to show that either A U ∩ W s ( p ) ≠ ∅   or A U ∩ W u ( p ) ≠ ∅   .
First, we remark that if M = S 2   or M = T 2   and generic property G5 is assumed, then the lemma can be easily. In this case, each branch of the stable and unstable manifolds of p   intersect and the intersection is transversally, there is a homoclinic tangle in the neighborhood of p   and this homoclinic tangle divides a small neighborhood of p   into infinitely many small rectangles. Choose a small neighborhood p   such that all of these rectangles have areas smaller than that of U   .
If p   is in the closure of A U   , then, for some integer i   , f i ( U )   intersects one of these small rectangles. By the area preserving property of f   , f i ( U )   can not be totally contained in any single rectangle and thus it has to intersect the boundary of the rectangle. This implies that f i ( U )   contains a point on either the stable manifold or the unstable manifold of p   .
For the general case, suppose that there is a hyperbolic periodic point p ∈ M   such that A U ∩ W s ( p ) = ∅   , we want to show that p   is not in the closure of A U   . Let V   be the connected component of M \ W s ( p ) ¯   containing A U   . Since the complement of V   is connected, the boundary of V   consists of finitely many connected pieces and each piece contains more than one point. Therefore V   has a prime end extension, V ^   with the extended map f ^   on V ^   . By Mather's theorem [13] , which is proved in this general setting, V ^   has no periodic points on the boundary of V ^   . This implies that p / ∈ V ¯   and since A U ⊂ V   , this implies that the closure of A U   contains no periodic point.
This proves the lemma.
To get rid of isolated points in the boundary of A U   , we let A = int ( A ¯ U )   . Then any isolated connected boundary piece contains more than one point. If the closure of A U   is periodic point free, then so is the closure of A   .

4 Isotopic to identity cases

In this section we prove our main lemma in the case where the map is isotopic to identity.
Lemma 4.1. Let f ∈ R   be a generic diffeomorphism of M   and f   is isotopic to identity. Let A ⊂ M   be an open, connected, periodic point free, f   -invariant set whose closure contains no periodic point. We further assume that A = int A ¯   . Then either A = M = T 2   or A   is homeomorphic to an open annulus and its prime end extension is a closed annulus.
Proof: We first assume that M ≠ T 2   . Since the Euler characteristic of M   is non-zero, by Lefschetz fixed point theorem, f   contains at least one fixed point.
This implies that A ≠ M   . Let Fix ( f )   be the set of all fixed point for f   . It contains finite number of points and it is contained in the interior of M \ A   . Let B ⊂ M   be the union of the connected components of M \ A   intersecting Fix(f )   . The set B   is closed and has finite number of connected components and each contains at least one fixed point of f   . Let C = M \ B   , obviously C   is open, A ⊂ C   and the closure of C   contains no fixed point. Let C ^   be the prime end extension of C   and let f ^ : C ^ → C ^   be the extension of f   . Then f ^   has no fixed point. The set C ^   is a compact surface with boundary, its topology is uniquely determined by the number of handles and the number of holes on S 2   . Let the number of handles of C ^   be k   and the number of holes be l   . The map f ^   keeps invariant of all the boundary pieces. The Euler characteristic number of C ^   is 2 − 2 k − l   . Even though f ^   may not neccessarily be homotopic to identity, but its induced map on homology is identity.
Hence its Lefschetz number is the same as its Euler characteristic number. Since f ^   has no fixed point, this implies that 2 − 2 k − l = 0   . This implies that either k = 0   , l = 2   or k = 1   , l = 0   . The latter implies that C   is a torus and thus M   is a torus, which we assumed was not the case. The first case implies that C ^   is a closed annulus.
We claim that A = C   . If not, there is a point x ∈ C \ A ¯   . i.e., x   is in the interior of C \ A   . For if no such point exists, then C ⊂ A ¯   . But C   is open, this implies that C ⊂ int A ¯ = A   , by the assumption on A   . Let B x   be the connected component in C \ A ¯   containing x   . Then B x   is closed and B x ∩ ∂ C = ∅   . For if B x ∩ ∂ C ≠ ∅   , then B x ⊂ B   , where B   is the set of connected components containing fixed points in the complement of A   , as defined previously. Since B x   has positive area, it must be periodic under f   . Let k   be the period of B x   . Let Fix ( f k )   be the set of all periodic points of f   with period k   . Again, Fix ( f k )   contains only finite number of points. Let B k   be the union of all connected components of C \ A   . The set B k   has finite number of, say l   , connected components and all components are periodic with period k   . Moreover, B k ∩ ∂ C = ∅   . Let C x = C \ B k   , then C x   is an open set and C x = int C ¯ x   . The prime end extension of C x   is a closed annulus with l   interior disks removed. and there is no periodic points of period k   in C x   . However, the Lefschetz number for f k   on C x   is l ≠ 0   , this implies that f k   has to have at least one fixed point in C x   . This contradiction proves that A = C   . i.e., A   is an annulus.
If M = T 2   , then either A = T 2   or there is a point x ∈ int ( M \ A )   . Let B ′   be the connected components containing x   in the complement of A   . Again, B ′   must be periodic and let this period be k   . Let B   be the union of all the connected components of M \ A   that are fixed under f k   . ∅ ≠ B ′ ⊂ B   . Let C = A \ B   . Then this case follows from above arguments by considering f k   on C   .
This proves our lemma.

5 A lemma on periodic points

Now we consider the case where f   is not necessarily isotopic to identity. To prove the same result, we need to prove a simple theorem on the existence of periodic points for maps of compact surfaces with boundaries.
Lemma 5.1. Let M   be a compact, connected surface, possibly with boundary. Assume that M   has nonzero Euler characteristic. Suppose that f : M → M   is a homeomorphism, then f   has a periodic point.
Moreover, for any positive integer n   , there exists infinitely many positive integers i   such that the Lefschetz number L ( f n i )   is negative.
Proof: Let L ( f )   be the Lefschetz number of the map f   . By Lefschetz fixed point theorem, it suffices to show that L ( f n )   is nonzero for some nonzero integer n   .
We consider homology groups of M   with real coefficients. If H 2 ( M ) ≠ 0   , by taking an interate of f   , we may assume that the induced map on H 2 ( M )   is the identity. Let t r 1 ( f * )   be the trace of the induced map f * : H 1 ( M ) → H 1 ( M )   , then L ( f n ) = 2 − t r 1 ( f * n )   if M   is orientable and without boundary, and L ( f n ) = 1 − t r 1 ( f * n )   if M   is non-orientable or has boundary. The theorem follows easily for the cases where the dimension of H 1 ( M )   is zero (the sphere and the projective plane). The torus, the Klein bottle, the Möbius strip and the annulus all have Euler characteristic zero and hence are excluded. The only cases we need to consider are where the dimension of the first homology H ( M )   are greater than or equal to 3.
Let λ 1 , λ 2 , … , λ l   be the eigenvalues of the induced isomorphism f * : H 1 ( M ) → H 1 ( M )   , where l   is the dimension of H 1 ( M )   . We can write λ i = r i α i   , where r i   is a positive real number and α i   is complex number on the unit circle, for i = 1 , 2 , … , l   .
There exists a sequence of integers { n k } k = 1 ∞   such that α i n k → 1   as k → ∞   , for all i = 1 , 2 , … , l   .
If r i > 1   for some i = 1 , 2 , … , l   , then t r 1 ( f * n k ) = ∑ i = 1 l λ i n k → ∞   as k → ∞   .
This implies that for large k   , L ( f n k ) ≠ 0   , the lemma follows. If r i < 1   for some i = 1 , 2 , … , l   , then we consider f − 1   , the inverse of f   . The same argument shows that L ( f n ) ≠ 0   for some negative integer n   . Since L ( f n ) = L ( f − n )   , the lemma again follows. Finally, if r i = 1   for all i = 1 , 2 , … , l   , then t r 1 ( f * n k ) = ∑ i = 1 l λ i n k → l   as k → ∞   . As j ≥ 3   , 2 − t r 1 ( f * n k ) ≠ 0   and 1 − t r 1 ( f * n k ) ≠ 0   for large k   . This shows that for infinitely many positive integer k   , the Lefschetz number L ( f n k )   is negative. Lefschetz fixed point theorem concludes that there are at least one fixed point for f n k   for such k   .
For any positive integer n   , replacing f   with f n   , the above arguments show that there are infinitely many positve integer i   such that L ( f n i )   is negative.
This proves the lemma.
For orientable compact surfaces, only T 2   and annulus have zero Euler characteristics.
Lemma  4.1 would follow from the above lemma if the periodic point free set A   in Lemma  4.1 has a prime end extension that makes it into a compact manifold with boundary. We have to show that A   can't have infinitely many boundary pieces.
The proof of the lemma is basically a verification of this fact.

6 General case

Let f ∈ R   be a generic diffeomorphism on the compact surface M   . Let A ⊂ M   be an open, connected, periodic point free, f   -invariant set whose closure contains no periodic point. Assume A = int ( A ¯ )   . For any integer i   , we let Fix ( f i )   be the set of all periodic points of f   with period i   . If A ≠ M   , M \ A   is nonempty and contains at least one connected component. The number of the connected components in M \ A   may be infinite. Let B i   be the set of all connected components of M \ A   containing a periodic point of period i   . B i   is closed and B i ∩ Fix ( f i ) ≠ ∅   , if Fix ( f i ) ≠ ∅   .
B i   has finite number of connected components. For any positive integers i , j   , each component of B j   is either a component of B i   or disjoint from B i   . If j = k i   for some positive integer k   , then B i ⊂ B j   .
Let C i = M \ B i   . Then C i   has finite number of boundary components. Let C ^ i   be the prime end extension of C i   and it is a compact surface with boundary. Let the number of handles of C ^ i   be m   and the number of holes be n   . For any integer k   , the number of handles of C ^ k i   is smaller than or equal to m   . Therefore, there exists an integer i *   such that the number of handles of C ^ k i *   is a constant for all positive integer k   .
If Fix ( f i )   is empty for all positive integer i   , then either A = M   , then we set C i = M   for all i   , or A ≠ M   , then we pick a point x ∈ M \ A   and let B i   be the connected component of M \ A   and its iterates under f   . Since f   is area preserving, B i   has finite number of components.
By our construction, f ^ i : C ^ i → C ^ i   has no fixed points for all i ≥ 1   .
We are ready to prove Lemma  4.1 for the general case.
Lemma 6.1. Let f ∈ R   be a generic diffeomorphism of M   . Let A ⊂ M   be an open, connected, periodic point free, f   -invariant set whose closure contains no periodic point. We further assume that A = int A ¯   . Then either A = M = T 2   or A   is homeomorphic to an open annulus and its prime end extension is a closed annulus.
Proof: First if there exists a positive integer k   such that f k   is isotopic to identity, the lemma is proved in the same way as Lemma  4.1 . One just need to consider f k   .
We first suppose that M ≠ T 2   .
As described before, there exists an integer i *   such that the number of handles of C ^ k i *   is a constant for all positive integer k   . We now consider f ^ i * : C ^ i * → C ^ i *   . It is a homeomorphism on compact surface, keeping invariant of all boundary pieces.
Suppose f ^ i * : C ^ i * → C ^ i *   has no periodic points, then by Lemma  5.1 , C i *   is an open annulus and C ^ i *   is a closed annulus. We claim that A = C i *   . Suppose not, then there is a point x ∈ C i *   , but x / ∈ A   . Since A = int A ¯   and by the definition of C i *   , x / ∈ ∂ A   . Let C x   be the connected component of C i * \ A ¯   containing x   . Then C x   an open disk in a annulus. Since f   is area preserving, C x   must be periodic and it contains a periodic point. This contradicts to the assumption that C i *   has no periodic point.
Now suppose that f ^ i * : C ^ i * → C ^ i *   has a periodic point x   with period p   . By the assumption on A   , x / ∈ A ¯   . By Lemma  5.1 , there is a positive integer j   such that the Lefschetz number of the p j   's iterate of f ^ i *   on C i *   is negative. i.e., L ( ( f ^ i * ) p j | C i * ) < 0   , where p   is the period of x   . Now consider the prime end extension of C i * p j   . By our choice of i *   , C ^ i * p j   is topologically C ^ i *   with finitely many, say k   , k > 0   , open disks removed and these disks are periodic with period p j   under the extended map f ^ i *   . We have the following relations on the Lefschetz numbers L ( f ^ i * p j | C i * p j ) = L ( ( f ^ i * ) p j | C i * ) − k < 0   . Lefschetz fixed point theorem implies that f ^ i * p j   has a fixed point on C i * p j   , which is impossible by the definition of C i * p j   . This contradiction shows that f ^ i * : C ^ i * → C ^ i *   has no periodic point and hence A = C ^ i *   is an annulus.
We are left with one case: M = T 2   and A ≠ M   . The same argument works in this case too.
This proves the lemma

7 Maps on annulus

Let U   be a connected open subset of M   and let A U = ∪ i = − ∞ ∞ f i ( U )   . If the closure of A U   contains no periodic point, then A = int ( A ¯ U )   is open, containing no periodic points in its closure. By the above lemma, if M ≠ T 2   , then A   is a union of finite disjoint open annuli, periodic under f   . The dynamics on Annulus have been well studied (cf. Franks [6] , Le Calvez & Yoccoz [3] The following lemma shows that, if A   is an annulus, we can perturb f   with an arbitrarily small C r   perturbation to create a periodic point in U   . The same result was also used in [8] 
Lemma 7.1. Fix f ∈ R   and assume M ≠ T 2   . Let U   be a connected open subset of S 2   and A U = ∪ i = − ∞ ∞ f i ( U )   . Assume that U   does not intersect any stable or unstable manifolds of hyperbolic periodic points of f   . Then for any C r   neighborhood V   of f   , there exists g ∈ V   such that the support of g − f   is contained in the interior of the closure of A U   and g   has a periodic point in U   .
Proof: Since U   does not intersect any stable or unstable manifolds of hyperbolic periodic points of f   , By above lemmas, f   has no periodic point in A U   and no periodic point in the closure of A U   and therefore, A = int ( A ¯ U )   is a union of finite disjoint periodic annuli. Without loss of generality, we may assume that A U   itself is an open annulus.
Using prime end extension, we obtain an area-preserving continuous map on the prime end closure of A U   , still denoted by A U ¯   . Let ( x , y ) , x ∈ R ( m o d 1 ) , y ∈ [ 0 , 1 ]   be a coordinate on A U ¯   . Since f   preserves invariant measure μ   , by Birkhoff Ergodic Theorem, for μ   -almost every point z = ( x , y ) ∈ A U ¯   , the rotation number ρ ( z , f ) = lim i → ∞ π x f ~ i ( z ) i   is well defined. Here π x   is the projection on A U ¯   into its first coordinate and f ~   is a lift of f   to its universal cover R × [ 0 , 1 ]   . A different lift of f   yields a different rotation number that differs by an integer.
Since there is no periodic points in A U ¯   , by Franks theorem [6] , there exists an irrational number α ∈ R   such that for almost all z ∈ A U   , the rotation number ρ ( z , f )   exist and ρ ( z , f ) = α   . In particular, if z   is in the boundaries of A U   then ρ ( z , f ) = α   .
Let γ   be a simple closed curve in the interior of A U   . We may assume that γ   is homotopically non-trivial in A U   . Take a small tubular neighborhood γ δ   of γ   in the interior of A U   and parametrize this tubular neighborhood by γ δ : S 1 × [ − δ , δ ] → : A U   for some small δ   . In fact, for convenience we may even assume that γ δ   is area-preserving. Let β : [ − δ , δ ] → R   be a C ∞   function such that β ( t ) > 0   for all − δ < t < δ   and β ( − δ ) = β ( δ ) = 0   and β   is C ∞   flat at ± δ   . i.e., all the derivatives of β ( t )   at ± δ   are zero.
Let h ε : A U → A U   be a C ∞   diffeomorphism such that if z / ∈ γ δ   , h ε ( z ) = z   and if z ∈ γ δ   , h ε = γ δ ∘ T ε ∘ ( γ δ ) − 1   where T ε ( θ , t ) = ( θ + ε β ( t ) , t )   for all θ ∈ S 1   and t ∈ [ − δ , δ ]   . We remark that h ε → Id   in C ∞   topology as ε → 0   and the mean rotation number for h ε   with respect to the area μ   is ρ μ ( A U , h ε ) = 1 μ ( A U ) ∫ − δ δ ε β ( t ) d t   Therefore for any ε > 0   , ρ μ ( A U , h ε ∘ f ) = ρ μ ( A U , h ε ) + ρ μ ( A U , f ) > α   , this implies that there exists a point y * ∈ A U   , such that ρ ( y * , h ε ∘ f ) > α   . Since ρ ( z , h ε ∘ f ) = ρ ( z , f ) = α   for all z ∈ ∂ A U ¯   , we conclude, from Franks' theorem [6] , that for any rational number p / q   , such that α < p / q < ρ ( y * , h ε ∘ f )   , there exists a periodic point of period p / q   for the map h ε ∘ f   .
Thus, for any ε > 0   , there are infinitely many periodic points for h ε ∘ f   in the interior of A U   . In fact, all of these periodic points have to pass through the strip γ δ   . However, these periodic points may be far away from U   . To find periodic points in U   , we need to do some estimates on these orbits.
Since A U = ∪ i ∈ Z f i ( U )   , for any point z ∈ γ δ ⊂ A U   , there exists an integer n z ∈ Z   and a neighborhood of z   , W z ⊂ A U   , such that f n z ( W z ) ⊂ U   . { W z , z ∈ γ δ }   forms an open cover for the compact set γ δ   . Let W z 1 , W z 2 , … , W z k   be a finite subcover of γ δ   and let N = max { | n z 1 | , | n z 2 | , … , | n z k | }   .
The integer N   chosen above has a very important property: for any z ∈ γ δ   , the orbit segment { f − N ( z ) , f − N + 1 ( z ) , … , f N ( z ) }   intersects U   at least once. Or equivalently, the set ∪ i = − N N f i ( U )   covers γ δ   . Since U   is open, this same property holds for all g   sufficiently close to f   in C 0   topology. i.e., the orbit segment { g − N ( z ) , g − N + 1 ( z ) , … , g N ( z ) }   intersects U   for all z ∈ γ δ   , provided that g   is sufficiently close to f   .
Above arguments show that if ε > 0   is small enough, h ε ∘ f   has infinitely many periodic orbits and all of these periodic orbits intersect U   .
This proves the lemma.

8 Maps on torus and Arnold's conjecture

The final case is where M = T 2   and f   has no periodic point. We will show that such f   is not generic and it can be perturbed to create a periodic point.
Let f * 1 : H 1 ( T 2 , R ) = R 2 → R 2   be the induced map on the first homology of T 2   . Let λ 1   , λ 2   be the eigenvalues of f * 1   , λ 1 = λ ¯ 2   . Since f   has no periodic point, the Lefschetz number L ( f k ) = 0   for all k   . This implies that λ 1 k + λ 2 k = 2   , for all k   .
We must have λ 1 = λ 2 = 1   . Since f * 1   and its inverse are both integer matrices, we have only two choices: f   is isotopic to identity, where f * 1 = I   or f   is isotopic to a Dehn twist, i.e., for some integer k ≠ 0   , f * 1 = ( 1 k 0 1 )   We first consider the case where f   is isotopic to identity. The proof is basically an application of the Arnold conjecture [1] as proved by Conley and Zehnder [5] .
Lemma 8.1. Let f : T 2 → T 2   be an area-preserving diffeomorphism such that f   is isotopic to identity. Then for any C r   neighborhood V   of f   , there exists g ∈ V   such that g : T 2 → T 2   has a periodic point.
Proof:
Let f ~ : R 2 → R 2   be a lift of the map f   on T 2 = R 2 / Z 2   to its universal cover R 2   . Let π i   , i = 1 , 2   be the projection of R 2   to its first and second coordinates respectively. The average rotation numbers for the map f ~   , are defined to be ρ i ( f ~ ) = ∫ ( π i ( f ~ ( p ) ) − π i ( p ) ) d μ .   The combination ρ ( f ) = ( ρ 1 ( f ~ ) , ρ 2 ( f ~ ) ) , m o d Z 2   is called the average (or mean) rotation vector for f   . The rotation vectors are well defined for maps isotopic to identity.
We want to do a small perturbation to f   so that each component of the average rotation vector is a rational number. This is easy: one composes the map f   with T ( ε 1 , ε 2 ) ( x , y ) = ( x 1 + ε 1 , x 2 + ε 2 )   , then ρ ( f ⋅ T ( ε 1 , ε 2 ) ) = ρ ( f ) + ( ε 1 , ε 2 ) , m o d Z 2   . By properly choosing small ε 1   and ε 2   , we obtain a rational rotation vector for f ⋅ T ( ε 1 , ε 2 )   .
There is a positive number i   such that the mean rotation vector for ( f ⋅ T ( ε 1 , ε 2 ) ) i k   is an integer vector, which is equivalent to zero on the torus. ( f ⋅ T ( ε 1 , ε 2 ) ) i k   is isotopic to identity and preserves a smooth area element. The Arnold conjecture, as proved by Conley and Zehnder [5] in the case of torus, implies that ( f ⋅ T ( ε 1 , ε 2 ) ) i k   has at least three fixed points, four if all non-degenerate. This implies that f ⋅ T ( ε 1 , ε 2 )   has periodic points of period k i   .
This proves the lemma.
We use Poincaré-Birkhoff Theorem for the case where f   is isotopic to a Dehn twist.
Lemma 8.2. Let f : T 2 → T 2   be an area-preserving diffeomorphism such that f   is isotopic to a Dehn twist. Then for any C r   neighborhood V   of f   , there exists g ∈ V   such that g : T 2 → T 2   has infinitely many periodic points.
Let ( ( x , y ) , m o d Z 2 )   be a coordinate systems on T 2   such that f   is isotopic to the map ( x , y ) ↦ ( ( x + k y , y ) , m o d Z 2 )   for some non-zero integer k   . We may assume, without loss of generality, that k > 0   . In this homotopy class, there are maps without any periodic point. For example, the map ( x , y ) ↦ ( x + k y , y + α )   has no periodic point if α   is irrational. However, there are periodic points when α   is rational. Our first step is to perturb the map so that it has a rational vertical rotation number.
Lift the map f   in the y   direction, we obtain a map on the infinite cylinder f ~ y : S 1 × R → S 1 × R   . Define the mean vertical rotation number ρ 2 ( f ~ y ) = ∫ ( π 2 ( f ~ y ( p ) ) − π 2 ( p ) ) d μ ,   where μ   is the area element. π 2 ( f ~ y ( p ) ) − π 2 ( p )   is independent of choices of the covering points. We define the mean vertical rotation number of f   to be ρ 2 ( f ) = ρ 2 ( f ~ y ) , m o d 1   . This is independent of the lift.
By composing f   with the map ( x , y ) ↦ ( x , y + ε )   for some small ε   , we obtain a map g   on the torus such that its mean vertical rotation number is rational. This implies that there is a positive integer l   such that the vertical rotation number of g l   is zero.
Now, choose a lift of g l   , G y   , in the y   direction such that its mean vertical rotation number is zero (instead of being a non-zero integer). Then G y   is an area preserving map on the infinite cylynder S 1 × R   which is also exact, i.e., integral of the 1-form y d x   over any closed curve on the cylinder is invariant under the map. Moreover, if we let G   be the lift of G y   to R 2   , we have that π 1 ( G ( p ) ) − x − k l y   is uniformly bounded for any p = ( x , y ) ∈ R 2   .
Poincaré-Birkhoff twist map theorem implies that there are at least two fixed points and infinitely many periodic points for G y   . This implies that there are infinitely many periodic points for g   on T 2   .
This proves the lemma.
The Poincaré-Birkhoff twist map theorem (or Poincaré's last geometric theorem) shows the existence of fixed points for area-preserving maps on the annulus with twist condition. Here we have an infinite cylinder with infinite twists on two ends.
There are two ways to work around this. One is to directly apply Birkhoff 's proof [2] to the annulus { | y | ≤ M }   with large M > 0   . The annulus is not invariant but the proof works in the same way, as long as one has exactness in the area preserving property. Another way to prove the result is to modify the map so that all horizontal lines are fixed for large values of | y |   . Again, the exactness is neccessary. Since these techniques are well known, we will not give details here.

9 Proof of the main theorem

Let M   be a compact surface and let R ⊂ Diff r μ ( M )   be the set of area-preserving diffeomorphisms on M   satisfying G1-G4. We first assume that r ≥ 16   . In addition, we assume that R   satisfies the following two conditions: for any f ∈ R   ,
  • G6 every invariant open annulus contains a periodic point;
  • G7 if M = T 2   , then there is a periodic point.
By Lemma  8.1 , G7 is an open and dense condition, hence generic. By Lemma  7.1 , G6 is a generic condition. Therefore, R   is a residual set. We claim that for any f ∈ R   , the stable and unstable manifolds of hyperbolic periodic points are dense in M   . Suppose this is not true and there exists an open set U ⊂ M   such that U   does not intersect stable manifold and unstable manifold of any hyperbolic periodic point. Then neither does the invariant set A U = ∪ i ∈ Z f i ( U )   . Then by Lemma  3.1 and Lemma  3.2 , the closure of A U   contains no periodic points. Let A = int ( A U ¯ )   , then the closure of A   contains no periodic point. Lemma  6.1 shows that A   must be an open annulus or M   must be a torus. This contradicts to conditions G6 and G7. This proves the first part of our theorem for r ≥ 16   . For lower smoothness, i.e., for r = 1 , 2 , … , 15   , we first note that the residual set R ⊂ Diff r μ ( M )   , r ≥ 16   constructed above is dense in Diff r μ ( M )   with r = 1 , 2 , … , 15   . Moreover, for each open set U ⊂ M   , if U   intersects a piece of stable (or unstable) manifold of a hyperbolic fixed point for some f ∈ Diff r μ ( M )   , then same is true for any map close to f   . Since M   has a countable basis of open sets, there is a residual subset R r ⊂ Diff r μ ( M )   for all r = 1 , 2 , … ,   such that every open set intersects a piece of stable (and unstable) manifold of some hyperbolic periodic point.
This proves the first part of our main theorem. For the second part of the theorem, we need the following lemma.
Lemma 9.1. Every prime end is an accumulation point of periodic points for a generic C r   surface diffeomorphism. More precisely, there is a residual subset R ⊂ Diff r μ ( M )   (This set can be chosen to be the same as above) such that for any f ∈ R   , we have the following property: Let V   be an open, f   invariant set V   with finite number of connected boundary pieces and each boundary piece containing more than one point, let V ^   be the prime end extension of U   , let z ∈ V ^   be a prime end, then there is a sequence of periodic points of f   , { p n } i = 1 ∞   such that p n → z   as n → ∞   .
We will not give a detailed proof of this lemma. The proof follows from Corollary 8.9 in Franks & Le Calvez [8] , which uses Conley index in a small neigborhood of the prime end circle to obtain periodic points (cf. Franks [7] and Le Calvez & Yoccoz [3] ).
We remark that even though their results are on S 2   , there is no difference nearby one piece of prime ends. We also remark that each prime end circle has irrational rotation, if a sequence of periodic points approaches the prme end circle, then the sequence of periodic orbits approaches every point in the prime end circle.
Now let f ∈ R   and U   be an open connected that contains no periodic point for f   . Assume that W s ( p ) ∩ U ≠ ∅   for some hyperbolic periodic point p   . Suppose that U   is not contained in the closure of W s ( p )   , we will derive a contradiction. Let V   be a connected component of M \ W ¯ s ( p )   whose intersection with U   is non-empty.
Such V   exists by our assumption. Let V ^   be the prime end extension of V   , then U   , as a subset of V ^   , is an open neighborhood of an arc in the prime ends. Since the rotation number on the prime end circle is irrational, this implies, by the the above lemma, U   contains infinitely many priodic points. But U   is periodic point free, by our assumption, this contraction shows that U ⊂ W ¯ s ( p )   .
This proves our main theorem.

10 Other problems and conjectures of Poincare

Poincaré already noted the importance of the generic properties area-preserving diffeomorphisms in his study of the three-body problem. The following two fundamental conjectures are due to Poincaré [17] .
Conjecture 1. For generic C r   area-preserving diffeomorphisms on compact surface M   , the set of all periodic points are dense.
Conjecture 2. There exist a residual set R ∈ Diff r μ ( M )   such that if f ∈ R   and p   is a hyperbolic periodic point of f   then the homoclinic points of p   is dense in both stable and unstable manifolds of p   . In other words, let J   be a segment in W s ( p )   (or W u ( p )   ), then W s ( p ) ∩ W u ( p ) ∩ J ¯ = J   .
In C 1   topology, both of the above conjectures are proved to be true. The first one is a consequence of the so-called closing lemma. It is proved by Pugh [18] and later improved to various cases by Pugh & Robinson [19] . A different proof was given by Liao [11] and Mai [12] . The second conjecture in C 1   topology is a result of Takens [23] . The high dimensional analog was proved by Xia [26] . It can also be regarded as a so-called C 1   connection lemma, first proved by Hayashi [10] and later simplified and generalized by Xia [26] , Wen & Xia [24]  [25] .
In C r   topology with r > 1   , little progress has been made for these two conjectures and it's known to be an extremely difficult problem (cf Smale). The local perturbation methods used in the C 1   case no longer seem to work and examples suggests that a more global approach has to be developed (Gutierrez [9] ).
While unable to prove these two conjectures, our main result offers a very strong evidence supporting the conjectures. Moreover, as an easy consequence, our result implies that the conjecture two implies conjecture one.
References

  1. V. I. Arnold. Mathematical Methods of Classical Mechanics. Springer-Verlag, 1978.
  2. G. D. Birkhoff. Dynamical Systems, volume 9. American Math. Soc. Colloquium Publications, 1966.
  3. P. Le Calvez and J.C. Yoccoz. Un théorm̀e d'indice pour les homómorphismes du plan au voisinage d'un point fixe. Ann. of Math. (2), 146(2):241–293, 1997.
  4. C. Caratheodory. uber die begrenzung einfach zusammenhangender gebiete. Math. Ann., 73:323–370, 1913.
  5. C. Conley and E. Zehnder. The birkhoff-lewis fixed point theorem and a conjecture of v. i. arnold. Invent. Math., 73(1):33–49, 1983.
  6. J. Franks. Rotation vectors and fixed points of area preserving surface diffeomorphisms. Trans. Amer. Math. Soc., 348(7):2637–2662, 1996.
  7. J. Franks. The conley index and non-existence of minimal homeomorphisms. Illinois Journal of Math., 43:457–464, 1999.
  8. J. Franks and P. Le Calvez. Regions of instability for non-twist maps. Ergodic Theory Dynam. Systems, 23(1):111–141, 2003.
  9. C. Gutierrez. A counter-example to a c 2   closing lemma. Ergodic Theory & Dynamical Systems, 7(4):509–530, 1987.
  10. S. Hayashi. Connecting invariant manifolds and the solution of the c 1   stability and ω   -stability conjectures for flows. Ann. of Math., 145(1):81–137, 1997.
  11. S.T. Liao. An extension of the c 1   closing lemma. Acta Sci. Natur. Univ. Pekinensis, 2:1–41, 1979.
  12. J. Mai. A simpler proof of c 1   closing lemma. Scientia Sinica, 10:1021–1031, 1986.
  13. J. Mather. Topological proofs of some purely topological consequences of carathéodory's theory of prime ends. in Selected Studies. Eds. Th. M. Rassias and G. M. Rassias, pages 225–255, 1982.
  14. F. Oliveira. On the generic existence of homoclinic points. Ergod. Th. & Dynam. Sys., 7:567–595, 1987.
  15. F. Oliveira. On c ∞   genericity of homoclinic orbits. Nonlinearity, 13:653–662, 2000.
  16. D. Pixton. Planar homoclinic points. J. Diff. Equations, 44:1365–382, 1982.
  17. H. Poincaré. Les méthodes nouvelles de la mécanique céleste. Paris, 1892.
  18. C. Pugh. The closing lemma. Amer. J. Math., 89:956–1021, 1967.
  19. C. Pugh and C. Robinson. The c 1   closing lemma, including hamiltonians. Ergod. Th. & Dynam. Sys., 3:261–313, 1983.
  20. C. Robinson. Generic properties of conservative systems, i, ii. Amer. J. of Math, 92:562–603, 897–906, 1970.
  21. C. Robinson. Closing stable and unstable manifolds on the two-sphere. Proc. Amer. Math. Soc., 41:299–303, 1973.
  22. C.L. Siegel and J.K. Moser. Lectures on Celestial Mechanics. Springer, 1971.
  23. F. Takens. Homoclinic points in conservative systems. Invent. Math., 18:267–292, 1972.
  24. L. Wen and Z. Xia. A basic c 1   perturbation theorem. J. Differential Equations, 154(2):267–283, 1999.
  25. L. Wen and Z. Xia. On c 1   connecting lemmas. Trans. Amer. Math. Soc., 352(10), 2000.
  26. Z. Xia. Homoclinic points in symplectic and volume-preserving diffeomorphism. Commun. Math. Phys., 177:435–449, 1996.

Department of Mathematics, Northwestern University, Evanston, Illinois 60208 E-mail address : xia@math.northwestern.edu