The invariant subspace problem and Rosenblum operators II

Abstract

Let $\mathcal{H}$ be the separable, infinite-dimensional complex Hilbert space. In the first paper of this series, we introduced, by means of the Rosenblum operators, the shift representation operators $K_x=\sum_{n=0}^{\infty}T^nx\otimes e_n$, which relate an operator $T$ with $r(T)<1$ to the unilateral shift $S$ and reveal its hidden analytic structure. In this paper we replace the pair $(H^2,S)$ by the Sobolev disk algebra $R(\mathbb{D})$ and the multiplication operator $M_z$, whose canonical left inverse $B$ plays the role of $S^*$, and study the two-parameter family $K_{x,y}=\sum_{n=0}^{\infty}T^nx\otimes B^{*n}y$. We prove that $T\in B(R(\mathbb{D}))$ with $r(T)<1$ is intransitive whenever there exist a nonzero vector $g$ and a nontrivial $M_z$-invariant manifold $\mathcal{M}$ such that $f(T)g$ has a zero in the closed unit disk for every $f\in\mathcal{M}$; the proof explicitly constructs a nontrivial invariant subspace of $T$. We further introduce the ideal property and show that every intransitive operator on $\mathcal{H}$ is unitarily equivalent to an operator on $R(\mathbb{D})$ with this property, so that the Invariant Subspace Problem for arbitrary operators reduces to a problem about a concrete function algebra. Applications include a characterization of intransitivity through the rationality of the generating function $\sum_{n=0}^{\infty}\langle T^n\xi,\eta\rangle z^n$ and new results on Pearcy's problem. Our main application is Halmos's third problem, which has remained open for more than fifty years: we reduce it to invertible operators with conjugate bi-geometric form, and we prove that $T^{-1}$ is intransitive for every invertible $T$ admitting a nontrivial projection $P$ with $PTP=TP$ and $PT(I-P)$ of rank one. Finally, we exhibit tridiagonal operators that are intransitive and admit infinite decreasing chains of invariant subspaces.