Limiting spectral distribution of XX′ matrices

The methods to establish the limiting spectral distribution (LSD) of large dimensional random matrices includes the well-known moment method which invokes the trace formula. Its success has been demonstrated in several types of matrices such as the Wigner matrix and the sample covariance matrix. In a recent article Bryc, Dembo and Jiang [Ann. Probab. 34 (2006) 1-38] establish the LSD for random Toeplitz and Hankel matrices using the moment method. They perform the necessary counting of terms in the trace by splitting the relevant sets into equivalence classes and relating the limits of the counts to certain volume calculations. Bose and Sen [Electron. J. Probab. 13 (2008) 588-628] have developed this method further and have provided a general framework which deals with symmetric matrices with entries coming from an independent sequence. In this article we enlarge the scope of the above approach to consider matrices of the form A_p = 1/n XX′ where X is a p × n matrix with real entries. We establish some general results on the existence of the spectral distribution of such matrices, appropriately centered and scaled, when p → ∞ and n = n(p) → ∞ and p/n → y with 0 ≤ y < ∞. As examples we show the existence of the spectral distribution when X is taken to be the appropriate asymmetric Hankel, Toeplitz, circulant and reverse circulant matrices. In particular, when y = 0, the limits for all these matrices coincide and is the same as the limit for the symmetric Toeplitz derived in Bryc, Dembo and Jiang [Ann. Probab. 34 (2006) 1-38]. In other cases, we obtain new limiting spectral distributions for which no closed form expressions are known. We demonstrate the nature of these limits through some simulation results.