I will present an overview of recent work from my lab suggesting that a common principle for population codes might be that neural activity is automatically organized into a discrete set of clusters. The argument is primarily based on the analysis of populations of retinal ganglion cells using two different approaches. Using a hidden Markov model (HMM) that can accurately capture the statistics of population activity, we find each latent state is a well-separated cluster. When we repeat the same visual stimulus, we find that neural activity patterns are highly variable but that they map onto just one or a few clusters. Thus, clusters serve as population codewords that exhibit error correction. In addition, the receptive field of each cluster is qualitatively different than receptive fields of its constituent neurons. Thus, clusters form a different basis set of visual features. Finally, we show that there exist simple, biologically plausible decoding algorithms that can readout cluster identity. We will describe how these properties can be combined together to constitute a system of hierarchical pattern recognition.Using a maximum entropy model (MaxEnt), we show that the retinal population is in a low temperature state, analogous to a spin glass in statistical physics, where each basin in the energy landscape is a cluster of neural activity. Because the properties of any MaxEnt model depend entirely on the constraining statistics (such as firing rates and pairwise correlations), the above results may also apply to neural populations elsewhere in the brain that have the same low-order statistics. To this end, we show that if we scale down all of observed pairwise correlations by a factor >2, the retina remains in a low temperature state. This suggests that other population codes with pairwise correlations as strong as the retina may also exist in a low temperature limit, where neural activity is organized into clusters. We present preliminary evidence that neural populations in primary visual cortex are also organized into clusters.