We address the problem of model-based reinforcement learning in infinite state spaces.  One of the simplest and most popular approaches is state aggregation: discretize the state space, build a transition model over the resulting aggregate states, then use this model to compute a policy. In this paper, we provide theoretical results that bound the performance of model-based reinforcement learning with state aggregation as a function of the number of samples used to learn the model and the quality of the discretization.  To the best of our knowledge, these are the first sample complexity results for model-based reinforcement learning in continuous state spaces.  We also investigate how our bounds compare with the empirical performance of the analyzed method.