In this paper we consider approximate policy iteration based reinforcement learning algorithms. In order to implement a flexible  function approximation method we propose the use of a non-parametric methods with regularization, providing a convenient way to  control the complexity of the function approximator through changing a single parameter.  This idea is explored in the context of policy  teration for the purpose of evaluating policies. Two specific ways of implementing regularization are proposed: One for LSTD, one for the recent (modified) Bellman residual minimization (BRM) method. We derive efficient implementations when regularized  solutions are sought for over reproducing kernel Hilbert spaces. For the BRM method we prove generalization bounds.