Relational Reinforcement Learning (RRL) addresses the use of relational representations of states and actions in RL rather than the usual attribute-values. Most works in this field aims at improving relational function approximation, or at adapting advanced techniques to the relational framework. However, little has been done so far to investigate basic Temporal Difference in RRL. In this paper, we propose adaptations of Sarsa and regular Q-learning to the relational case, by using an incremental relational function approximator RIB. In the experimental study, we highlight how changing the RL algorithms impacts generalisation in relational regression.