Conversation with a remote person can interfere with performing vision-based tasks. Two experiments tested the role of general executive resources and spatial attentional resources in this interference. Both experiments assessed performance in vision-based tasks as participants engaged in a language task involving a virtual remote speaker. In both experiments, the language task interfered with the vision task more when participants were speaking or planning what to say next than when they were listening. In Experiment 1, speaking or planning what to say next were also associated with higher interference from a visual distractor than listening, indicating that preparing to speak and speaking pose higher executive requirements than listening. In both experiments, localizing the voice of the remote speaker to the front of participants slightly reduced interference in comparison to other directions. This suggests that remote conversation requires spatial attention resources for representing the position of the remote person.