Purpose: Vision restoration training (VRT) in hemianopia patients leads to visual field enlargements, but the mechanisms of this vision restoration are not known. To investigate the role of residual vision in recovery, we studied topographic features of visual field charts and determined residual functions in local regions and their immediate surround. Methods: We analyzed High Resolution Perimetry visual field charts of hemianopic stroke patients (n = 23) before and after 6 months of VRT and identified all local visual field regions with (“hot spots”, n = 688) or without restoration (“cold spots”, n = 3426). Topographic features of these spots at baseline where then related to (i) their respective local residual function, (ii) residual activity in their spatial neighbourhood, and (iii) their distance to the scotoma border estimated in cortical coordinates following magnification factor transformation. Results: Visual field areas had a greater probability of becoming vision restoration hot spots if they had more residual activity in both local areas and in a spatially limited surround of 5° of visual angle. Hot spots were typically also located closer than 4 mm from the scotoma border in cortical coordinates. Thus, restoration depended on residual activity in both the local region and its immediate surround. Conclusions: Our findings confirm the special role of residual structures in visual field restoration which is likely mediated by partially surviving neuronal elements. Because the immediate but not distant surround influenced outcome of individual spots, we propose that lateral interactions, known to play a role in perceptual learning and receptive field plasticity, also play a major role in vision restoration.