As a further optimization I added Voxel Traversal support for my Area Lights and Point Lights, so that they could also benefit from the grid optimization if the user had requested it. This also had a pretty massive effect on the Venus De Milo scene.