To remove a building hold space and left click on the buildings hitbox.
If you want to remove from the product slot, you can shift left click the building.