Zones, including ports, should not grow from empty lots if they are not connected to a road. Roads include undergrounds utilities (i.e., water and power), so without a continuous road connection to utilities, the vacant lot should remain empty. The only exception to this is that ports do allow "stacking" up to 2 blocks deep from the water to simulate the industrial offshore backend of sea ports like warehouses, rail, landside shipping, etc.
You can tell if a zone is connected to a road and buildable using the inspection button from the control panel on the left and clicking on the zone like the screenshot below.