I’ve heard it was the oyster shells they used for landfill.
The name Pearl Street is an English translation of the Dutch Parelstraat (written as Paerlstraet around 1660).