Essentially, they are. They’re not built below a “ground water” elevation, because the construction itself would be so problematic at that point. But if you look at a simple diagram of a pressurized water reactor (the most common commercial nukes in the USA), they are built with:
1. Concrete “reactor pools” (including the primary piping to the steam generators, and the bottom section of the steam generator itself) that are normally filled with water during operation anyway (that is, the reactor, primary loop piping and steam generators are built to operate inside a pool filled with water) and
2. Safety injection tanks, which are massive suspended tanks of borated water (because boron stops the fission reaction by absorbing the fission particles) within the containment building that in the event of a severe LOCA (loss of coolant accident) are gravity-fed to continue to flood the reactor pool.
In the case of the most severe and dire failures on the PWR plants I’m familiar with, when the operators / owners are ready to sacrifice the plant to prevent catastrophe, then the entire lower half of the containment vessel itself can be flooded, which also includes the reactor pool (say if the pool integrity itself had been somehow damaged in a catastrophic failure of the reactor, for example). At that point, all of the plant equipment in the bottom of the containment would be sacrificed as unusable, but the vessels themselves should be prevented from continuing to produce unmanageable heat and fission.