The Online Etymology Dictionary says,
“c.1300, from O.Fr. matremoine, from L. matrimonium “wedlock, marriage,” from matrem (nom. mater) “mother” + -monium, suffix signifying “action, state, condition.” Related: Matrimonial.”
Patrimony, on the other hand, relates to inheritance. Isn’t it awesome how history and culture bleed through language?