The Magi did not arrive while Jesus was still lying in the manger. They came much later. Remember, the Magi were the ones that told Herod they were looking for the King of the Jews and when he had been born, which caused the Slaughter of the Innocents in which Herod killed every male child under the age of two. So Jesus was a toddler by then.