I had this problem and have been searching all over for an answer. Similar to the OP, for awhile now I’ve been pasting vector objects into Photoshop with no problems, everything looked nice and smooth when pasted either as a smart object or as pixels. Out of the blue, I all of the sudden started getting very jagged, pixelated images when pasted at any resolution as smart objects, pixels, whatever… It didn’t make any sense, especially when I KNEW the process worked just fine before.
The solution I finally figured out for my situation was fairly simple. Somehow “Anti-alias” had become unchecked in the pasting options. After you choose whether you want to paste as a Smart Object, Pixels, etc. you are able to resize/rotate the object and apply the pasted settings. Make sure the “Anti-alias” box is checked when you apply the paste settings and your images should look nice and smooth, even if you made the pasted vector larger before hitting the checkmark (or enter).
Obviously if you zoom in more than 100% you’ll notice pixels (you are pasting into a raster document), but it won’t be hard jagged edges anymore. Hope this helps!