It’s a violation of YouTube’s Terms of Service:
http://www.youtube.com/t/terms?hl=en_US
“You agree not to access User Submissions (defined below) or YouTube Content through any technology or means other than the video playback pages of the Website itself, the YouTube Embeddable Player, or other explicitly authorized means YouTube may designate.”
Probably not illegal in that you could be arrested, but Youtube could certainly sue you. Would they? Probably not, as @Thammuz said plenty of people do it without repercussion. But they could.