You could cut some basic shapes of clothes out of colored felt, like shirts, pants, dresses, hats, coats etc. and put a piece of velcro on the back. Then you can take a bigger piece of felt and use a black marker to make an outline drawing of a boy and a girl (or a mother and father) with a face drawn on, so it’s obvious who’s who, then have your son pick out a red shirt for the Dad or a pair of blue pants for the mom and have him stick them onto the people outlines. You could do the same thing with fruits and vegetables, only you will have to get a bigger piece of felt (which you could actually staple onto a frame to make it easier to use) and then you would draw the outline of a carrot and apple and lemon and pear etc. and then have those same shapes and their appropriate colors cut out the same size in felt with velcro on the back.
You could also take just about any kind of magazine, but especially ones that have advertisements for clothing and cars in them. Go through the magazines so that you know what kinds of pictures are in each magazine (maybe make a little list for you to look at) then have you son go through one of the magazines to look for a blue dress or to look for a red car.
You could also get a bunch of random household objects that have distinct colors, make sure that you get at least 3 items of each color (black, white, red, yellow, orange, blue, green) and put them all into the middle of the floor or a bowl on the table. Then pick a color and have him find the other objects with the same color and match them up.