Hi, would you mind giving me a sample of your capture? I would like to take a look at the actual quality of the video before commenting on the quality of your encode, or discuss about the proper techniques of encoding a video.
But here's the general guideline, if you are using the mkv container, use the h264 codec. If you decide to use mp4v, divx or xvid (avs codec family), use the avi container. And whenever possible, choose xvid over mp4v.
The standard nowadays is pretty much [mkv/h264] or [avi/xvid] (I leave out [mp4/h264] deliberately). For streaming media, it's flv, avi, or wmv.