I am facing the same exact issue.
Had to reboot the Machine to make sure video is not at its max use from resources.
Still same low quality video.
My Setting =
Record as .avi
Video Compression = H264 Native
Video Compression = 1 Highest
Sound Compressor = MP3
Sound Format = 44100 Hz, 128Kbps, Stereo
What else could be done ?