Thanks for responding!
I tried the first thing you said and it was indeed checked, but it didn't seem to fix the issue. Another thing I want to note is it might not be as random as I thought it was.
BUT, as I am writing this, I just fixed the issue. If anyone else runs into this issue here is what you need to do: split each segment around the middle before the silent glitch happens. It doesn't matter exactly where it is, just split it. That way the part with the "fade in" and the part with the "fade out" are not connected. Don't drag or move any clips, just keep it split and there will be no sound or visual difference.
Here is a picture:
As you can see, it fades in at the start and fades out at the end, but in the middle area where the red line is, I split the video even though there is no logical reason, it just fixes this bug.