Alright! So I tried that and it didn't seem to really help much so I decided to just pop up the gearbox and investigate. After I got everything opened I realized that a screw, the one under the sector gear (with the hurricane shaped nut?) and noticed it was quite loose. After I tightened that up a bit, but not to the point that it was locked down I closed up the box and put it back in the receiver with the pistol grip on...
After I tested a few shots of pulling the trigger hard, it worked, pulled lighter still worked, but minimal pulling effort on semi, it still rattled away on auto... Since I am not going to be pulling the semi super light this is manageable for now, but whenever I need to open the box up again I will probably tighten that screw a bit more and see if it helps.
Thanks for the tips ThunderCactus