The trigger contact is dirty/worn/damaged by the spark when the connection is on.
Depends on the extent of the damage you may need to sand paper the contact or replacing it.
A MOSFET can prevent this in the future AND increase your rate of fire, but that will also makes the wiring more complicated, and you will need space for the MOSFET. If you have a rear wired M4 and the battery is in the stock, the buffer tube is a nice place for the MOSFET, but if you uses a buffer tube lipo that already occupy the buffer tube, good luck... You may still fit both in the buffer tube, but the space will be very limited.
__________________
|