I guess you could go back to using the loading rod...
But seriously, you have a point, but methinks it's more of an issue with Bios where they're more porous than the harder "normal" BBs. Ultimately it's about relative hardness, but most of the parts in the loaders are made of swiss-cheese plastic. I don't think the loading action is significantly more stressful than say, the act of the nozzle loading the bb into the chamber though, especially with some nozzles being metal.