I think it's funny that people who carried carpet bags a long time ago were nicknamed “carpetbaggers.” If that was supposed to be a derogatory term, they could have done a lot worse.
I suppose that Southerners must have never seen a carpet bag before. Otherwise, calling these people carpetbaggers would have been no different than Americans calling visiting foreigners “suitcasers.”
The carpet bag probably represented something to them. It may have been a luxury item that they did not have access to in their poor section of the nation. After the Northerners came to be associated with them, I doubt a Southerner would dare carry one around.