so's your old man

Inf. The same to you!; Drop dead! (old. A catch phrase indicating basic disagreement or hostility.) Bill: You're acting like an idiot! Tom: So's your old man! I don't know what you said, but so's your old man!

The same to you, buddy! Casting another's parent (the “Yo Mama” jokes) or parentage (“you son of a bitch!”) in an unfavorable light has always been part of popular speech. “So's your old man!” is no more than a comeback that is used when (1) you don't want to provoke your antagonist very much or (2) you can't think of a more clever rejoinder. You might have said “Aw, your mother wears combat boots,” which, before women served in combat in the military, implied that she was less than feminine. That was a dangerous thing to do: fathers and other male relatives were fair game, but the female side of the family, and especially one's mother, was out of bounds before “Yo Mama” jokes crossed that boundary.