be the death of

Cause the death of something or someone, as in This comedian is so funny, he'll be the death of me. Although this phrase can be used literally, meaning "to kill someone or something," it has also been used hyperbolically (as in the example) since the late 1500s. Shakespeare used it in 1 Henry IV (2:1): "Since the price of oats rose, it was the death of him."

To distress or irritate to an intolerable degree.