bane of (one's) existence
The source or cause of one's misfortune, unhappiness, frustration, or anxiety, usually used hyperbolically. I swear, this project is the bane of my existence. I've been working on it for months and still haven't made any real progress! Jane has been constantly annoying me all week. She's been the bane of my very existence!