on the downside
Considering the negative, adverse, disadvantageous, regrettable, etc., aspects of a situation, especially one that is or would be otherwise positive or beneficial. My son will automatically have EU citizenship because he's being born in Ireland, but on the downside, we have to apply separately for American citizenship.