go hard with (someone)
To be difficult, problematic, or unpleasant for someone; to be to someone's disadvantage, harm, or misfortune. It will go hard with John if he doesn't realize that Tommy is only interested in him for his money. It's going to go hard with us if the government cuts our funding any further.