be off (one's) guard

To not be especially careful or vigilant; to not be prepared for danger or surprises. We'll wait until he's off guard, and then we'll take him and his company for all they're worth! Don't be off your guard, there are a lot of suspicious characters around here!