keep one's distance

Stay away; also, remain emotionally remote. For example, It's wise to keep one's distance from any wild animal, or Since the family argued with him, Harry's been keeping his distance. [Late 1500s]