drop away
1. . Lit. to fall off; to fall away. The leaves were still dropping away from the trees in November. The dead branches dropped away from the tree.
2. Fig. [for a group of people] to decline in number over time through disinterest or attrition. His friends gradually dropped away as the years passed. As the other contenders dropped away, Mary's chances for election improved.