drop by the wayside

and fall by the wayside

1. Lit. to leave a march or procession in exhaustion to recover beside the pathway. A few of the marchers dropped by the wayside in the intense heat.

2. Fig. to fail to keep up with others. Many of the students will drop by the wayside and never finish. Those who fall by the wayside will find it hard to catch up.