lunatic fringe
the more extreme members of a group. Most of the members of that religious sect are quite reasonable, but Lisa belongs to the lunatic fringe. Many people try to avoid eating a lot of fat, but Mary is part of the lunatic fringe and will eat anything.
on the fringe
1. Lit. at the outer boundary or edge of something. He doesn't live in the city, just on the fringe.
2. Fig. at the extremes of something, typically political thought. He is way out. His political ideas are really on the fringe.