twiddle one's thumbs

Be bored or idle, as in There I sat for three hours, twiddling my thumbs, while he made call after call. This expression alludes to the habit of idly turning one's thumbs about one another during a period of inactivity. [Mid-1800s]