The 3 most other brothers left Delano at an early age