The former Tour de France winner said he was "gutted" after sustaining a "small leg fracture" while taking part in snowcross training.
Channel 4 said Sir Bradley had been keen to continue but would not appear after the fourth show in the series.
Last year, injuries to a number of celebrities prompted a review of safety procedures on the reality show.
"Seen a specialist, I have a small leg fracture and need to rest for 3-6 weeks. Good news no surgery or cast required".