"I'm really sad even talking about it," said Lythgoe during a Thursday's broadcast of Ryan Seacrest's KIIS-FM radio show.
"It is one of those things that's a love of your life that you need to give up, purely and simply because I have to move on and the program has to move on as well. I hope they don't tweak it too much because it's a very simple format and that's the beauty of it."
Lythgoe confirmed earlier this week that he would "step back from [his] day-to-day producing work on American Idol" after previously serving in that role for Idol's first seven seasons. He reiterated those comments when talking to Seacrest.
"I've just decided to step back from the day-to-day running of American Idol basically," he said. "You know what the schedule was like, it was 24/7 and then some."
Lythgoe explained that "each season, it's been getting tougher and tougher" because more stuff has been put on his plate -- from serving as So You Think You Can Dance's executive producer and judge to organizing Idol's annual Idol Gives Back fund-raising event.
"I just needed a break to go and do something else basically," said Lythgoe, who will be devoting his time to an as-of-yet unidentified new venture with Idol creator Simon Fuller while also traveling to South Africa, Australia and Canada to work on local versions of So You Think You Can Dance.
Lythgoe had served as Idol executive producer along with Ken Warwick and Cecile Frot-Coutaz, and Seacrest asked him point blank who would be replacing him on the day-to-day operations.
"It's such a wonderful team now. Everybody -- thank goodness -- knows what they're doing," coyly answered Lythgoe.
"Really I'm just such a control freak that I didn't always let them off the leash. It's still going to be just as good and just as strong as ever."
Seacrest also asked Lythgoe if stepping down from Idol was "the most difficult professional decision" he's ever had to make.
"Not really," replied Lythgoe. "Coming to America was probably the most difficult decision. You're giving up everything in your own country and trying to make a go of it in another country. So that was probably the most difficult decision and that worked out so brilliantly I've decided I'm going to stay."