Simon Cowell is apparently unsure whether Demi Lovato and Britney Spears will return to The X Factor's judging panel for the reality singing competition's third season.

While judge L.A. Reid announced last week he will not be coming back to The X Factor, Cowell joked Lovato will most likely reprise her role as a judge but on one condition -- that she return "with a muzzle," Cowell told E! News.

"I know that she wants to come back, but I don't like to make predictions. We'll give it a couple weeks."

As for Spears, Cowell said her participation will depend on how busy the pop star's schedule will be.

"I don't know yet, to be honest with you. I haven't had the conversation with her whether she'd want to, what other commitments she's got," Cowell explained, adding that he would, however, like Spears to return.

"Yeah, I think I do, yeah. But like I said, and it goes back to L.A., it all depends on how much time they can give us."

Cowell also told E! News he's upset to see Reid go but understands why the producer made the decision to leave the show.

"I'm sad because he's a friend, but when someone feels they're neglecting something, which he obviously felt he was, then you've got to get someone else in who can give you the time and commitment you need," said Cowell.

"And I've never been afraid of change on any of these shows."