"He's worried she won't be able to deliver good advice, and he's afraid contestants won't take her seriously," a source told Us Weekly in its April 9 issue, adding there are now more likely candidates that Cowell is considering for The X Factor's second-season judging panel.
"He'll probably go more in the direction of Mariah Carey or Janet Jackson."
If accurate, Cowell's new concerns would represent a significant change of opinion for The X Factor creator and judge -- who had stated he felt Spears would be a very intriguing addition to the show last month.
"I would be fascinated to hear what she had to say. She's still hot, she's still having hit records and she's still controversial, there's a reason for that," Cowell said during an appearance on The Billy Bush Show radio show in February.
"The fact that she's one of the most talked about -- not just pop stars -- but people in the world today, means that you've got this star power... This girl has got resilience. I would think that if she's managed to sustain a family, a career, which is tough, [The X Factor would be] a walk in the park."