The X Factor executive producer and judge said it's necessary to restrict the amount of British talent on an American show and, when taking into account himself and Jones, two may simply be too many on the Fox reality singing competition.
"He got a lot of stick. I don't think he did a bad job but, like America's Got Talent, there is a Brit limit on the show," Cowell told reporters while attending Britain's Got Talent auditions this week, the U.K.'s The Daily Mirror newspaper reported.
"Whether Steve's the guy or not, it comes under the heading of growth in general so there will be some tweaks to the show," Fox entertainment president Kevin Reilly told reporters during a Television Critics Association press tour panel in Los Angeles at the time.
"Hosting, as we know, it is a much harder job than meets the eye. I think everybody now has come to realize the value of [American Idol host] Ryan Seacrest."
Last month, The X Factordenied Jones is unlikely to return to the show's second season after The Daily Mirror reported Fox executives had already decided not to renew the host's contract, labeling his absence from the next edition "pretty much a done deal."
Reports that The X Factor was considering replacing Jones or giving him a new co-host first surfaced in October.