The 62-year-old television personality said as much to Cosmopolitan UK at the 2014 MTV EMAs on Sunday.
Osbourne served as a judge on the reality competition for five seasons, but told the magazine the series should quit while it's ahead.
"I think it's time they all went home," she said of the remaining mentors.
"I think it would have been lovely to end it on top last year when the numbers were great. Now, they're going down and it's a shame to keep something going when it's going downhill. You should always stop when you're on top."
Creator Simon Cowell, Louis Walsh, Cheryl Fernandez-Versini and Mel B currently serve as judges. Season 11 premiered to 10.09 million viewers at the end of August, but average viewership has fallen since Season 10, Osbourne's last season as a mentor.
Copyright 2014 United Press International, Inc. (UPI). Any reproduction, republication, redistribution and/or modification of any UPI content is expressly prohibited without UPI's prior written consent.