The Norwegian boss is under intense pressure after a 5-0 home thumping by Liverpool and a one-sided 2-0 defeat at the hands of Manchester City on Saturday.
Ferdinand has been one of many ex-United players to voice support for his former team-mate, who led the club to a second-place finish in the league last season.
The ex-England defender said fans started the season full of hope after the high-profile signings of Cristiano Ronaldo, Jadon Sancho and Raphael Varane before results nosedived.
“I sit here now and I don’t think we can challenge to win this league,” Ferdinand said on his Vibe with Five podcast.
“I look at our team every week and I’m wondering what are we going to do tactically. I don’t see any philosophy or an identity in the Man United way of playing, whatever that should be from the management. I sit here confused, looking at the team.”
Ferdinand said United had made steady progress during Solskjaer’s nearly three years at the helm but now he felt there is a need for a change.
“I was always deep down a bit sceptical,” said the 43-year-old. “Could he take us on to be champions? I wasn’t sure, I wasn’t fully convinced — I hoped he would be able to do that.
“But the showing with the squad that he accumulated, to the beginning of this season –- and what I’ve seen this season — I just feel it might be about the time now for the baton to be handed over to somebody else who can take us on now.
“And I think Ole would leave now with his head held high because from when he came in to where he got us to the beginning of the season I think it’s positive.”
United forward Marcus Rashford acknowledged that the Red Devils are going through a “difficult patch” but said the club had to maintain the ambition to end a four-year trophy drought this season.
Rashford was speaking from Windsor Castle on Tuesday after a ceremony for the royal honour he was awarded following his campaign to help disadvantaged children.
On the field, the England international has recently returned to fitness after shoulder surgery, scoring three times in six appearances.
“I know at the moment we’re in a difficult patch but the desire and ambition never changes,” said Rashford.
“We all play for the badge and the club, something for me I’ve dreamed of since I was a kid.
“For the players that we sign, they see Manchester United as the great club it is and they want to give everything they can for the badge.
“And those things remain the same whether you’re in good spells or bad spells.”