Commons Speaker John Bercow is facing down the biggest threat yet to his authority as Tory MPs demanded his resignation after he declared that he voted Remain in the Brexit referendum.

With a vote of no confidence looming after a controversy-strewn tenure in the role, Tory MPs questioned how Mr Bercow could now be seen as neutral during parliamentary Brexit debates, while the Opposition rallied to his defence.

Leader of the Commons David Lidington warned there would be "strong" reaction to MrBercow's latest outspoken move, as he insisted the Speaker must command the confidence of the whole House.

A possible compromise solution was signalled by former culture secretary John Whittingdale, who suggested Mr Bercow's natural term of office was coming to a natural end.

Mr Bercow was plunged into fresh controversy after a video emerged of him talking to students at Reading University on February 3, in which he said: "Personally, I voted to Remain.

"I thought it was better to stay in the European Union than not."

In the video, obtained by the Sunday Telegraph, Mr Bercow says immigration has been a good thing for Britain.

He also referred to "untruths" during the Brexit campaign, how "promises were made that could not be kept", and expressed hopes that Parliament would maintain changes to working hours and health and safety protections after Brexit.

After Mr Bercow drew fire from some for branding US president Donald Trump a "racist and sexist" as he effectively banned him from addressing Parliament during his state visit, a number of Tory MPs said his position as Speaker was no longer tenable when the Commons returned from recess on February 20.

Mr Lidington told the BBC's Andrew Marr Show: "There will be strong reaction among some MPs to what he said at Reading, particularly after what he said about the state visit earlier in the week.

"Ultimately, the Speaker has to command the confidence of the House of Commons as a whole.

"John has his very strong supporters as well as his strong critics in the House of Commons, but we shall have to see how members as a whole respond."

The parliamentary website states: "The Speaker is the chief officer and highest authority of the House of Commons and must remain politically impartial at all times.

"On election the new Speaker must resign from their political party and remain separate from political issues even in retirement."

Mr Bercow's spokeswoman said how the Speaker cast his ballot in the Brexit poll, or Strictly Come Dancing, had no impact on his ability to deal fairly with all MPs.

She said: "Mr Bercow voted in the EU referendum, along with millions of others.

"The record shows that he has rigorously facilitated the raising of concerns of those on both sides of this argument, as he does on every other issue.

"The Speaker's impartiality is required on matters of debate before the House, and he has been scrupulous in ensuring that both sides of the argument are always heard."

The spokeswoman said Mr Bercow's record showed he was neutral in the chamber "irrespective of how he voted in a referendum, general - one would hope for himself - local, or Strictly Come Dancing."

James Duddridge, who tabled the motion of no confidence in the Speaker over his Trump comments, insisted that the Speaker is is "no longer impartial".

Speaking on BBC Radio 5live's Pienaar's Politics, he said: "I think there will be a vote of no confidence and I think he will go.

"There's absolutely no way Speaker Bercow can sit in the chair on European issues.

"When you become Speaker you must be impartial.

"He's no longer impartial, he's no longer able to continue to do the role, which is why I think the House will vote him down in a vote of no confidence.

"In reality he may see the lie of the land and go before he's pushed."

He added he had been "amazed" by the number of people to privately voice to him their support for his motion.

Mr Whittingdale told ITV: "John was elected with a very firm pledge that he wouldn't stay for more than eight or nine years, and we are pretty much getting close to the end of that period.

"So I wouldn't expect him to stay for much longer."

Tory MP Alec Shelbrooke told the BBC: "I don't think John Bercow's position as Speaker is tenable any more.

"He has to resign."

Ukip MP Douglas Carswell savaged Mr Bercow's attempt to stop Mr Trump addressing Parliament as "an absurd overreach ... student politics".

But he cautioned Brexit-backing MPs against trying to unseat the Speaker, warning it could create "Commons chaos" and increase the chance of defeat in key forthcoming votes on EU withdrawal.

Writing on his blog, the Clacton MP said: "Parliament has some big Brexit battles ahead. We need to win them. So Brexiteer MPs should base everything they do in the Commons on a simple litmus test: does it benefit Brexit, or not?

"That's my perspective on the latest effort to unseat the Speaker - and why I won't support it."

Downing Street refused to be drawn into the row, insisting Mr Bercow's position is a matter for MPs.

Prime Minister Theresa May's deputy spokesman told a regular Westminster briefing: "The position of the Speaker is a matter for Parliament and the MPs in Parliament."

Asked if the Government would allocate time for a debate and vote on the Speaker, the spokesman said: "It's a hypothetical situation in any event, but it would be a matter for the House authorities."