The government has described Northern rail services as "unacceptable" as delays continue to plague its trains.
Transport Secretary Grant Shapps said he was "not prepared" to tolerate the commuter line operator's poor performance.
He told the BBC: "There's a couple of ways that can go, but one is to strip the franchise."
Northern said the late delivery of big infrastructure projects had disrupted its services.
The RMT union described the comments as a "distraction" to draw attention away from rail fare increases coming into effect from Thursday.
In October, Mr Shapps said he had taken steps that could lead to Northern losing its franchise.
Giving evidence to the Commons' Transport Select Committee last year, the transport secretary said that he had issued a "request for proposals" from the company and the Operator of Last Resort.
This could mean Northern's service being brought under government control.
The rail operator company has faced several issues in recent years, such as cancellations after the introduction of new timetables.
In a BBC interview on Thursday, Mr Shapps was asked whether his intention was to remove Northern's franchise.
He replied: "That's right. In the autumn I wrote to the necessary parties in this with what's called a request for proposal, and that's simply where you say: 'Look, I'm going to take action.'
"There's a couple of ways that can go, but one is to strip the franchise, one is to have a short-term contract, but yes, exactly as you said, I'm simply not prepared for the service on Northern to carry on as it is and I am taking action."
David Brown, managing director at Northern, said the company had faced challenges outside of its control in recent years.
One includes the late delivery of major infrastructure projects, such as the North West electrification, more than two years late.
Mr Brown added: "Arriva and Northern remain fully committed to delivering the transformation of the North's railways and improving customers' experience. We are delivering the biggest transformation of local rail for a generation."
The RMT union's general secretary Mike Cash also called the announcement a "distraction" from fare rises.
He said: "If Grant Shapps was serious, he would set out a timetable for removing the Northern rail franchise from Arriva and their replacement with the public sector operator."
Millions of commuters now have to pay an average of 2.7% more for train tickets, with some annual passes to increase by more than £100.
The Department for Transport confirmed that Mr Shapps had started the process that could strip Northern of its franchise.
Shadow transport secretary Andy McDonald said that Northern "should have been stripped of its franchise years ago".
Labour mayors Steve Rotherham and Andy Burnham previously called on the government to take control of the service.
A review of UK railways is already taking place.
It is being led by Keith Williams, the former boss of British Airways, and its findings are due to be published soon.
Source: BBC