Official statistics showed passengers made 119 million fewer journeys in 2015-16 than the year before, a fall of 2.6%.
London also recorded its first drop in bus use since 2012 as congestion increased.
Transport for London (TfL) said it expected new "Hopper" fares to reverse the decline.
Campaigners blamed congestion, which they said has slowed the pace of buses to almost walking speed, for the decline.
The Department for Transport (DfT) said a drop in bus mileage was "largely due" to a fall in the number of services subsidised by councils.
Bus fares had increased 1.8% but the DfT said this was "similar" to other price rises in line with the 1.6% Retail Price Index of inflation.
There were 4.5 billion journeys made by bus in England in 2015-16, the lowest figure since 2006.
In the capital, bus usage was down 3% in a year, to 2.29 billion journeys.
Research by the Campaign for Better Transport (CBT) has found subsidies for bus routes had been reduced by £78 million since 2010, leading to cuts in services, particularly in rural areas.
Lianna Etkind, public transport campaigner at the CBT, said: "The news that bus use is falling in London is worrying, but as buses are now moving only marginally faster through the city than the average adult can walk, it's hardly surprising.
"Congestion is bringing London to a standstill and needs to be urgently tackled."
Gareth Powell, TfL's Director of Strategy for Surface Transport, said there had been a "small reduction" in bus journeys due to congestion caused by a number of factors, including development across London and "increased internet delivery traffic".
But "bus network reliability has now stabilised," he said, and the completion of major road projects and the introduction of a "Hopper" fare was expected to have a positive effect on passenger numbers.
A DfT spokesman said: "We provide some £250 million a year to support bus services in England and about £1 billion is spent annually to give nearly 10 million older and disabled people in England off-peak bus travel."
Source: BBC News