But the 6.9% swing from Labour to Conservative wasn't enough to oust Mr Whitehead, who was elected for a fourth time.
The result was clear by 3.30am, but wasn't declared by Mayor Liz Mizon until 5am after a recount was called for the neighbouring Southampton Itchen seat.
"I am delighted to be re-elected once again to Southampton Test, and, above all, delighted the people of Southampton Test have put their confidence in me and I will do my best for them in return.
"It is also a victory that shows that the values of fairness and social justice are burning brightly in Southampton. I will take that with me for what I think will be a difficult period for Parliament and the country over the next couple of years."
Speaking about his 5.5% majority, Mr Whitehead added: "I think that is a healthy majority under the circumstances and I attribute that to the extraordinary hard work of the Labour party workers and activists."
He said British politics was entering new territory, with no single party able to claim moral right to a monopoly of power in parliament.
However, the former city council leader would not be drawn on the future of Gordon Brown, saying his party leader had run a strong campaign.
"It is too early in what is a very uncertain result to say anything about the future of the Labour Party," he said.
"Gordon Brown ran a very strong campaign and that has been reflected in the Southampton result. He deserves credit for that."
A disappointed Councillor Moulton said he was proud the Conservatives had run a positive campaign.
"To reduce a majority of nearly 8,000 to 2,000 is quite an achievement," the 33-year-old said.
"I don't think we could have done anything more. We have increased our vote by almost 40% and are the only party to have gone up."
The city councillor, who arrived at the count at 10pm, added: "I will possibly run again, but my immediate priority is to spend time with my family."
With the Labour vote slashed, and the party losing its grip on power nationally, there was a subdued atmosphere at Southampton Guildhall.
Turnout was up on the last two elections to 61.1% (44,187), compared with 57.5% in 2005 and 56.3%.
Lib Dem candidate David Callaghan, who polled 9,865 votes, down -1.8%, said: "I am slightly disappointed. I was hoping that we would do a little bit better.
The freelance journalist, who was also running for re-election at Sutton Borough Council, added: "I would have liked to have spent more time on the campaign, but I also have to fight my own seat. It shows that if we had more time and resources then we would have done really well."
Pearline Hingston, the Ukip candidate whose purple hair match that of her party's logo, said: "I would like to say thank to all of the Southampton people who voted for me and I am sorry that I did not do better."
A clearly disappointed Chris Bluemel, the Green Party candidate, said: "It was a seat we never expected to win. I am not disappointed by the number, the most disappointing thing for me was that we came so far behind Ukip, a party that is so far to the right."