The Football Association has cleared Mark Clattenburg of any wrongdoing following the alleged comments made to Adam Lallana.
Saints took their complaint to the FA when the Professional Game Match Officials Limited ruled that there was no case to answer.
But the FA has responded in similar fashion.
An FA statement says: "The FA can confirm it received a complaint from Southampton concerning referee Mark Clattenburg and his alleged comments made to their player Adam Lallana.
"The FA has given its full consideration to this matter and it is our view that the comments, as disclosed in Southampton’s letter of complaint, do not constitute misconduct under The FA’s Rules.
"Consequently, The FA will not be taking any further action in relation to this matter and now considers this case to be closed."
Saints are understood to be prepared to draw a line under the incident now but believe it will be very difficult for Clattenburg to be able to officiate any of their future matches following the controversy.
The club are also believed to feel that the situation has highlighted a number of flaws in the system, and that similar complaints in the future should be looked at by an independent panel rather than by the PGMOL body which includes former referees.
Furthermore, Saints are adding their support to the growing calls for all referees' communications to be recorded during matches to remove any grey areas over conversations had between players and officials, something which many are surprised is not already happening in every professional match.