We spoke to the guy in the jersey ... who tells us Lachey and his buddies -- all die-hard Bengals fans -- had been talking trash to him about the jersey for most of the 2nd half of the game.

The Chargers fan says Lachey made one particularly offensive comment about Olshansky that crossed the line ... and that's when the fan's wife shot back to Nick, "Well, he lasted longer than your boy band."

We're told Lachey flipped out and called the wife a profanity ... along with "a host of other profanities."

Tensions exploded ... and the Chargers fan says Nick lunged towards him ... grabbed him by the throat and shoved him to the ground. The fight can be seen in the footage obtained by TMZ.

As we previously reported, stadium security quickly responded and ejected Nick and his buddies. The Chargers fans were NOT asked to leave.

After the incident, Nick went to Twitter and BRAGGED about the situation -- saying, "Just got kicked out of the chargers stadium and couldn't be prouder! Go Cincy Bengals."

He later acknowledged the altercation ... tweeting, "i simply don't appreciate someone turning around and putting their finger in my face. call me crazy, but that deserved a reaction."

We're told the Chargers fan plans to file a police report against Lachey.