It has been four years since Matt Lauer was ousted from the Today show.
On Nov. 29, 2017, NBC revealed that the former anchor was fired due to a complaint of "inappropriate sexual behavior in the workplace." Since then, Lauer, 63, has "lost a lot of friends," a source tells PEOPLE.
"People would check up on him for a while, but that's stopped to a degree," the source says. "He just stays to himself, he doesn't really reach out to people very much anymore or engage them and so he's been losing touch with a lot of people."
Lauer's "lavish lifestyle" also had to change, the source says. "When he left the Today show, he didn't get paid a penny after he got fired. NBC stuck with that. He's presumably sitting on mountains of money but then again, he had a lot of money in real estate."
Lauer was fired one month after sexual abuse allegations against Harvey Weinstein kicked off the #MeToo movement. In the four years since, Lauer has kept a low profile.
"He only wants to talk to people who are gonna take his side," the source says of Lauer. "He still feels like he got railroaded."
At the time of his firing, a source told PEOPLE that Lauer was let go due to sexual misconduct throughout 2014, including at the Winter Olympics in Sochi that year. Another source at the time said Lauer had viewed his relationship with the woman who made the complaint as "consensual" and was "dumbfounded" by the accusation.
Details of the complaint were not made public until Oct. 8, 2019, when Variety published an excerpt from Ronan Farrow's book, Catch and Kill: Lies, Spies, and a Conspiracy to Protect Predators, in which former NBC News employee Brooke Nevils alleged that Lauer anally raped her in his hotel room at the Sochi Olympics. Nevils told Farrow that she was "too drunk to consent" and also stated multiple times that she did not want to have anal intercourse.
In the book, Nevils also said she had more sexual encounters with Lauer in New York City, telling Farrow: "It was completely transactional. It was not a relationship.
At the time, Lauer penned a lengthy letter in response, claiming the encounter was "extramarital, but consensual." He said the encounter in Sochi was the beginning of his affair with Nevils and "the first of many sexual encounters between us over the next several months."
In his response, Lauer, who pointed out what he claims are "contradictions" in Nevils' story, stated that he has "never assaulted anyone or forced anyone to have sex. Period."
The allegations prompted several of Lauer's former colleagues to speak out. At the time, Savannah Guthrie, who spoke alongside co-anchor Hoda Kotb on-air, said "we are disturbed to our core" by Nevils' claims.
Former Today co-host Katie Couric recently spoke about Lauer's scandal. In PEOPLE's cover story last month, she said she was "shocked" when she learned of the allegations and called Lauer's behavior "grossly inappropriate" and "callous."
"And that's not the Matt I knew," Couric said. "There's a duality in human beings, and sometimes they don't let you see both sides."
How Matt Lauer's life has changed since being fired from the Today Show four years ago