Dr Hoefflin, speaking on behalf of the Jacksons, said they were "outraged" at unofficial biographer Ian Halperin's claims the star met male building workers for sex sessions.
He said: "This man has caused huge upset and hurt to Michael's family, particularly to his mother Katherine at her time of grieving."
Dr Hoefflin also said he took the singer to the Playboy mansion, adding: "I know Michael Jackson was not gay. The truth is he lusted after beautiful women.
"I knew he was not gay because in all the time that I knew him that we were best friends - and i was his confidante and doctor - I never had any information that he was anything but straight.
"Also, I talked with all his security and they tell me there is no way he could ever have slipped out for sex with a construction worker. They said that it was impossible for Michael to have left the hotel and house without them around.
"Since Michael's children were born, he was always with them. It was very rare that he would go out without children. It was very rare that he would go out at all.
"We would also spend time together over the years going through books and magazines. Michael would always look at magazines of pretty girls, never boys.
"And finally, I know factually he had sexually relationships with women."
Dr Hoefflin, 63, said: "Michael loved beautiful women.
"After Thriller came out he asked me to take him to the Playboy mansion to see the stunning girls.