'It's all mental with him' - Billionaire's late wife reveals

But it took billionaire Warren Buffett several decades to start donating his massive fortune — a decision his late first wife Susan grappled with, according to the new HBO documentary "Becoming Warren Buffett."

"The biggest thing about making money is time," Buffett, now worth more than $73 billion, says in the documentary about his decision to save his earnings.

"You don't have to be particularly smart, you just have to be patient. Susie didn't want to wait as much as I did — she never quite appreciated compounding like I did."

In 1977, when their three children were grown, Susan left the family's home in Omaha to focus on her charitable work.

"My mom moved to San Francisco and I think one of the reasons it was so important for her to leave Omaha was because she just left like she was kinda trapped in this environment, that everybody knew who she was, and she couldn't have her own identity," said the couple's youngest son Peter.

"[My dad] knew that there was something [my mom] needed to do, and that she really recognized that the money gave us all, and her, a choice in a lot of ways that a lot of people didn't have," their eldest son Howard said.

Susan chose to dedicate her time and money to charity, while Buffett continued working and growing his fortune (the two remained close friends, and legally married, until her death in 2004).

"It was a time where Warren was criticized. There was this very, very rich man who was getting richer every year, and really wasn't giving any money away and there was terrific criticism by some people, which Warren never said anything about," said Sandy Gottesman, a friend of Buffett's, as well as an investor and early backer of Berkshire Hathaway.

"That is a disagreement we have," Susan said in an interview recorded prior to her death, which is included in the documentary. "I run a foundation now. I think we should be giving more money away. But I understand why we don't — because it's business."

She continued: "To me, the crux of it is: It wasn't the money itself. You can see that in the way he lives. He doesn't buy huge paintings or build big houses or anything like that. It's all mental with him, and the money is his scorecard. He used to say to me, 'Everybody can read what I read. It's a level playing field.' And he loves that because he's competitive. So he's sitting there all by himself in his office reading these things that everybody else can read. But, he loves the idea that he's going to win."

In 2006, two years after Susan's death, Buffett ramped up his giving, promising to gradually donate all of his Berkshire Hathaway stock to philanthropy.

In 2010, he teamed up with fellow billionaires and close friends Bill and Melinda Gates to start the Giving Pledge, vowing to donate 99% of his fortune to charitable causes in his lifetime.

He said in his 2010 Giving Pledge letter: "To date, 20% of my shares have been distributed (including shares given by my late wife, Susan Buffett). I will continue to annually distribute about 4% of the shares I retain ... this pledge will leave my lifestyle untouched and that of my children as well. ... And I will continue to live in a manner that gives me everything that I could possibly want in life."

'It's all mental with him' - Billionaire's late wife reveals

Enhance Your Pulse News Experience!

JOIN OUR PULSE COMMUNITY!