A united coalition would be the first to challenge the dominance of Mugabe's ZANU-PF at the polls.
"This is just the beginning of the building blocks towards establishing a broad alliance to confront ZANU-PF between now and the next election in 2018," said Morgan Tsvangirai, the leader of the Movement for Democratic Change (MDC)
"This is our collective struggle and I call upon the people of Zimbabwe to join us in working together... so that we can reclaim our country, our freedom and our dignity," Tsvangirai told journalists after signing the agreement.
Mujuru, who leads the National People's Party, was fired from Mugabe's government and the ruling ZANU-PF party in December 2014 and later formed her own party.
She said she was looking forward to working with Tsvangirai and that discussions had been on the table for a while.
"This is something we have discussed since last year," she said.
"We have taken more than six months to say the least, of consulting, discussing..."
The two leaders would not divulge the terms of their agreement, but Tsvangirai revealed that he was planning to enter into similar agreements with other smaller political parties.
Mujuru, a former ally of Mugabe was expelled from ZANU-PF after Mugabe's wife, Grace, accused her of plotting against the president and fanning factionalism in the party.
She has denied the charges against her.
A united coalition would be the first to challenge the dominance of Mugabe's ZANU-PF at the polls, after years of vote rigging allegations against the party which has been in power since independence from Britain in 1980.
ZANU-PF has endorsed Mugabe who turned 93 in February as its candidate for next year's election but the liberation movement has recently been hit by factionalism over Mugabe’s succession, with two main factions jostling to succeed him in the event of his death.
Although Mugabe has refused to name a successor, his wife Grace and vice president Emmerson Mnangagwa have been touted as potential replacements.