The International Court of Justice, based in The Hague, said it could not take action on the request because "no decision has been taken by the competent authorities, on behalf of Bosnia and Herzegovina as a state".
The appeal was lodged on February 23 under a push by Bakir Izetbegovic, the Muslim member of Bosnia's tripartite presidency, despite opposition to the move from the country's ethnic Serbs and Croats.
According to the Dayton peace agreement that ended the country's 1992-1995 conflict, key decisions cannot be taken without the consensus of representatives of the three main ethnic groups.
Mladen Ivanic, the Serb member of the presidency, argued that the appeal request should have been decided by the presidency as a whole.
He welcomed the court's decision as proof that "no one can make any more decisions on behalf of Bosnia except its institutions".
But Izetbegovic, at a separate press conference, described the court's decision as "politically motivated".
He had claimed there was no need for the presidency to agree on the appeal because the request was made by a lawyer the presidency appointed in 2002.
Many officials in Bosnia had said the move could spark a new political crisis in the deeply divided Balkan country.
In the original case lodged by the Muslim-dominated government of Bosnia in 1993, Sarajevo accused Belgrade of orchestrating a genocide through widespread "ethnic cleansing" during the war, which claimed more than 100,000 lives.
In 2007, the ICJ found only one act of genocide -- the massacre of nearly 8,000 Muslim men and boys by Bosnian Serb forces in Srebrenica -- and said there was not enough evidence to suggest that Belgrade was directly responsible.
The ICJ did find, however, that Serbia, which gave political and military backing to Bosnian Serbs, had broken international law by failing to prevent the slaughter.
Izetbegovic had said that Bosnia had "new arguments" in its case, notably those presented during the trial of Ratko Mladic, the Bosnian Serb wartime army chief who is awaiting judgement at a UN tribunal.
The 10-year deadline for Bosnia to request an appeal of the ruling expired on February 26.
Eyewitness? Submit your stories now via social or:
Email: news@pulselive.co.ke
