The application was filed on February 25 by agent Sakib Softić on behalf of Bosniaks led by Bakir Izetbegović, the Bosniak member of the BiH Presidency.
The Croat member of the Presidency, Dragan Čović, confirmed to reporters in Mostar the ICJ had rejected the application.
A letter from the court said the application had been rejected because it had not been filed by relevant BiH institutions, Bosnian media reported.
The unilateral application was filed in spite of warnings by Serb representatives, who said it was illegitimate.
They also warned that Softić, appointed by the BiH Presidency for the lawsuit proceedings, was an illegitimate agent.