The decision followed an application filed by the opposition parties, who were upset by Zuma's decision to remove several cabinet ministers in March, including the widely respected finance minister, Pravin Gordhan, which prompted credit ratings agencies to downgrade South Africa's rating.
Speaker Baleka Mbete, who is also the chairwoman of the ruling African National Congress (ANC) had earlier said that the parliament had no power to make such a decision.
The opposition urged that a secret ballot would give ANC legislators, who are the majority in parliament, confidence to freely vote against Zuma without fearing intimidation or persecution for their decision.
"The speaker of National Assembly has constitutional power to prescribe that a no-confidence vote be conducted in secret ballot," Chief Justice Mogoeng Mogoeng said in his ruling.
He added that under the principle of separation of powers, his court had no legal basis to order a secret ballot, and the decision remained entirely with the Assembly speaker.
Opposition leaders have welcomed the court ruling describing it as a victory for democracy.
"This is a time for democracy. Court has made it clear that no political party can compel its members of parliament to vote against their conscience," Mosiuoa Lekota Leader of the Congress of the People said outside court.
A representative of the Democratic Alliance said they would write to the speaker asking her to set a date for the no-confidence motion.
President Jacob Zuma has been accused of many scandals including using millions of state funds to upgrade his rural home.
