ISTANBUL—A United Nations court ordered Turkey to release an international war crimes judge arrested in the nation’s post-coup crackdown so he can resume his role in an appeals case against a Rwandan convicted of genocide.

The U.N. Mechanism for the International Criminal Tribunals said in a ruling published Tuesday that Aydin Akay’s detention in Turkey violated his U.N. diplomatic immunity and the principle of judicial independence, as well as Turkey’s obligations to the United Nations.

Mr. Akay, 66 years old, was the presiding judge in a case and oversaw procedural motions at the U.N. court set up to hear appeals from the Yugoslav and Rwanda war crimes tribunals when authorities arrested him at his home in Turkey in September.

Turkey alleges the judge is one of tens of thousands of secret members of the religious group led by U.S.-based Turkish cleric Fethullah Gulen, whom President Recep Tayyip Erdogan blames for the failed July coup. Mr. Gulen denies any role in the coup. Mr. Akay has denied being a Gulenist.

The Turkish Foreign Ministry didn’t respond to requests for comment. The government hasn’t participated in the U.N. court hearings that led to the ruling.

Previously, officials familiar with the case have said they don’t believe Mr. Akay’s position as a U.N. court judge provides immunity from domestic Turkish criminal investigations.

The U.N. court has the power to refer Turkey to the Security Council for possible sanction if Turkey ignores the court ruling, according to a person familiar with the situation.

Mr. Akay hasn’t been charged by Turkey. The president of the U.N. court said his imprisonment has delayed his court’s work and violated a main principal on which the U.N. body was established: the impartiality of judges free from interference from their home country or any other nation.

“The right to an independent and impartial tribunal is an absolute right that may suffer no exception. To uphold this right…diplomatic immunity is a cornerstone of an independent international judiciary, as envisaged by the United Nations,” Judge Theodore Meron wrote in his ruling.

Mr. Akay spent a career in Turkey’s Foreign Ministry defending the state against human rights allegations. His defense lawyers now say he is a victim of state overreach.

Turkish officials have accused him of ties to Mr. Gulen due to the fact that he, along with tens of thousands of Turkish citizens, downloaded a chat app called ByLock, which the Turkish intelligence agency believes was a means of communication used by followers of the cleric.

Authorities have compiled a list of more than 200,000 ByLock users as part of the government’s investigation into alleged Gulenist plots in Turkey. Around 9,000 alleged users have been detained since July.

However, Turkish intelligence officials have told the Journal that ByLock wasn’t used by the coup plotters, and that by 2015 the app had fallen out of favor among so-called Gulenists.

Mr. Akay has told Turkish investigators that he used ByLock, in addition to other commercially available chat apps, to discuss personal issues with friends, according to court documents reviewed by The Journal.

The Turkish Justice Minister’s office and the prosecutor assigned Mr. Akay’s case have declined to comment, citing privacy grounds.

Mr. Meron’s ruling on Tuesday was precipitated by a motion made by the defense lawyers for a former Rwandan official, Augustin Ngirabatware, who had been convicted of inciting genocide. Mr. Akay was the presiding judge over his appeals proceedings, which were due to start in the fall in The Hague but have been delayed due to Mr. Akay’s detention.

Mr. Ngirabatware’s lawyer says there is new evidence that he believes will exonerate his client and reverse his 30-year conviction.

Mr. Meron’s ruling Tuesday ordered Turkey to release Mr. Akay by Feb. 14.