By Emre Peker

BRUSSELS—European Union officials are struggling to figure out how to improve vital economic and security cooperation with Turkey amid a widening political rupture that threatens the fraught relationship.

After Turkish President Recep Tayyip Erdogan won a referendum this month on constitutional amendments that will drastically expand the power of his office, European officials said the changes to Turkey’s democracy would make it incompatible with EU standards.

That brings Brussels and Ankara to a crossroads they have tried to avoid for at least half a decade: deciding the fate of Turkey’s all-but-dead talks to join the EU.

“There’s an obvious crisis in EU-Turkey relations,” European Parliament member Kati Piri, who handles relations with Turkey, said Wednesday. “The EU should officially suspend the accession talks if the constitutional changes are implemented.”

EU officials have signaled, however, that they wouldn’t unilaterally end Turkey’s aspiration to join the bloc. “It is for Turkey to clarify its intentions toward the EU,” European Commission Vice President Valdis Dombrovskis said Wednesday after a meeting of the bloc’s executive body.

For many officials in Brussels, a move by Ankara to drop its bid for EU accession would helpfully eliminate what has been a venue for mutual recriminations and mistrust.

Although Ankara says joining the EU remains a strategic objective, Mr. Erdogan has called for a referendum on the issue. He has also said he would back reinstating the death penalty, which was abolished to secure membership talks and has been called a red line by EU officials.

Mr. Erdogan’s political survival depends in part on the support of Turkish nationalists, whom he wooed in the referendum by bashing Europe with accusations of anti-Muslim views, fascism and Nazi practices. He faces re-election in two years, when the enhanced executive powers will kick in, and is therefore unlikely to abandon the rhetoric, an EU official said.

Such squabbles illustrate how far apart Turkey and the EU have drifted in recent years after seeking closer ties for more than half a century. A failed mid-July coup in Turkey accelerated the deterioration. Mr. Erdogan placed Turkey under a state of emergency and has presided over the arrests or detention of over 100,000 civil servants—over 1,000 people were detained on Wednesday—in an effort to root out supporters of the alleged coup plotters.

The EU has called the emergency rule undemocratic. The president hit back that Turkey’s Western allies were siding with putschists.

Adding to long-simmering tensions, the Council of Europe—a 47-member body including all 28 EU nations and Turkey—decided on Tuesday to start monitoring Turkey for the first time since 2004, making it the only country to come under renewed scrutiny over concerns about democracy, rule of law and human rights.

The developments make it harder for policy makers to find common ground. At stake is close cooperation on global issues such as fighting Islamic State, ending the Syrian war and addressing the refugee crisis, which destabilized Europe until Brussels struck a deal with Turkey to curb illegal immigration.

“If some people think they can wag their finger from Europe to Turkey and get it in line, they’re mistaken,” Mr. Erdogan’s spokesman, Ibrahim Kalin, said on Tuesday.

Mr. Kalin, defending Turkey’s record in keeping its promises to the EU under a March 2016 migration deal and as a candidate country, added that Brussels failed to deliver on pledges. Illegal migration from Turkey to the bloc almost stopped, but the EU has yet to grant Turkish nationals visa-free travel to Europe. A joint push to accelerate Ankara’s membership talks is dead.

As Brussels seeks to navigate the challenges, it is pondering how to re-establish the EU as an anchor for democratic reforms in Turkey. EU officials said Europe shouldn’t turn its back on the nearly half of Turkey’s voters who rejected Mr. Erdogan’s proposals in the referendum. The changes were approved with 51% support, in a vote marred by irregularities.

“We need Europe more than ever,” said Zekine Ozkan, a Turkish expatriate in France who traveled Tuesday to Brussels to protest the referendum outcome outside the European Parliament.

While the EU is trying to regain leverage it lost when Ankara’s membership push stalled, Turkey wants to revamp a two-decade-old customs union to bolster its slowing economy.

Updating the customs union and broadening its scope could help start mutually beneficial cooperation, the EU’s enlargement chief, Johannes Hahn, said. Mr. Hahn will seek a mandate Friday from EU foreign ministers to revamp Turkey ties based on a transactional relationship that both sides privately recognize as a good way forward.

Still, details of a future arrangement remain unclear and at least three previous resets have failed since 2012.

“It’s important to find a more realistic relationship between Turkey and Europe,” Mr. Hahn said. “The whole accession negotiation…has overshadowed this.”