The provision applies to all public officials, including health workers and local government employees, requiring them to alert authorities if they suspect someone is at risk of becoming radicalized.

Though avenues exist elsewhere for teachers to report students who they believe may be at risk, experts said they weren’t aware of any other countries have implemented a legal requirement to do so.

Many Muslims complained that the new requirement would erode trust between the community and public-sector workers and stifle debate, as students and others could feel like they are under surveillance.

The U.K., like other Western countries, is scrambling to respond to the threat of homegrown extremism leading to terrorist activity, including by trying to stem the spread of radicalization.

But some of the measures Britain is taking—including the new requirement for public-sector workers—go further than other countries.

Prime Minister David Cameron’s government is overhauling surveillance laws and plans to introduce legislation to Parliament this fall. Mr. Cameron has said he also plans a counter-extremism bill that would include banning orders for extremist organizations and steps to ensure colleges and universities don’t give a platform to extremist preachers.

Mr. Cameron has said the government planned a harder approach to tackling “the radical narrative that is poisoning young minds,” saying the attack in Tunisia last week that killed up to 30 Britons showed the need for governments to do more to fight radicalization.

More than 700 people have traveled from Britain to Syria or Iraq since the conflict started; On Wednesday, British police said they suspect that a family of 12 British-Bangladeshis had traveled to Syria to join the self-declared caliphate.

Speaking to Parliament on Wednesday, the prime minister said teachers and other public sector workers “have a duty to deal with radicalization and confront extremism.” He added that it was an issue not just for police and the government, but “is an effort for us all.”

The government says it has been training public officials to recognize signs of radicalization. According to a training guide, teachers and others should look out for people who show “feelings of grievance and injustice,” “a need for identity, meaning and belonging” or “a desire for political and moral change.”

The government says the requirement is intended to protect vulnerable people from being drawn into terrorist activity and not about spying on them. A government spokeswoman denied that the U.K. was singling out Muslims. She said the policy seeks to root out radicalization of any sort, whether it be neo-Nazism or Islamic extremism.

But many Muslims say the government’s approach makes them feel they are being targeted, alienating the community while adding legitimacy to the narrative of extremists who say that the Muslim world is under threat.

They and some counter-extremist analysts also say the government’s approach risks taking a step toward establishing thought crimes. Some law enforcement officials also say the new requirement will be difficult to enforce.

The Muslim Council of Britain, which represents more than 500 Muslim organizations, says the government is unfairly suggesting that Muslim communities are leading young people to extremism. It also disputes the idea that Muslims aren’t doing enough to combat radicalization.

On a recent evening in Blackburn, a town in the northwest of England where nearly a third of the population is Muslim, a group of imams, business owners, policemen and other professionals convened to discuss what the new measures meant for them.

Nasir Hafezi, a lawyer who has represented clients accused of terrorist offenses, said British Muslims are now subject to a greater degree of scrutiny than ever before. Overt displays of religiosity or political criticism—suddenly showing a preference for Islamic garb or voicing opposition to Western foreign policy in the Middle East—could now bring the attention of the security services, Mr. Hafezi told the audience.

“There is no published list of what constitutes a nonviolent, extremist idea, and ‘British values,’ as defined, isn’t a clear enough benchmark,” said Mr. Hafezi. “So when do we know when a line has been crossed?”

In the wake of the Charlie Hebdo attack in Paris in January, a government minister sent a letter to imams across the country urging them do more to stamp out radicalization in Britain, provoking an indignant response from Muslim leaders who said the community was being unfairly scapegoated for the actions of a minority.

In a speech last month, Mr. Cameron highlighted the role that families and communities play in countering radicalization and suggested some Muslims had “quietly condoned” Islamic State militant extremist ideology. The speech prompted criticism, including from a former Conservative government minister who said Mr. Cameron risked “demoralizing” British Muslims.