LONDON—This week’s deadly suicide bombing has reignited a long-running debate about how British intelligence decides which potential extremists to watch given the limited resources at its disposal.

Like Khalid Masood, who killed five people in an attack in London in March, the suspected Manchester bomber, Salman Abedi, had been known to security officials. But both appeared to be on the periphery of investigations, illustrating the challenges in monitoring a growing roster of potentially dangerous radicals.

While its island geography gives Britain an advantage in monitoring new arrivals, homegrown extremists—many with loose or no concrete ties to terror groups—are often difficult to track.

The number of known Islamist extremists in Britain has risen to more than 3,000, a British intelligence official said, almost doubling from a decade ago. But watching suspects around the clock placed an intensive demand on resources, and intelligence services can only follow a small percentage of these people full-time, officials said.

MI5, Britain’s domestic intelligence agency, is managing about 500 active investigations, a U.K. security official said. Abedi was part of a wider net of people being investigated because they are suspected of being a threat to national security, referred to by MI5 as “subjects of interest.” Officials declined to say what reported actions had put Abedi in that category.

U.K. security, police and intelligence agencies say the pace of threats is quickening and unprecedented: A senior government official said they have disrupted 18 separate terrorist plots in Britain since June 2013, five of them since the Westminster attack in March.

A U.K. security official said Monday’s attack in Manchester may prompt intelligence agencies to examine procedures for prioritizing suspects. “Intelligence is a moving picture; it requires an evolving response,” the official said. Even though the government has increased counterterror budgets, the official said, resources are finite and all suspects can’t be actively monitored.

Raffaello Pantucci, director of international security studies at the Royal United Services Institute, said intelligence agencies have been rethinking how to prioritize the ever-expanding pool of suspected assailants. Officials will also examine how to ensure potential threats don’t slip through the dragnet, he said, particularly when they’re traveling in and out of the U.K., as Abedi had, most recently by taking what relatives said was a trip to Libya before returning to England last week.

“Quite clearly, there has been a slip-up,” Mr. Pantucci said. “The fact that he was able to get in and out of the country like that is quite worrying.”

He said intelligence agents prioritize suspects based on activities they have recently been involved in, the people they are close to and the chemicals or tools they possess. MI5 prioritizes investigations according to the risk they are judged to carry, he added.

The four broad categories are: Priority 1, where intelligence suggests attacks are being planned; Priority 2, where intelligence suggests high- or medium-risk activity, such as terrorist training; Priority 3, where uncorroborated intelligence points to possible threats to national security; and Priority 4, assigned to cases where individuals risk re-engaging in extremist activity.

One reason Abedi may not have been a top priority for the security services, analysts say, was that his strongest known connection was to Libya, rather than to Syria or Iraq. The emergence of Islamic State in those countries has proven a magnet for foreign-born fighters, meaning those returning to their home nations from the violence of the would-be caliphate are the ones squarely in the authorities’ sights. British investigators told French authorities that Abedi had probably also traveled to Syria, according to the French interior minister.“Syria has tended to be the focus of public concerns about returning jihadist wannabes for obvious reasons—it’s a much larger conflict. But there will now certainly be questions asked about whether enough attention was being paid to Libya as a laboratory of atrocity,” said Tim Wilson, an expert in terrorism at the University of St. Andrews in Scotland.

Analysts say sifting through the intelligence on potential terrorists is never easy, yet the U.K. has been relatively successful in thwarting plots. Monday’s attack was the first such large-scale bombing in 12 years, though other terrorist acts have taken place.

Stopping terrorists requires “an art, not an algorithm,” said Samir Puri, lecturer in war studies at King’s College London. “It’s not as if individuals like [Abedi] have free rein to do what they want. In the main, they get caught.”