BRUSSELS—A resurgent al Qaeda is attempting to take advantage of a weakening Islamic State to challenge its rival for pre-eminence in the global jihadist movement, the European Union’s police agency warned Thursday.

As a U.S.-led international military coalition rolls back Islamic State’s self-declared caliphate in Iraq and Syria, al Qaeda undertook efforts in 2016 to replace it as leader of the movement, Europol said in its annual report on terrorism.

Al Qaeda’s challenge to the extremist Sunni Muslims of its rival is occurring despite its messages in Arabic that tend to “downplay its international aspirations.” Its propaganda aimed at the West, however, echoes Islamic State’s calls for terrorist attacks, the agency said.

Europol said that al Qaeda in the Arabian Peninsula, the organization’s affiliate in Yemen, began publishing short English-language commentaries aimed at guiding would-be attackers in the West, offering advice on what groups to target in attacks and how to maximize the impact of strikes.

Among the report’s other findings, it said the amount of propaganda generated by Islamic State fell, as military pressure on it grew and countermeasures by social media companies became more effective, Europol said.

The total number of jihadist terrorist attacks slightly decreased in 2016, to 13 from 17 the year before. While the deaths caused by those attacks fell during the same period from 150 to 135, the casualty toll is still far higher than in other recent years, Europol said.

Meanwhile, arrests for terrorism-related offenses continued to increase for the fourth straight year, the agency said.

Rob Wainwright, executive director or Europol, said the need for improved information-sharing on suspected militants has become more pressing in the past two years.

“Jihadist terrorism has an international character and therefore needs an international answer from cross-border law enforcement,” he said.

While terrorist attacks in 2016 involved a wide range of available weapons—the attack in Nice, France, on Bastille Day was carried out by a truck—explosives were used in about 40% of the assaults, owing to “their high impact and symbolic power,” Europol said.

Europol warned that weaponized drones could become a threat in Europe, having already been deployed in Syria and Iraq.

The number of propaganda videos produced by Islamic State slowly began to decline in 2015, according to Europol, which pointed to anti-propaganda measures by social media companies that have started to disrupt Islamic State communication.

Europol cited the 2016 effort by Facebook Inc., Microsoft Corp. , Twitter Inc. and Alphabet Inc.’s YouTube to create a joint database to quickly identify and remove pictures and videos that promote terrorism.