KASTRUP, Denmark—Sweden began enforcing tighter border controls Monday to curb the influx of asylum seekers, prompting Denmark to begin similar checks in a further weakening of Europe’s principle of open borders.

Sweden warned weeks ago that it would impose systematic identification checks at its borders, saying that the country of close to 10 million people was already straining to cope with the estimated 160,000 migrants who arrived in the Nordic nation last year.

Fearing that it would become the new destination for migrants unable to reach Sweden, Denmark said Monday that it was stepping up controls along its border with Germany.

“We cannot deprive people of their rights to seek asylum,” Danish Prime Minister Lars Lokke Rasmussen said in a televised address. “But we can make sure that those who do not have legal reasons to come to Denmark are turned away at the border.”

The border checks highlight Europe’s struggle to preserve its ideals of free travel and devise a common response to the biggest flow of displaced people since World War II. The European Union has focused on encouraging states on the bloc’s edge, principally Greece and Turkey, to slow the stream of migrants by more forcefully policing their borders.

Within Europe, several countries, including Macedonia and Hungary, have built razor-wire fences on their frontiers.

To the north, Sweden, Denmark and Norway have chosen to erect administrative barriers, rather than physical ones. In Storskog, the Arctic border post between Norway and Russia—along an Arctic route some refugees and other migrants have chosen as more obstacles emerge on the trail through southeastern Europe—Norwegian authorities have stopped allowing in asylum seekers since Nov. 30.

“These are asylum seekers that we think won’t be persecuted in Russia,” Norway’s Justice Minister Anders Anundsen said late last year. “It’s an abuse of the asylum system when they choose to travel from Russia to Norway to seek asylum.”

Mr. Rasmussen, the Danish premier, said Monday that he had ordered an increased police presence at all checkpoints after his government failed to reach an agreement with Germany to have transport companies verify passenger identifications on board trains and ferries.

“We do not want to see immigrants walking along our highways,” he said. “We want peace and order.”

Germany said introducing controls at inner-European borders wasn’t the right solution and urged European countries once again to work together to better protect the region’s external borders. “This highlights that we need a joint European solution,” government spokesman Steffen Seibert told reporters.

Germany, one of the main destinations for migrants pouring into Europe, has also sought to stem the human tide also by imposing document checks at some of its borders.

For Sweden, the new border regimen marks a dramatic shift from its previous migration policy. In 2013, Sweden was the first country to offer permanent residence to all Syrians fleeing war in their homeland. The open-door policy quickly made it the destination of choice for many seeking to start to a new life. By mid-2015 Sweden had taken in more Syrians than any other European nation on a per-capita basis.

Even as the number of migrants fleeing to Europe surged over the summer, Sweden’s Prime Minister Stefan Lofven joined a rally in central Stockholm in September saying that his Europe “doesn’t build walls.”

By November, however, Mr. Lofven changed tack, warning that Sweden was reaching the limits of the number of migrants it could absorb amid a sharp rise in anti-immigrant sentiment as well as violent attacks targeting migrants.

ENLARGE

With about 60% of migrants arriving without documents, Swedish police first began conducting spot checks in November. That meant undocumented travelers could still apply for asylum upon arrival in Sweden.

The new rules are far more strict because they are systematic and conducted by train and other transport operators before migrants enter Sweden. Those without official identification are turned back.

Transport companies have complained that the controls place too much responsibility on their shoulders, that their staff are poorly prepared to check documents and that stations aren’t designed to restrict pedestrian access to trains and buses. Swedish train operator SJ has suspended services to Denmark until it can iron out such problems.

On Monday, the new policy was on full display at the busy Kastrup railway station, which serves Copenhagen’s main airport and is the last train stop before the bridge to Sweden.

There, all passengers seeking to board Sweden-bound trains were subject to ID checks. Among them was Ali Reza Bali, a 16-year-old who had hoped the last leg of his 5,000-mile trek from war-battered Afghanistan to Sweden would be a straightforward ride over the bridge from Denmark.

That hope faded in the walkway approaching platform one.

With no identification document, Mr. Bali and his traveling companion Mustafa Sainzara, also 16, were held back by security guards as the train to Sweden moved east. After an epic journey through Iran to Turkey, then to Greece and across central Europe, the train tickets in their hands were rendered useless.

Mr. Bali bought a Danish SIM card for his mobile phone and began frantically calling contacts for help.

“I don’t know what I am going to do now,” Mr. Bali said. “I have family in Sweden and now I don’t know how to get to them.”