BRENNER, Italy—The Syrian family waited on the station platform under the cover of darkness. Just as the train to Germany was about to depart, they slipped through the doors, taking advantage of nightfall to avoid Italian police.

Whether they would make it to their destination was a question. Just over the mountainous border in Austria, police are working to catch the scores of migrants who attempt the same and return them to Italy. Most of them will try to cross again.

The situation is a microcosm of the European Union’s fraught migration policy, where a patchwork of immigration rules, different levels of tolerance for asylum seekers and economic prospects encourage migrants to scramble to reach the most hospitable countries.

The result is increasing tensions among neighbors throughout the continent, as migrants attempt to sneak onto trucks in France’s Calais to head to the U.K., or Germany sends police to the Serbian border to try to stop crossing from the east.

Some in Italy bristle at the idea they aren’t doing enough. “We do our best … but these migrants manage to cross anyway,” said Giovanni Pinto, head of immigration and border control at Italy’s Interior Ministry.

A surge in arrivals—102,000 have landed in Greece and Italy by sea so far this year, and hundreds have died in the Mediterranean crossing—has prodded the EU to search for solutions. But so far progress has been limited.

European countries put off until after the summer any decision on a plan to relocate thousands of migrants throughout the union, EU officials said Monday, amid bickering over how many people each country should take.

Italy and Greece made up nearly 80% of illegal border crossings into the bloc last year, according to EU data. However, most African and Middle Eastern migrants plan to head for Northern Europe as soon as they land, say government officials and aid organizations.

Only about a third of the 170,000 migrants who came to Italy by boat last year applied for asylum there, say aid groups and the EU. Instead, Sweden and Germany received half of the 600,000 asylum applications filed in the EU in 2014.

EU law says all migrants should be fingerprinted immediately upon arrival and must apply for asylum where they arrive. If they travel to another country after being fingerprinted, they should be returned to their country of arrival.

But large numbers of migrants arriving in Italy go unregistered and travel north undisturbed. Germany has pressed the Italians to be more diligent in fingerprinting new arrivals.

Italian officials say the huge number makes it impossible to register everyone, but have stepped up efforts since German authorities complained.

EU law isn’t clear on whether police can force them physically to be fingerprinted, and once migrants have made their way into the middle of Italy normal practice is to cite them for not carrying identification and then let them go. Other countries “are asking for [tighter controls] because they want to save themselves,” says Mr. Pinto.

Omar, a 26-year-old from Senegal who declined to give his last name, said he arrived by boat from Libya and spent a year in reception centers in Sicily before heading for Germany. He evaded police checks by switching buses and traveled at night until he reached Milan, a major hub for migrants heading north.

From Milan, Omar hopped a train bound for Munich. During the journey, Italian police forced him off twice. But on the third attempt, he made it to his destination, where he is living in a center in the suburbs and intends to apply for asylum. “I really hope I get asylum here,” he said. “If I don’t, I don’t know where to go.”

The Italians have acceded to German and Austrian demands to intensify the focus of existing joint border patrols between Italian, Austrian and German police on migrant crossings. But Italian police working along the Austrian border say they feel patronized by their German and Austrian peers.

“If I need a guardian, we are basically saying there’s something wrong with how we do our job,” says police officer Mario Deriu, whose team patrols the area around Brenner.

Even when the migrants are caught in Northern Europe, it can be tough to return them. One 34-year-old Syrian said he threw away his train ticket as soon as he reached Innsbruck so that the Austrian police would have no grounds to return him to Italy when they stopped him, because they couldn’t prove where he had come from. The most they could do is fine him for not having a ticket. He was eventually allowed to remain and apply for asylum in Austria, he said.

And when authorities issue orders for migrants to be returned to Southern Europe, few are carried out. Last year, Germany requested 35,000 migrants return to European countries, but only 15% were executed, due to legal and bureaucratic challenges.

Along Italy’s border with France, French police started systematically patrolling trains from Italy in April. That helped drive up the number of French requests to repatriate migrants to Italy to 2,500 so far this year, triple the number for all of 2014.

Moreover, French police simply send migrants they stop at the border town of Ventimiglia back to Italian authorities there. But the Italians manage to process only a third of these returns, so the French police often just free the migrants at the border and tell them to cross back into Italy by foot, says Stefano Cavalleri, head of local police union SAP.

The migrants then sleep at the train station or local parks and try to cross again the next day, he said.

In Brenner, the frustration is evident. “All these procedures make no sense,” said Mr. Deriu. “We are trying to solve a humanitarian problem with police means.”