The prime minister acknowledged the uncertain outcome early Friday. “At this time more than anything else, this country needs a period of stability,” Mrs. May told reporters upon winning her own seat. “If the Conservative Party has won the most seats and most votes, it will be incumbent on us to ensure that the country has that period of stability.”

With most races decided, projections put her ruling Conservatives short of the 326 seats needed to win a majority in Britain’s 650-seat Parliament. Early Friday, the British Broadcasting Corporation said it saw the Conservatives winning 318 seats, compared with 330 in the outgoing Parliament. The main opposition Labour Party gained ground to win what some projections put at 267 seats.

The pound sank sharply against the dollar after the exit polls and remained low through the night but the U.K.’s FTSE 100 stock index rose more than 1% in early trading.

Labour leader Jeremy Corbyn called on Mrs. May to resign. “The prime minister called the election because she wanted a mandate and the mandate she got is lost seats…and a lost mandate,” he said. “I would think that that’s enough to go.”

Steven Fielding, professor of political history at the University of Nottingham, said Mrs. May’s future as prime minister was at risk.

“The capital she had with her own party—that’s been spent, that’s all gone,” he said. “If you call an election to reinforce your authority, to reinforce your negotiation hand and you don’t get that endorsement, clearly people are going to ask questions about you.”

Projections early Friday suggested a Labour gain of more than 35 seats. In other surprises, the Scottish Nationalist Party lost more than a dozen seats and the Liberal Democrats appeared to have gained several seats.​

In a major win in London, the Labour Party gained Battersea from the Conservatives, a sign that anti-Brexit sentiment may have hurt the Conservatives in areas where a majority of people voted to stay in the EU. In another significant result, Labour Party held on to Bridgend in Wales, a key Conservative target that Mrs. May visited near the start of the campaign.

The political uncertainty comes as Britain prepares to begin talks on June 19 on extricating itself from the EU after 44 years. The two sides already face a tight timetable to agree on a host of complex issues. Depending on the election’s final results, forming a working government could turn out to be a drawn-out process that would threaten to leave Brexit negotiations on hold.

The prospect of a stable coalition similar to the one formed quickly in 2010 between the Conservatives and the smaller Liberal Democrats seems distant. The Liberal Democrats and the Scottish National Party have ruled out joining a coalition, though leaders have said they would consider a looser alliance.

In the event of a hung parliament in the U.K., the incumbent government gets first shot at crafting an administration. The Conservatives may be able to turn to longtime allies in Northern Ireland to support their policy program.

The result marks a startling gain for Mr. Corbyn, a veteran left-winger. He squeaked onto a Labour Party leadership ballot in 2015 and was expected to come in a distant last in a field of four. Instead, Mr. Corbyn galvanized young backers to win that contest. Support from younger voters during the recent seven-week campaign—and a manifesto that struck a chord with the wider public—helped Labour close a yawning polling gap with the Conservatives, though no poll before the elections put Labour in the lead.

If in the coming days the Conservatives can’t build a majority large enough to govern, Mr. Corbyn may get a chance. He can potentially draw on a broader range of center-left parties to build a looser parliamentary alliance, including the SNP, Liberal Democrats and Greens.

Either outcome leaves the path to Brexit more unclear.

Mrs. May, who campaigned for the U.K. to remain in the EU before the June referendum mandated its exit, has laid out plans for a clean break with the bloc, including leaving the European single market to gain tighter control of immigration.

Mr. Corbyn’s Labour Party said it would prioritize retaining the benefits of the single market and keeping closer links to the EU, hinting at a softer approach to talks. But with the party’s backers split between young, urban voters and traditional working-class strongholds that favored Brexit, it has struggled to lay out a clear plan on the issue.

Potential partners have conflicting and sometimes irreconcilable objectives. Some, such as the SNP and Liberal Democrats, hope to keep the U.K. in the single market, complicating the two main parties’ plans. Northern Ireland’s Democratic Unionist Party, a potential ally for the Conservatives, would likely demand its region gets special treatment in any Brexit deal as the price of its support.

To the surprise of most observers—and likely to Mrs. May’s detriment—the election campaign wasn’t dominated by Brexit. Voters paid more heed to bread-and-butter issues such as health care and pensions.

It was on one of those issues that Mrs. May appears to have made a damaging judgment, presenting a complex plan to finance elderly care that bombed with graying voters and their likely heirs, who feared they would have to pay more.

The campaign was twice suspended in response to terror attacks, allowing the Labour Party to focus on cuts to police numbers during Mrs. May’s six years as the minister responsible for security. Any new government will face the immediate challenge of overhauling the country’s counterterrorism strategy.

In Mr. Corbyn, Mrs. May and her team encountered a more formidable foe than they had anticipated. His avuncular style and refusal to abandon long-held but unpopular positions contrasted with what many saw as Mrs. May’s robotic campaign appearances and policy reversals.

Maria Caulfield, a Conservative lawmaker in Lewes, said the results would change Parliament. “It’s going to be a tense few days while we find out the results of this election and seeing who is going to be forming the next government,” she said.