By Jeremy Page, Felicia Schwartz and  Carol E. Lee

A month after tough talk on China helped win Donald Trump the presidency, Beijing’s top diplomat visited Trump Tower and delivered a lecture on the need to respect Chinese interests.

It didn’t go over well. A day later, Mr. Trump challenged Beijing with a veiled threat to bolster U.S. ties to the democratic island of Taiwan.

Since then, both sides have maneuvered, largely behind the scenes, to lower the temperature, setting the stage for Mr. Trump’s first meeting with China’s leader, President Xi Jinping, later this week.

The summit now offers a test of that strategy and a chance for the two leaders, if they choose to take it, to recalibrate relations and reduce the risk of flare-ups over trade, North Korea or the South China Sea. With both presidents deeply invested in projecting strength and the promise of national rejuvenation, a disappointing summit could plunge ties back into turmoil.

Mr. Trump’s administration in recent weeks has dropped talk of blocking Chinese access to its South China Sea outposts and toned down its attacks on China’s currency policy. It didn’t join the United Kingdom, Germany and other allies criticizing China over the alleged torture of human-rights lawyers.

After challenging China on its most sensitive issue—U.S. ties with Taiwan, which Beijing considers a rebel province—Mr. Trump reversed course in February, clearing the way for the two-day gathering at Mar-a-Lago, the president’s private club in Palm Beach, Fla., which starts Thursday.

In Beijing last month, Secretary of State Rex Tillerson described the basis for U.S.-China ties as “non-conflict, non-confrontation, mutual respect, and win-win cooperation,” the exact language used in the past by Mr. Xi. The U.S. has long avoided echoing those words, fearing they amount to an agreement to steer clear of such contentious issues as Taiwan and the South China Sea, one of the world’s busiest trade routes. Mr. Tillerson’s hosts were delighted. Many U.S. officials were stunned.

The tone has been set in part by a charm offensive by China’s U.S. ambassador, Cui Tiankai, toward Jared Kushner, Mr. Trump’s son-in-law and a senior adviser, according to officials and government advisers in the two countries.

The White House declined to comment on Mr. Kushner’s contacts with Mr. Cui but said he had worked with Mr. Tillerson on persuading Mr. Trump to reverse course on Taiwan after other officials failed. China’s foreign ministry declined to comment.

White House officials said Mr. Trump would press Mr. Xi to agree to a “results-oriented” relationship with action on two significant issues: curbing Chinese exports to shrink the U.S. trade deficit, and policing Chinese companies that trade with North Korea to hobble Pyongyang’s nuclear and missile programs. North Korea launched a medium-range ballistic missile off the east coast of the Korean Peninsula on Wednesday morning local time.

Mr. Xi’s priority is to stabilize relations with Washington as he focuses on assuring his dominance in a Communist Party leadership reshuffle later this year, Chinese political insiders said. A crisis in U.S. relations, especially a trade war that slows China’s economy, presents one of the few threats to his authority.

China, however, has become accustomed to Mr. Trump’s rhetorical posturing and given no public signs of concession. It has reiterated demands for the U.S. to lift restrictions on technology exports to China and repeated a proposal, already rejected by Washington, for North Korea to suspend its nuclear program in exchange for a halt to U.S.-South Korean military drills.

Beijing’s strategy, Chinese government advisers said, is to offer to buy more U.S. goods and to invest more in the U.S., while pressing Mr. Trump to endorse a new framework for relations that acknowledges China’s status and territorial interests.

“We’ve moved from a period of great uncertainty, when the Chinese were very insecure about what Trump might do to harm their interests, to the Chinese potentially thinking they’ve got this guy in their pocket,” said Bonnie Glaser, a China expert at the Center for Strategic and International Studies.

Many veterans of negotiating with China, inside and outside the U.S. government, said Mr. Trump’s administration has lost leverage by backtracking on its own positions, as well as on longstanding tenets of U.S. policy, without extracting significant concessions from Beijing.

Some of these specialists feel the summit may be premature. The White House has neither resolved internal disputes over its China policy nor filled key posts to oversee it.

Confronting China, especially on trade, was central to Mr. Trump’s nationalist platform, but foreign-policy and pro-business officials now seek a more pragmatic approach.

Officials in both countries hope the meeting will allow Messrs. Xi and Trump to establish a personal rapport. The first ladies are invited, and the two leaders are expected to stroll in the grounds, similar to Mr. Xi’s 2013 summit with former President Barack Obama in California.

One fear for China is that Mr. Trump could embarrass Mr. Xi if the summit fails to live up to expectations, according to people on both sides briefed on negotiations.

“The Chinese should be well advised not to underestimate President Trump, and where he could lead things if he feels he’s being played,” said one U.S. official briefed on trade negotiations.

Mr. Xi, on the other hand, “can’t be seen making any major concessions on this trip” that might be construed as a trade-off for Mr. Trump’s observing the longstanding agreement on Taiwan, said a Chinese official close to the leadership. Even so, “China can always buy more Boeing planes,” the official said.

During the presidential campaign, Mr. Trump, a Republican, promised to label Beijing a currency manipulator his first day in office and to slap a 45% tariff on imported Chinese goods. His win plunged U.S.-China ties into chaos.

Mr. Trump, as president-elect, broke decades of diplomatic protocol on Dec. 2 by taking a congratulatory phone call from President, Tsai Ing-wen of Taiwan.

A week later, Mr. Xi sent State Councilor Yang Jiechi, China’s top foreign-policy official, to meet with members of the future administration at Mr. Trump’s signature building in New York City. Both Mr. Cui and Mr. Kushner attended, said people briefed on the meeting.

Mr. Yang, who is known for his abrasive style, told his audience—which included Steve Bannon, Mr. Trump’s chief strategist—about respecting China’s core interests, especially regarding Taiwan, said the people briefed.

“That lecture and its well-practiced server did have a negative impact on those he delivered it to,” said a person who spoke with those who attended.

A senior administration official said the attendees told their Chinese guests to expect a fresh perspective in relations but it was “ultimately for the president to decide.”

A day later, Mr. Trump suggested in a TV interview he might not abide by the longstanding one-China policy, an agreement by the U.S. not to pursue diplomatic relations with Taiwan.

China’s foreign-policy establishment quickly met to brainstorm a fresh approach, according to people who took part. They studied copies of Mr. Trump’s book “The Art of the Deal.” Several Chinese think tanks were commissioned to file reports on Mr. Trump’s personality.

Opinion was divided. One group saw Mr. Trump as a deal maker less interested in maintaining U.S. alliances in Asia and who could work with Beijing. Others regarded him as a serious threat to China’s economy and security.

The consensus emerged that Mr. Trump cared most about trade and U.S. jobs—areas where Beijing could offer compromise—and that he could be persuaded to maintain the status quo with Taiwan.

Mr. Xi sent a New Year’s greeting card to Mr. Trump in early January. Mr. Trump didn’t immediately reciprocate. On Jan. 13, Mr. Trump told The Wall Street Journal he considered the one-China policy negotiable.

Chinese government advisers saw a possible path to Mr. Trump through his family. Leading the way was Mr. Cui, whose easy manner contrasted sharply with Mr. Yang’s.

“Personal diplomacy matters,” said a person who has dealt with both men. “And in this White House, personal relationships are especially important.”

Family ties

Beijing found in Mr. Kushner a more direct line between Mr. Trump and Mr. Xi. Both presidents rely on a small circle of trusted advisers. Since Mr. Kushner became involved, U.S. officials have toned down their language.

“Trump’s son-in-law is key,” said Wu Xinbo, director of the Center for American Studies at Fudan University in Shanghai. “First, he’s our ambassador’s main point of contact with Trump. Second, he’s the main figure for passing ideas and suggestions on China policy.”

On Feb. 1, Mr. Cui persuaded Ivanka Trump, Mr. Kushner’s wife and the president’s elder daughter, to attend a Lunar New Year party at Beijing’s embassy in Washington.

The next day, Ms. Trump posted a video of her daughter, Arabella, singing a New Year’s song in Mandarin. The clip went viral in China, helping tamp down public criticism over Mr. Trump’s failing to send a Lunar New Year message, an annual White House courtesy.

Mr. Trump sent a holiday greeting to Mr. Xi a week later, in time for the last day of the New Year’s festivities.

The following day, Mr. Trump spoke with Mr. Xi by telephone for the first time since his inauguration and pledged to follow the one-China policy.

Whether Beijing offered anything in return was unclear. When asked that at a news briefing, White House spokesman Sean Spicer said: “The president always gets something.”

U.S. policy experts have expressed concern that Mr. Kushner, a 36-year-old former real-estate developer, doesn’t have the experience or expertise to effectively negotiate with Mr. Cui, a 64-year-old career diplomat.

The Kushner family’s real-estate firm was in advanced talks with a politically connected Chinese insurance firm, Anbang Insurance Group. Co., for an investment of as much as $1.25 billion in a New York real-estate project. Those ended last month after an outcry over possible conflicts of interest. Mr. Kushner had earlier sold his stake in the project and other properties to family members.

Mr. Kushner, who also has been asked to work on relations with Mexico and the Middle East, may be stretched thin. He traveled to Baghdad this week with U.S. officials, where he met the Iraqi prime minister at a military briefing on the fight against Islamic State.

Asked about Mr. Kushner’s heavy portfolio, the White House press secretary said Monday that Mr. Kushner oversees a team. His involvement in China and other foreign-policy issues stems from his work coordinating meetings with foreign officials during the transition, Mr. Spicer said.

There is one measure of Mr. Kushner’s influence that will be closely watched by Chinese officials during this week’s summit: whether Mr. Trump, like Mr. Tillerson last month, speaks of a relationship based on “non-conflict, non-confrontation, mutual respect, and win-win cooperation.”

The phrase was first used by Mr. Xi in 2013 to define a “new model of great power relations” between the U.S. and China. It was never endorsed by Mr. Obama’s administration because, officials said, it implied acceptance of Chinese dominance of Asia.

Mr. Tillerson’s use of the phrase in Beijing was a surprise, according to U.S. officials. “It wasn’t in the talking points, so I guess it remains a mystery,” one official said, adding that Mr. Tillerson had been “warned against the traps when the Chinese try to get you to buy into their formulations.”

Chinese state media has since applauded Mr. Tillerson, as have some Chinese officials. “Happily surprised,” one adviser to the Communist Party leadership said.