VIENNA—The chief of the United Nations’ nuclear watchdog agency is headed to Tehran for meetings with President Hasan Rouhani and other senior officials in a sign of the group’s expanding role in forging a nuclear accord with Iran, despite concerns voiced by experts and U.S. lawmakers.

The Obama administration is positioning the International Atomic Energy Agency as a linchpin in its efforts to block Iran’s path to a nuclear weapon if a landmark deal is reached this month, according it a weight that worries some on Capitol Hill and in the Middle East.

The Vienna-based IAEA will be tasked with policing Iran’s nuclear sites and ensuring it doesn’t work on a covert weapons program, as well as extracting answers from Tehran on its covert nuclear activity in past years. The U.S. and others suspect those efforts were directed at developing the technologies for building an atomic bomb.

The U.N. agency will also have a critical say in the timing of the lifting of international sanctions by judging when Iran has completed initial steps to roll back its overall nuclear program.

“They have a very significant role here in how we are moving forward,” said a senior U.S. official who is taking part in the Iran talks.

But many U.S. lawmakers and Mideast diplomats question whether the IAEA will have the mandate, manpower and resources to effectively police Iran.

They noted that the agency has failed to detect covert nuclear programs in Iran, Syria, North Korea and Libya in past decades. And they said Tehran has repeatedly stonewalled the IAEA’s decadelong probe into allegations Iran secretly developed nuclear weapons technologies.

“I’m concerned. While I think the IAEA does some excellent work in general, the nature of what increasingly seems to be what they’re going to be called on to do…is greater than their present capacities,” said Sen. Bob Menendez (D., N.J.), a member of the Senate Foreign Relations Committee.

In particular, he said, the agency lacks the intelligence capacity to know whether Iran is cheating on any final deal, and would need more resources from the U.S. and five other global powers to fulfill an expanded role.

“This is a very large role that is being contemplated for a good agency, but one that is being stretched beyond its traditional limits,” Mr. Menendez said.

IAEA Director Yukiya Amano has been a regular at meetings with Secretary of State John Kerry and Iranian and Western diplomats in Vienna in recent days. The Japanese diplomat’s presence was designed to ensure the agency had full knowledge of the fine print of the agreement and could make “expert suggestions” going forward, the American official said.

“The IAEA has to have the ability to go anywhere, any time, in these inspections,” said the chairman of House Foreign Affairs Committee, Ed Royce (R., Calif.) on Wednesday. “If they can go in before equipment is moved or hidden, and based on their suspicions of this ability given Iran’s past history of cheating on every agreement, that is a big step forward.”

Iran’s paramount political figure, Supreme Leader Ayatollah Ali Khamenei has voiced public contempt for the IAEA in recent days, raising renewed questions about Tehran’s willingness to cooperate with the agency.

Iranian officials have repeatedly charged the agency with relying on fabricated documents to raise questions about Tehran’s past nuclear work. They also have accused the IAEA of leaking the names and whereabouts of nuclear scientists who were assassinated in recent years. The IAEA denies both charges.

“The agency has time and again shown it is not independent and just and, therefore, we are pessimistic about it,” Mr. Khamenei said in a nationally televised speech last week, according to his website.

The IAEA said Mr. Amano would meet Mr. Rouhani and other senior Iranian officials in Tehran on Thursday, but didn’t mention Mr. Khamenei.

The Obama administration initially described the international diplomacy with Iran as following two separate tracks.

One, led by the U.S. and five other global powers, has been focused on capping Iran’s production and stockpile of nuclear fuel in exchange for a lifting of international sanctions. The second, solely involving Iran and the IAEA, has sought to address the accusations that Tehran has secretly developed nuclear weapons technologies, which Iran has repeatedly denied.

These two tracks have increasingly converged in recent months, however, as Iran has balked at allowing international inspectors into its military sites and refused to answer the IAEA’s questions about its alleged weaponization work.

Mr. Amano entered into an agreement with Mr. Rouhani’s government in 2013 to find answers in 12 areas the IAEA says raise suspicions that Iran worked covertly to develop warheads for its ballistic missile program. Last month, he said a broader agreement between the U.S., Iran and the other world powers could “facilitate” Iran answering the weaponization issues.

Current and former IAEA officials said an Iran deal will require the agency to be granted more resources and powers if it is to be successfully implemented.

Iran has promised to give the IAEA more leeway to conduct inspections in the country by implementing an agency statute called the Additional Protocol. U.S. officials have said they are seeking additional rights for the agency that go even further, but acknowledge Tehran would have a right of appeal.

Iranian diplomats in Vienna in recent days have suggested that even the Additional Protocol won’t give the agency the anytime, anywhere powers that many on Capitol Hill are demanding.

“We will not allow anybody to enter the military complexes, because the [Additional Protocol] isn’t about letting inspectors visit and have a free hand in wherever they want to go,” said one of Iran’s lead negotiators, Majid Ravanchi, in an interview published Wednesday by the Bulletin of Atomic Scientists.

Olli Heinonen, formerly the top weapons inspector at the IAEA, said that to effectively monitor an agreement and ensure Iran answers questions about its past nuclear work, the agency will need additional access rights and resources that ensure timely access to all locations of interest

“This is not about following flows of nuclear material but to assess the completeness and truthfulness of Iran’s declarations” about its activities, said Mr. Heinonen, who is now at Harvard University’s Belfer Center.

Other nonproliferation experts said doubts about the IAEA’s ability to oversee an accord are overblown, as long as Iran isn’t able to lock inspectors out of certain types of sites.

“It will not be difficult to monitor the nuclear activity for the period of the limits,” said Mark Fitzpatrick, a former State Department official now at the International Institute for Strategic Studies in London.

Mr. Fitzpatrick said even after the tight limits on Iran’s nuclear activities lift, improved technology should allow effective monitoring of Iran’s nuclear work.

Some U.S. lawmakers said that while the U.N. watchdog has a role to play, its powers depend in part on the strength of the agreement and the involvement of the U.S. and other Western powers.

“The IAEA has always been essential to the inspection regime. The chances were slim-to-none the Iranians would allow American-backed officials to inspect their military sites,” said Sen. Chris Murphy (D., Conn.), a member of the Senate Foreign Relations Committee. However, Mr. Murphy said that while the group is “good at inspections,” it is “not very good at enforcement,” which will have to remain in the hands of the U.S. and other global powers that can levy sanctions.

“The IAEA will be successful so long as we have a clear process by which sanctions snap back in place if there’s a violation of the access,” Mr. Murphy said.

To date, though, Iran has only partially addressed two of the 12 areas of concern, Mr. Amano has said.