Toshihiro Nikai, secretary general of the Liberal Democratic Party said that if coronavirus cases rise to the point that hosting the event became "impossible" then "we would have to give up".
Tokyo, Osaka and several other prefectures entered a quasi-state of emergency this month, asking bars and restaurants to shorten their hours, and four more prefectures were set to be added, local media reported.
"If it seems impossible (to host the Olympics) any more, then we have to stop it, decisively," Mr Nikai said.
Cancellation is "of course" an option, he said, adding: "If the Olympics were to spread infection, then what are the Olympics for?"
The world's biggest sporting event has already been delayed by a year and is being held without international spectators.
The Tokyo Olympics Organising Committee responded with a statement saying all those involved in preparing for the Games remained fully focused on hosting them in the summer.
"Prime Minister Suga has repeatedly expressed the government's commitment to holding the Tokyo 2020 Games", the statement said.
"All our delivery partners including the national government, the Tokyo Metropolitan Government, Tokyo 2020 Organising Committee, the IOC (International Olympic Committee) and the IPC (International Paralympic Committee) are fully focused on hosting the Games this summer."