The short answer: President Carter wanted to tax imported oil, Congress did not.

The medium answer: Initially, President Carter with congressional leaders wanted to tax oil imports to both balance the budget & reduce inflation. After a few months congressional leaders changed their minds against the tax. President Carter then vetoed the budget bill that did not include that tax.

Here is President Carter's own perspective from his memoir, Keeping Faith Memoirs of a President. Included later is the entire section but I will quote his key points now. It is important to note that these are direct quotes but I removed parts in the middle get at his core logic.

"During the early months of 1980 the most serious domestic problem was inflation. In March, to meet the problem, I called upon Congress to make major revision in the federal budget I had presented in January. My new goal was to reduce the deficit to zero. To demonstrate their resolve at the time, the congressional leaders had even called me to exercise my authority to impose an oil-import fee, in order to reduce oil imports and raise additional revenue, which would insure that we would not develop a budget deficit. I welcomed the suggestion... After I announced in March, with congressional support, that the fee would be imposed, the oil industry and other lobbyists had gone to work on Capitol Hill. In this particular case, the oil companies were able to marshal public support against increased gasoline prices, and the congressional leads simply abandoned their commitment and capitulated."

In summarize his view, to achieve a balanced budget & decrease reliance on foreign oil, President Carter accepted a proposal from congressional leaders to tax foreign oil. Interest groups against the tax got the public to be sensitive to higher gasoline prices causing congressman to switch against the tax. 

Lastly, there are his comments with reports on June 4th, 1980. Linked Here.

"Back in March, after very close consultations with the leadership in both the House and Senate, I decided and announced to the public that I would impose an oil conservation fee on imported oil to this country. Both the congressional leaders and I decided then, and I still maintain and am convinced, that this is important for the energy security of our Nation, for the economic security of our Nation, and also for the national security of our country."

Also of note:

"REPORTER. Mr. President, what's your alternative if they override the veto? Do you have other things that you could do?

THE PRESIDENT. There are some other alternatives. I hope that they will not override. But if I don't get but one vote on the Hill, I'm still going to veto this bill, because it's right and it's important to us to send to the American people and to the foreign countries a clear signal that we mean business in conserving oil."

What did Congress have to Say?

Tip O'Neil & Robert Byrd were in favor of the bill. These are likely the folks President Carter was referring to when he said "Congressional Leadership" in the passages above. Linked Here. 

According to an article from CQ Almanac (1980),  "Adding ammunition to the president's efforts to sustain the import levy, House Speaker Thomas P. O'Neill Jr., D-Mass. gave notice that he would do everything he could to keep the resolution to overturn the fee from reaching the House floor — including asking the Rules Committee to “sit on it.” 

and

"Senate Majority Leader Robert C. Byrd, D-W.Va. tried the carrot and stick approach — linking retention of the fee to an election-year tax cut. “…[I]f the fee goes, in my judgment,” Byrd warned, “so will the tax cut.""

Finally, "The majority leader did slip in criticism of Carter during the last hour of debate June 6. “The president has not done a good job of selling the case for the fee,” Byrd said." I haven't been able to find a different source for that last quote from Byrd.

Despite Congressional leadership backing, here is how the deal fell apart.  

"The majority of members remained unconvinced by their leaders' arguments, however. Rep. Charles A. Vanik, D-Ohio chairman of the Ways and Means Trade Subcommittee, said that after three days of hearings the panel found the fee should be abolished for three reasons: its inflationary impact; its questionable conservation effects; and its serious legal and administrative problems.

Vanik added another telling justification for defeating the fee. “Candidates for public office,” he said, “do not believe they can survive the burden of a 10-cents-per-gallon gasoline tax.”

"The depth of opposition to the fee was clear June 4, when the House passed the disapproval resolution by a 376–30 vote. (Vote 263, p. 80-H)

The 30 negative votes were all cast by Democrats, including top leaders such as Majority Leader Jim Wright, D-Texas and Rules Committee Chairman Richard Bolling, D-Mo. Speaker O'Neill also supported the fee."

What did Republicans have to Say?

In Carter's memoirs he said Ronald Reagan, campaigning at the time, was putting pressure on congress for tax cuts. So far I have not been able to find comments from Reagan on this specific tax or veto. Broadly speaking though, he stated his position on approach of the Carter administration in the October 28th, 1980 debate between him and President Carter. Link Here. 

In that video at 6:55 he says, "The President's economic plan calls for increasing the taxes to the point that we finally take so much money away from the people that we can balance the budget in that way. But we will have a very poor nation and unsound economy if we follow that path."

Richard Nixon gave his thoughts on the matter of taxing imported oil 7 years prior while he was also trying to deal with inflation. Link here.  On April 18th, 1973, in a special message to Congress he said the following,

"Effective today, I am removing by proclamation [4210] all existing tariffs on imported crude oil and products. Holders of import licenses will be able to import petroleum duty free. This action will help hold down the cost of energy to the American consumer."

His reasoning,

"In order to avert a short-term fuel shortage and to keep fuel costs as low as possible, it will be necessary for us to increase fuel imports. At the same time, in order to reduce our long-term reliance on imports, we must encourage the exploration and development of our domestic oil and the construction of refineries to process it."