His comments follow reports that Chinese-owned Nexen is looking to team up with CN to ship crude from the Alberta oilsands to Prince Rupert, B.C.
The plans came to light after Greenpeace released Natural Resources Canada memos Monday, including a possible rail-based alternative to the Northern Gateway proposal to connect Alberta's oilsands with Kitimat, B.C.
There may be more fear than substance to the anti-oilsands activists' warnings, however, with Northern Gateway aiming to ship 550,000 barrels per day.
"That would require operating 10 trains of 100 cars a day roughly," Oliver said. "That's one train coming and going very hour. It obviously would need off-loading facilities and so on."
Still, transporting oil by rail is increasing in Canada.
Oliver's officials said in the first four months of 2013, around 272,000 barrels per day of oil travelled by rail in Canada -- up 77% from the same period in 2012.
Other demand for oil on the West Coast could come from media mogul David Black who wants to build a refinery in Kitimat, B.C.
Conservative MP Brian Jean, whose riding is home to oilsands activity, said oil trains are going to increase.
"I want to see oil carried by pipelines, but until we have the capacity to carry the oil by pipeline it's bluntly going to go on rail," Jean said.
NDP natural resources critic Peter Julian said that makes him nervous, especially when he thinks of the deadly summer explosion of oil-transporting rail cars in Lac Megantic, Que.
"(The) public's confidence in the Conservative government's ability to properly manage safety systems has been really tested," he said.
Julian accuses the Tories of trying to "skimp or cut back on safety protections," but Oliver disagrees.
"We've invested over $100 million for rail safety, we've increased fines for companies that break regulations and we've required rail companies to submit environmental management plans," he said.
A June study by the Manhattan Institute found that pipeline transport of oil is safer than rail.
The study found oil pipelines had only 0.58 serious incidents per billion ton-miles in the U.S., compared to 2.08 incidents for oil-carrying trains.