If you had asked me a couple of years ago, I would have said "Absolutely tear it down!", but now I'm cooler to the idea.
JTK is right on that simply replacing the Gardiner with a utopian roadway is ridiculous. All of the previous proposals that the Waterfront corporation put forward a few years ago (oh, to make my living as a civic planning consultant) were 100% contigent on a) the Front Street extension, and B) an unprecedented increase in Public transit spending and infrastructure.
Well, the Front St. extension, which I thought was reasonable, was shot down, and that massive public transit overhaul? Yeah right. It's so frustrating that between the city council and the provincial government, we can't get any momentum for long term projects. As soon as an election is on the horizon, all those bold visions go out the window. I'm still waiting for that Airport rail-link.
So for better or worse, I bet we're going to be stuck with the Gardiner for another 20 years. So what now?
Charlatan's right about the Wall of Condos. Toronto's classic skyline, anchored by the CN Tower, Royal York Hotel and all those stately Bank buildings, is steadily being hidden by bland choc-a-block condo towers. So even if we do remove the Gardiner, the city core is a long way from the waterfront.
What I propose is that we bring the city right up close to it, and squeeze that cut as much as possible.
Build towers right up against it with several closed in pedestrian crossways connecting them on several levels. Shoulder some large garden and tree terraces at Gardiner height alng the way. If you're going to have billboards and video screens advertising to drivers, we might as well go big. Think Blade Runner big.

Down at street level, allow street vendors and small businesses to populate the bottoms of these. Improve the lighting, put in public art and work with the industrial setting to create kinetic urban environment that isn't a dead zone.
Above all, demand that all buildings presenting along that corridor must have bold and sophisticated designs. No more dull Taupe clad boxes. I drive the Gardiner every day, and with those new gleaming towers, not fifty feet away looming over you, it's not too hard to seeing this happening. This is the kind of plan on that private dovelopers can get on board with, doesn't depend on 20 years of Toronto centric Provincial governance, and circumvents Nimby obstruction.
For so many people, the view from the Gardiner is where they get their first impressions of the city. If we're going to lose the old face of the city, we'd better get busy creating a new one.