Basically the entire Inner Harbor area is a must-see. The Yard is three blocks from there. Fells Point is real nice as well, and I would recommend it, along with Canton further east, and Little Italy just north (best restaurants in the city from all accounts).
Also, don't forget to take the ballpark tour (which I mention because I'm a tour guide there
).