When in Reykjavik you definitely have to check out the City Hall, or Ráðhús. It's the big, grey, bowed building that juts out over the town lake, and though not everyone was pleased with it when it was erected (the bowed roof reminded many of the hundereds of American army quonset huts left scattered throughout Reykjavik after WWII, and with that the fact that too many Icelanders ended up living in those unheated corrugated iron nightmares ), its become a certain civic landmark.
Go ahead and walk the thin bridge from the duck-feeding corner of the town lake into the City Hall, get yourself a cup of coffee and a slice of cake at the Ráðhús Café, pick up a few tourist brochures and take a good long look at the giant relief map of Iceland that is usuallly on display. I personally sepnd at least a half an hour rediscovering it every time I see it, but then again I've got a definite thing for maps of any kind. This one, though, takes the cake.