Vilnius used to have a fortress. Historical records of the city show that there was once an area that was surrounded with walls, and fortified. There was once the Upper Castle, and the castle was built on top of a hill, which is now known as Gediminas Hill. Now is it a green area, and is open to everyone who wants to climb the hill up to the top.
Very little remains of the Upper Castle. There are still structures remaining, but it definitely doesn't look like a castle anymore. The only intact building is the Tower, which is now a museum. It shows the history of the city as well as of the country.
If you climb up the hill, you can get a glimpse of the new part of town: we never really explored that area, but it was interesting to see it regardless. It was also interesting to have this feeling of awareness that there is a vast totally different area beyond just the Old Town. There's a river that bisects the city, and the new town is beyond, with skyscrapers and all.
There is a funicular that goes up the hill, in case you don't fancy hiking up. However, it doesn't operate in winter it seems. Nevertheless, it provides spectacular views of the city from various directions. So go up if you're in town.