Aside from hanging out with my former PhD adviser on the day before, I didn't really do much sight-seeing. The day after, we headed back to the Islands (we would find ourselves quite often in and near the Islands during this trip), and this time, we checked out the Sainte-Chapelle, or the Holy Chapel. It is a royal chapel constructed in the Gothic style, and is part of the medieval palace that used to occupy this location.
There are two levels. The lower level is this dark room that has a very interesting ceiling. But I recommend going first to the higher level, especially if the crowds haven't been there yet and you're relatively early.
The thing to see here are the massive stained glass that surround the upper chapel. This is Paris' oldest stained glass, dating from the 13th century. This church was built in just 6 years, having been completed in 1248. There are several scenes that are being depicted, there are written guides available telling you which Biblical story is being depicted by which glass panel. If you really want to know more, then an audioguide is also available for a small fee.
Try to get here early, as the queues can nevertheless get long, or get a combination ticket with the Conciergerie (get it at the Conciergerie where the queues are almost nonexistent) and skip the queues after visiting the other site. I am of the opinion that one's experience here would be more serene and peaceful if there were less people milling around strutting their selfie sticks.