On the corner where Castle Street meets White's Villas stands a granite tower known locally as Goat's Castle. It has been part of Dalkey's streetscape for centuries, and unlike most medieval buildings, it never really stopped working; it served as the town hall for a long stretch of its life and now houses the Dalkey Castle and Heritage Centre.

The tower is one of the better surviving examples of medieval defensive architecture in the area. It rises three storeys, built from roughly coursed granite with dressed stone quoins at the corners, and a vault sits above the ground floor. Its defensive intentions are easy to spot once you know what to look for: a stair tower projects from the southeast corner and climbs above the parapet, and the ground floor chamber is covered by a barrel vault running east to west.

The entrances tell a story of their own. Visitors today come in through a round-arched doorway on the Castle Street side, to the south, but this is a later addition, the original entrance was on the north, where traces of the original staircase survive. A drawing made by the artist Beranger in 1765 shows the south entrance as it looked at the time, damaged and blocked up, with another flat-arched entrance visible on the eastern side.

Inside, the building wears its centuries of adaptation openly. The ground floor chamber keeps its medieval character, lit by a deeply splayed embrasure to the east and a tall single opening in the southwest, with floor corbels still visible in the upper northeast corner. The first floor, measuring 10.6 metres by 5.95 metres, was remodelled at a later date as a council chamber, fitted out with wood panelling, double-light windows in the south wall and a large fireplace inserted into the southern wall. The second floor has a round-arched window at its western end. A parish hall, a far more recent addition, was built against the tower's northern wall;  proof that this medieval building is still finding ways to be useful to the town around it.