That date, however, tells only part of the story. The building that stands today was not completed until 1717, and by the nineteenth century it had fallen into such disrepair that only one wing remained habitable, occupied by the schoolmaster while the rest crumbled around him. The fact that it still functions as a school at all is, quietly, a rather unlikely outcome.

The college was founded and endowed by Elizabeth Villiers, and built as a free school for Protestants. The architecture is formal and considered: two storeys over a semi-basement, with a hipped slated roof and a symmetrical nine-bay entrance front facing west. The three end-bays at either side project slightly as break-fronts, the central bay of each blocked, lending the facade a rhythm that stops just short of austerity. A plain cut-stone surround frames the entrance door and extends upward to enclose a first-floor oculus, and the bays flanking the door are lit by tall round-headed windows at ground level with circular windows above. The rear elevation, facing east, mirrors this arrangement almost exactly, though its entrance is set at basement level. A stone gateway with corniced piers topped by balls marks the approach from the town. Writing in 1750, the historian Charles Smith called it "an elegant building, composed of one main structure and two returns", which is a fair description of the H-plan that gives it its distinctive shape. By 1791, however, the school was recorded as being in a very unsatisfactory state. It closed in 1821, and the buildings were left to deteriorate into ruins. Reconstruction followed between 1827 and 1829, and a further wing was added to the north side in 1878. The core of the early eighteenth-century structure survives, though the internal fittings and roof belong to later phases of its complicated history.