The fort is commonly known by the name Cromwell, though the historian Westropp pointed out that the name may predate the Cromwellian era entirely, possibly deriving from the Irish crom-choill, meaning a sloping wood. A map drawn by Baptist Boazio around 1590, during the reign of Elizabeth I, already marks the location as "Cromcoel." In the Ormond manuscripts, the same fortification is called Fort Fleetwood, a name that has largely been forgotten.

The fort was probably built in 1653, at the same time as a companion fortification on the far side of Valentia. A blockhouse, in this military context, is a compact masonry strongpoint designed for artillery and close defence, and the one at Fort Point survives in good condition. Its walls average two metres thick, and eight gun-embrasures remain, with three in each of the seaward faces, each splayed wide to allow a broad field of fire. Two internal buildings are incorporated into the landward side of the structure; one contains a small vaulted room that was probably used as a powder magazine. A window placed in the western flank of the gateway appears to have been positioned deliberately to allow musketry defence of the entrance. The fort was formally disestablished in 1669, after which most of its guns were redistributed to other Munster fortifications. The Ormond manuscripts record that four cannon were still buried at the site as late as 1684. The traveller Richard Pococke noted two cannon here in 1758, and those same guns are now kept within the lighthouse enclosure. About two hundred metres back from the blockhouse, on the landward approach, an earthen bank and external fosse, that is, a ditch, survive in bastioned outline. A fosse is simply a defensive ditch, and this one originally fronted a system of four bastions, projecting angular platforms intended to eliminate blind spots in the fort's field of fire. Three were demi-bastions, half-sized versions, with one complete bastion at the south. The complete bastion survives; the others are largely gone, though their shapes are legible in the angular outline of a later masonry enclosing wall. Within this defended area, and apparently unconnected to the seventeenth-century works, stands a prehistoric standing stone, 3.43 metres tall, its base packed with smaller stones and a leaning slab propped against its face. A visitor in 1839, Chatterton, noted that the stone was "very easily shaken," which may be why the supporting slab was placed there at some point afterwards.