George Cabell, Sr. (1766–1823) was the first in a family of practicing physicians to earn an official medical degree. After studies at Hampden-Sydney Academy, he completed his medical training at the University of Pennsylvania. Dr. Cabell then returned to the Lynchburg area, where the Cabell family owned significant acreage.

As the site to build his new home, Dr. Cabell chose a promontory where the James River is joined by Blackwater Creek and in 1806 began construction of a stately Federal-style house. In 1815 Point of Honor was completed. The “Point” in the property’s name refers to the land overlooking the James, but the derivation of “Honor” remains a mystery; there is, for instance, no record of a duel being fought on those grounds.

The property was successively home to a number of families famous in Lynchburg’s history: the Daniels, the Langhornes, and the Owens. In 1928 it was purchased by Lynchburg banker James Gilliam, Sr., and in 1939 he donated it to the city. Over the following decades, the house was neglected and fell into decay. In 1970 Katherine Garland Diggs left a bequest for its restoration. Through the cooperative efforts of the Lynchburg Historical Foundation, The Garden Club of Virginia, the Katharine Garland Diggs Trust, and the City of Lynchburg, Point of Honor was restored and opened to the public in 1977. It is now on the National Register of Historic Places.

While the house was being restored, The Garden Club of Virginia was approached about restoring the grounds in front of the mansion. Meade Palmer, landscape architect for The Garden Club of Virginia, drew up a plan that featured a long circular cobblestone drive leading to the house from the front gate. The cobblestones were edged with large, flat, rectangular flagstones, which would have been appropriate in the early period of the house by providing a smooth surface for carriage wheels while the cobblestone center offered traction to the horses. The walkway to the parking lot was laid of stones matching those in the drive. Large English boxwood were planted along the steps to the front porch and in the circle.

Subsequent gifts from the Garden Club of Virginia included a visitor center reception terrace and additional plantings according to a plan by Rudy J. Favretti, then landscape architect for the GCV, as well as trees and shrubs in the parking lot. When the Bertha Green Webster Carriage House was built in the 1990s, and a kitchen recreated, the GCV sponsored landscaping around the Carriage House and, with the advice of orchardist Tom Burford, planted a demonstration heirloom apple orchard. In 2006, GCV landscape architect Will Rieley designed a stone walk from the Carriage House to the back grounds.