The colonial site first belonged to Richard Ingersoll (died 1644). In 1670 his wife left the land to her second husband upon her death, and thence to his daughter Sarah, whose second husband Joseph Holten deeded the lot (and perhaps also the house) to his son Benjamin Holten. Benjamin died in 1689, and his will records both the land and a house. Given this background, it is believed that Benjamin Holten built the house circa 1670 in a typical “one-room” layout. The house has been extended six times since until 1832.

The house is historically interesting as the home of Sarah Holten, who in 1692 gave testimony against Rebecca Nurse which led to her death in the Salem Witch Trials. During the American Revolutionary War, it was the home of Judge Samuel Holten, who served in the Continental Congress, including as its president pro tempore, was a signer of the Articles of Confederation, and who was an early member of the United States House of Representatives (March 4, 1793-March 3, 1795).