The lot on which the Hall-Haskell-House sits was owned by a succession of people who played notable roles in Ipswich’s early history. Thomas Perin bought the property in 1691 and later sold the property and house to Henry Wise, son of the famous Reverend John Wise in 1733.

The land then went through a series of owners and at some point the original house was removed. No record has-been found as to what happened to the original dwelling. Mary Hall acquired the property and present house in 1819. She ran a general store downstairs to generate income and lived above the store. In 1825 the Haskells bought the property and later sold it to Abraham Caldwell, a well known builder and cabinet maker. He changed the roof line of the house from gambrel to its present pitch. The house was purchased by-John Heard, a prominent citizen in Ipswich in 1864.

The house had been slated for demolition back in the early 80’s and a group of people formed an organization to save the so called “Little Red House” as it was so often-identified by that name. The group also became part of a standing committee of the Town of Ipswich since it is a town owned piece of property.