Construction began in 1911 but completion of the original design did not occur until the early 1930s. The institution was planned as a farm colony where by patients were put to work raising animals and growing food. Superintendent Charles S. Little told the New York Times, In order to make this plan a success, it is necessary to begin to train the feeble minded when they are children. The feeble minded, if taken at an early age can be trained to do things better than if the education of which they are capable is postponed until the less pliable years. The site was named for William Pryor Letchworth, who served on the New York State Board of Charities from 1873 to 1896. Letchworth Village was one of the largest and most progressive facilities for the mentally retarded in the United States. Situated on 2000 acres of farmland with the Towns of Haverstraw and Stony Point. It was designed as a self-supporting community comprised of 130 field stone buildings.
The facility closed on March 31, 1996, but administrative offices remained open until 2002. The campus sprawls across the boundaries of 2 towns. Some of the buildings located within one of the towns have been adaptively-reused, while much of the other section is neglected.