Edward Ayler was born in Virginia in 1855 to enslaved parents Morgan and Matilda Ayler. After the war Ayler and his family were brought north to Portsmouth by Joseph Macomber, a Quaker. There, Morgan Ayler went from working on Macomber’s farm to leasing the land to owning his own farm, which he ran with Edward. The Aylers converted to Quakerism and were active and well-regarded members of the community. After his father's death, Edward took over the farm and his high-quality produce won a number of local awards. He remained at the farm for the rest of his life, passing away in 1935. His obituary, published in the Newport Mercury, describes him as “one of the oldest and best-known citizens of Portsmouth.”