After 56 years of public life, Colonel Weber then retired from active political and business activities and lived quietly until his death at the age of 84 on December 18th, 1926. He died in his home on Abbott Road in the newly formed City of Lackawanna. He was survived by his wife, Alice, and five children Mrs. Howard O. Cobb, Mrs. Godfrey L. Carden, Mrs. Frank M. Brinker, Mrs. George S. Goodrich and Mrs. Edward Meinel. Colonel Weber's family plot is located in Forest Lawn Cemetery, Buffalo. His marker is inscribed with 'Colonel 89th U.S. Infantry'.
Upon the death of the Colonel, Representative Clarence MacGregor made the following statement from the floor of the House, Colonel Weber left behind him a record of a life of devotion to the public good and a memory that will be cherished by all who had the good fortune to know him.
The Colonel was also affiliated with many local organizations including the Grand Army of the Republic, Post Number 9 and the Loyal Legion. From 1864 he was a member of the Washington Lodge of Masons in Buffalo and at the time of his death in 1926, had been the oldest living mason in the country. He was a former president and treasurer of the Ellicott Club for Men and a life member of the Young Men's Association. During his retirement he wrote his autobiography in 1924, which was a widely acclaimed success. He also found time to write many papers on canal improvements and immigration problems which were widely published.
There are many streets in Lackawanna named for Colonel Weber including: Weber, Shamokin, Cobb, Carden, Brinker, Goodrich and Josie (who was Colonel Weber's maid). Also, in 1944, the John B. Weber Post of the Veterans of Foreign Wars No. 898 was established.