In the 1850 census, there is a blacksmith named Julius C. Johnson, 52, born Connecticut, in Middlebury, Addison County, VT. His wife is Harriet, 48, born Vermont; his son is William, age 20, born New York, a saddler. There are other Johnsons nearby, including Edward G., a merchant. They look to be in the commercial district, if there was one, of Middlebury.
In 1860, Julius and Harriet A. are still in Middlebury.

From http://kinnexions.com/kinnexions/johnson/rr01/rr01_272.htm
874. Julius Christopher Johnson (Christopher7, Hamlin6, Joseph5, Joseph4, Isaac3, Isaac2, John1). Born on 19 Jun 1790. Died on 21 Jul 1861.

He first married Anna Rockwell, on 25 Oct 1820 in Leicester, Vt.

Of Salisbury.

They had the following children:
i. Mary. Born in 1821.
ii. Julius A.. Born N.Y. City, on 26 Apr 1826.
iii. William F.. Born N.Y. City, in 1830.

He second married Harriet A. Severance, daughter of Moses Severance & Kezia Andrews, on 16 Nov 1831 in Salisbury, Vt. Born on 1 Jan 1800. Died on 22 May 1882.

She married 2nd Chester Elmer.

(If that William F. is accurate, then this is not a connection)