Black Baltimore, 1820-1870