From Cosmological Structures to the Milky Way