The Mathematical Structure of Classical and Relativistic Physics