A Window into Zeta and Modular Physics