We consider the motion of a classical colored spinless particle under the influence of an external Yang-Mills potential A on a compact manifold with boundary of dimension ≥3. We show that under suitable convexity assumptions, we can recover the potential A, up to gauge transformations, from the lens data of the system, namely, scattering data plus travel times between boundary points.