We show how effect algebras arise in physics and how they can be used to tie together the observables, states and symmetries employed in the study of physical systems. We introduce and study the unifying notion of an effect-observable-state-symmetry-system (EOSS-system) and give both classical and quantum-mechanical examples of EOSS-systems.