Previously we have investigated the medial layer graph $\mathcal{G}$ for a finite, self-dual, regular or chiral abstract 4-polytope $\mathcal{P}$. Here we study the Cayley graph $\mathcal{C}$ on a natural group generated by polarities of $\mathcal{P}$, show that $\mathcal{C}$ covers $\mathcal{G}$ in a readily computable way and construct $\mathcal{C}$ as a voltage graph over $\mathcal{G}$. We then examine such symmetric graphs for several interesting families of polytopes of type {p,q,p}, p = 3, 4, 5.