Yes, gravity bends light just like it would bend the path of an object passing by at high speed. If one galaxy is lined up behind another in just the right way, the intermediate one acts as a lens, and we see the light of the far one as a ring around the intermediate one. Of course, this kind of alignment doesn't happen very regularly, which is why "clean" examples are rare. Gravitational lensing is more often seen as a series of complicated arcs like this:
