I guess what is important to keep track of in all that, is the name "theory of relativity".
As in, it's all about different perceptions, relative to where the perception was made. So to person X, outside of the range of gravitational pull of a black hole, it might take 4 hours to see another person Y come out of the black hole, while to the person Y it might seem like it only took 4 minutes.
The same way, when the universe was young, and alot more close together, there was more intense gravity between holes, planets and stars on the whole (since gravity becomes weaker at greater distances). So time indeed did pass faster. Gravity was higher, so time-space was warped to a greater degree.
At least, that's how I understand it
