Congratulations on getting the colouring working, photoshop can be a bit bewildering when you're not used to it but it's a brilliantly effective tool.
I used the rubber stamp tool with a soft brush and did it carefully "by hand", so to speak. Erasers are great when you have layers underneath that you want to show through, but with a single layer image like this it would mean you'd lose the grain of the paper, and the texture of the line at the edges where you've erased. Because there's a regularity to the lines you're getting rid of, using any kind of colour replace doesn't really work because you'll still be able to see the overall pattern.
The way I did it (it's kinda boring) create a new layer (ctrl+shift+n) so that you can correct any mistakes easily, then select the rubber stamp tool (s). Define the point it will work from by holding down alt and clicking. After you've let go of alt, wherever you click next will copy the first point you've clicked, if you work the brush to the right it will copy what's to the right of that point, move to the left it will copy to the left etc.
Pick out an area that has about the same tone/pattern as the area that's been obscured by the blue/red line (so where the line crosses the white paper, i selected an area of the paper just underneath and brushed it along, where it crosses pencil I selected an area of pencil with a similar kind of pattern). I used a soft-edged brush and tried to blend it carefully.
Bear in mind that you can use any brush type and opacity when you're doing this, I tend to use quite soft-edged ones unless I'm working on something with an obvious repeating pattern. Also, there's a tickbox up the top that lets you choose to either copy from all layers or only the one you're currently working on.
Once you're done and happy with the results, merge the layer down into the original pencil one (ctrl+e merges the currently selected layer with the one beneath it OR merges all layers you have linked together).
One fun thing to play with once you have the drawing is different ways of colouring it. Here's one of the things I'm working on at the moment - it's basically a line drawing of a girl coloured with various different scanned textures/photos/paintings/parts done in photoshop, with the colours and layering heavily reworked:
A girl contemplates romance/suicide in 2005 (In the end she will become a "Suicide Girl")
here's the original drawing:
Sorry for the long post, basically what i meant to say was "(s), alt-click where you want to copy from, then click where you want to copy".