Once you get a dark area in a scan to be pure black or a light area to be pure white, other parts of your image are most likely clipping severely. You’re probably not taking it this far, it usually ends up being overly contrasty.
If you’re being more gentle with it you may lose paper texture, noise or detail in some parts of the shadows and highlights, but not everywhere.
In Photoshop’s levels tool you may hold the ALT key pressed while using the sliders to see exactly where the channels are clipping. I found this to be really helpful. Unfortunately Gimp doesn’t have this kind of functionality built in.
Setting shadows and highlights manually so that they’re just about to clip is good practice. You won’t get the blackest blacks and whitest whites this way, but you’re not losing any information either. It keeps the fidelity of your image perfectly intact.
To correct the color manually you could try setting the levels of the individual channels the same way. If the darkest and brightest parts of your image are black and white you should end up with fairly neutral colored shadows and highlights.
If your image is still not contrasty enough at this point, you could use the curves tool to carefully bend the shadows and highlights a little further into saturation without hard clipping anything or messing up the mid tones.
Generally I wouldn’t expect a scan to have perfect blacks and whites. Paper and ink tones are always going to be a little off and uneven. Those are the limitations of the medium. I try to accept them.