Here's the deal...
find out what the RMS power rating of the infinity speakers are. RMS is how much power the speaker needs to run efficiently and sound clean. Chances are, the stock deck doesn't have enough power. Aftermarket speakers require more power than stock's do mostly because they are made much better, have a composite cone, thicker coils, a heavier magnet, etc. And, because they require more power than what the stock stereo can put out... they could sound even worse than stock at moderate volumes. At low-mid volume, all is cool... but crank up the vol and then you're forcing the speakers to run a marathon without the proper nourishment. Power is food for the speaker... you can't feed a gorilla on a baby's diet.
Even though the stock stereo claims to be 160watts, it really isnt. It can push 160watts max power thru 4 channels for 12 seconds with 80% distortion - and sounds like crap at that level. The real question is...what is the RMS power rating of the stock stereo? Go to a store and ask to see what a 160watt (RMS) amplifier looks like...notice how much bigger it is than the stock stereo. Am i making any sense at all here???
I can't tell you how many times customers used to come in and insist they wanted nice speakers installed to their factory deck. I'd explain why nice aftermarket speakers require more than what the factory deck could provide, they wouldn't believe me... I'd install it for them, and then they'd complain it didn't sound as good as they thought and it wasn't worth the money.
IMHO, and from years of experience - if you spend the money on quality speakers, then you should buy an amplifier that will do them justice. But if you're a person who does not turn up their music, then maybe you've got nothing to worry about. But seriously... who doesn't turn up their music when their cruising on the highway with sunroof and windows open? otherwise, you wouldn't hear anything except wind noise.