This is a tough one to answer. I think that if I had an elf and a dwarf in the same party at some point I might have a better take on this, but I haven't, so here's what I've found.
When the terrain is pretty much open (no difficult terrain or blocking squares), then the speed of the characters typically doesn't make a big impact on encounters. Everyone has a speed of 5 or 6 and there isn't a major issue.
Provide some blocking terrain that the the characters must move around and the difference between a speed of 5 and 6 shows up slightly, but not too much. And even difficult terrain isn't usually that big of a deal.
The only time I see it make a noticeable difference is along the boundaries of the difficult terrain. If two characters (speed 5 and 6) start 4 squares away from the terrain, the character with speed 5 gets up to the edge, but can't proceed as they don't have 2 squares of movement left to enter. The character with speed 6 gets 1 square into the difficult terrain. But, once they are in, the extra cost to move through reduces the speed 6 character to an effective speed of 3 and the speed 5 character to 2. So there is still only a 1 square difference.
But, as I said, I haven't seen any encounters where I have a speed 5 and a speed 7 character at the same time. So, I haven't had much "real-world" experience in analyzing this situation. In my current campaign, I have one character playing a Slayer and they have a stance that gives them +2 to their speed while charging (speed 7). But, it is only while charging, so it isn't an "always on" thing.
Overall, I'd say that the difference between speed 5 and 6 is noticeable, but not major. The difference between speed 5 and 7 is probably a lot more noticeable, but I don't know for sure.