I don't have all the OS's in front of me right now, but with Windows XP, right-click on My Network Places, and choose Properties. Find the network connection you're using, right-click on it, and choose Properties. Then choose Internet Protocol (TCP/IP), and Properties. Choose Use the following DNS server addresses, don't put in any addresses, and press Advanced. Choose the DNS tab. Then press Add, and add the DNS servers you wish to use in the order you wish to use them.
I would suggest that someone who plays well with others will enter their ISP's DNS servers they'd get by DHCP as the first two, then some public servers after that. Note that you can put in a huge list if you want. Depending on your system, only 4 will show in the window, but there is a scroll bar there. I have 10 servers in my list right now. I'm reasonably sure that I'll never need that many, but you could enter even more if that's what you want.
And no third-party software needed.
As I recall, Windows 9x/ME only allows three or four servers entered manually, but that should be enough for most folks.
And this works even if you have a router. You don't need to make any changes to your router settings.