Definitely staying in your niche is more important than just having a big number to boost ego. Haters usually come to everyone at some point - those are the people who don't really do anything themselves, so they attack those that do.