Mortgage-Any disadvantages of going with alternative lenders? (i.e. not a Big 5 bank)

I'm sure this question has been asked before, but I did a quick search and wasn't able to find much.

Is there anything bad about using an alternative lender for a mortgage? I'm assuming when you go through a mortgage broker, they send out your application to both alternative lenders and the big 5 banks and will present the various options to you. If the alternative lenders have lower rates, is there anything bad about going with them?

I once heard that it doesn't look as good on your credit report if you have a non-bank lender, but on the flip side, if you are saving money and it doesn't really make that much of a difference, then it shouldn't matter?

Any insights would be appreciated!

Thanks,
Kevin.