Just as something being "fishy" describes something suspicious, to "smell a rat" is to suspect trickery or treachery.
Trickery is as in fraud. Treachery is as in betrayal. Also, in criminal culture, "a rat" is a police informant. This term has spread into popular culture; "ratting" or "ratting out" is to inform on someone and betray that person to an authority of some kind, whether lawful (police) or unlawful (organized crime).
Example: "I considered buying that car at nine thousand dollars, but I smelled a rat. The car should have been worth much more. I looked deeper and discovered that the engine will need to be replaced. That'll add thousands of dollars to the price. I want a good car for $9000, but I want one in good condition."
When a dog bred to hunt rats acts like it is suspicious, it probably smells a rat. This is the origin of the expression.