Whether or not that last panel is a fair point will vary by character.

Some characters know they can't tell sometimes, because they're not sure.

Other characters, like Ellen, think they know, and don't question their conclusions much. This does not necessarily mean they actually can tell, but they might!