Usually The first thing people notice is your pronunciation. They’ll judge how good/bad you are in the language based on that. Sometimes, though, you won’t even get that far. They’ll look at you and assume you don’t speak Chinese.
Priming is the effect where being exposed to a stimulus (e.g. a situation, a conversation, etc.) unconsciously influences later responses to later stimulus. In the language context, priming occurs whenever you talk to someone for the first time. When you are using Chinese to talk to someone you’ve just met, you are priming them to adjust to what they believe your Chinese level is. If they think you sound fluent, then they will speak normally (or close to normal); if they have trouble understanding you, they will speak slowly and try to use “simple words” (which sometimes makes the situation worse). If you’ve primed them to believe your Chinese is bad, they’ll believe your Chinese is bad.
In some cases, your appearance is what primes the other person. Without having spoken, they unconsciously go into what I call “listening-to-foreigner mode”. In listening-to-foreigner mode, they don’t understand anything you are saying. They try to guess on your intent with every syllable that comes out of your mouth. This dual-processing takes a toll: they can’t actually listening to what you are saying, because they don’t have time to listen and process your words. That’s why sometimes even when you’re right, as far as they’re concerned, you’re wrong.
It is really frustrating when, at a restaurant, for example, you ask a waiter for something in perfect Chinese, only to have the waiter look at your Chinese friend in confusion. It’s not always your fault (well… sometimes it is ^_^;;). They want to avoid a misunderstanding and think it’s easier to ask your friend sitting next to you.
How to be understood
How you start a conversation can determine how well people will understand your Chinese. If you get the basics down, they will be primed for good Chinese and you’ll encounter this situation less.