Your mental image library of objects in nature tends to be 'generalized' so your brain can recall it easier. Until you have trained your mind to recall and 'see' things properly, you will draw with what you think they look like.
Basically, why you use reference is so you can draw "what it really is" (what you need to do) vs "what you think it is" (what you are doing right now). Once you practice enough with reference, just start taking reference away.
Putting it into another context, if you are learning a new language and you don't know how to say a specific thought/word, you don't attempt to formulate the word by making it up with what little knowledge you have of the language, right? You save alot more time and energy to just go look the word up.