Experience is a lot of it- you don't shop for an 
item for several years without getting an idea of 
what's out there and what it costs.  But if you 
are new it's not such a big deal- you do what you 
would do for anything else- you shop around.  
Compare price ranges for cels of characters you 
like, if there are a few cels around you should 
be able to get an idea if it is going to be high 
end.  How common a character is will play a part 
too.  FY TV cels may be in demand, but a Miaka 
cel is going to be a lot less expensive than a 
Noriko or Hotohori.  Also, if you look through 
the forum, you'll see what characters are more 
popular and more in demand, and so are more 
expensive.  If you see "I want a Van with wings!" 
10 times, then you'll know that he's probably 
pretty rare and expensive.
Last thing to keep in mind- prices CHANGE.  There 
are peaks and valleys, and if something is too 
expensive, sometimes patience will reward you 
with a lower price later on.  Keep in mind it's 
probably hard to put an exact dollar value on a 
cel- think in terms of ranges.  If you pay an 
extra $20 on a $200 cel, that's really only 10%, 
and is probably not a big deal.  The nice thing 
about cels is they tend to retain a lot more 
resale value than most hobbies- if you buy a 
video game it's almost impossible to sell it 
later for even 50% of what you paid, whereas with 
cels you can often get what you paid back.