So therefore I suggest the following:

If you have 20 ratings, leave the 3-5 (other number can be inserted) lowest ratings out for your average. Then atleast people can see you have a good average and it was just 1 or more people that just gave stupid scores.
(I suggest that 10-20% lowest are ...