Retiredrookie (great name), you probably experienced a "loss" of points because of a vote(s) you received that lowered your average rating. As you only have 18 votes so far, a single vote of "1" would actually have a decent impact on your rating (which is now 3.2).
When your rating moves, so does your total points. Why? Your total points is calculated as follows:
Rating X Activity Points = Total Points
On the other hand, just one "5" rating would have the opposite effect.
Just a little turbulence that everyone experiences at the beginning.
Regards,
George
p.s. I don't drink coffee but go crazy when the wife asks me to pick her up one from Timmy's and the line up in 10 minutes long! Don't you people like coffee from any other place? :-)