This algorithm works in any closed polygon as long as the polygon's sides don't cross. Triangle, pentagon, square, even a very curvy piecewise-linear rubber band that doesn't cross itself.
1) Define your polygon as a directed group of vectors. By this it is meant that every side of the polygon is described by a vector that goes from vertex an to vertex an+1. The vectors are so directed so that the head of one touches the tail of the next until the last vector touches the tail of the first.
2) Select the point to test inside or outside of the polygon.
3) For each vector Vn along the perimeter of the polygon find vector Dn that starts on the test point and ends at the tail of Vn. Calculate the vector Cn defined as DnXVn/DN*VN (X indicates cross product; * indicates dot product). Call the magnitude of Cn by the name Mn.
4) Add all Mn and call this quantity K.
5) If K is zero, the point is outside the polygon.
6) If K is not zero, the point is inside the polygon.
Theoretically, a point lying ON the edge of the polygon will produce an undefined result.
The geometrical meaning of K is the total angle that the flea sitting on our test point "saw" the ant walking at the edge of the polygon walk to the left minus the angle walked to the right. In a closed circuit, the ant ends where it started.
Outside of the polygon, regardless of location, the answer is zero.
Inside of the polygon, regardless of location, the answer is "one time around the point".