A concave polygon is defined as a polygon with one or more interior angles greater than 180°. It looks sort of like a vertex has been 'pushed in' towards the inside of the polygon. Note that a triangle (3-gon) can never be concave.
A concave polygon is the opposite of a convex polygon. See Convex Polygon.
In the figure above, drag any of the vertices around with the mouse. Take note of what it takes to make the polygon either convex or concave. Also change the number of sides.
Some of the diagonals of a concave polygon will lie outside the polygon. In the figure on the right, the diagonal at the top of the polygon is outside the polygon's interior space. (In a convex polygon, all diagonals will lie inside the polygon).
The area of a concave polygon can be found by treating it as any other irregular polygon. See Area of an Irregular Polygon
Regular Polygons are never concave by definition. See Regular Polygon Definition.