Vertices are points in space: They define the structure of other sub-objects (edges and polygons) that make up the poly object. When you move or edit vertices, the connected geometry is affected as well. Vertices can also exist independently; such isolated vertices can be used to construct other geometry but are otherwise invisible when rendering.
An edge is a line connecting two vertices that forms the side of a polygon. An edge can't be shared by more than two polygons. Also, the normals of the two polygons should be adjacent. If they aren't, you wind up with two edges that share vertices.
A border is a linear section of a mesh that can generally be described as the edge of a hole. This is usually a sequence of edges with polygons on only one side. For example, a doesn't have a border, but the object has several: on the lid, on the body, on the spout, and two on the handle. If you create a cylinder and then delete an end polygon, the adjacent row of edges forms a border.