You can also use the delete key or the contextual menu (right click).
You can also delete control points by selecting an area and pressing the delete key.