A hierarchically organized visual place memory enables a robot to associate with its respective knowledge efficiently. In this paper, we consider how this organization can be done by the robot on its own throughout its operation and introduce an approach that is based on the agglomerative method SLINK. The hierarchy is obtained from a single link cluster analysis that is carried out based on similarity in the appearance space. As such, the robot can incrementally incorporate the knowledge of places into its visual place memory over the long term. The resulting place memory has an order-invariant hierarchy that enables both storage and construction efficiency. Experimental results obtained under the guided operation of the robot demonstrate that the robot is able to organize its place knowledge and relate to it efficiently. This is followed by experimental results under autonomous operation in which the robot evolves its visual place memory completely on its own.