This paper uses GeoEye-1 imagery and airborne lidar (Light Detection and Ranging) data to map buildings and their rubble in Port-au-Prince caused by the Haiti earthquake on 12 January 2010. This is achieved by performing an object-based one-class-at-a-time land cover classification of the image and lidar data using spectral, textural and height information. Classification accuracy is about 87 percent overall, and approximately 80 percent for buildings and rubble. Comparison of manually-selected 200 actual damaged buildings within an area of two sq. km in the city center shows an accuracy of over 90 percent for building and rubble mapping. 3D building models for approximately 55,000 buildings covering an area of 30 sq. km over Port-au-Prince were generated. It is found that most of the damage is to the concrete and masonry structures in the well planned areas of the city and very little damage to the shelters and the temporary type houses with metal sheet roofs. The study demonstrates that fusing optical imagery and lidar data can effectively map the nature, severity, extent and damage patterns caused by earthquakes in densely populated urban areas like Port-au-Prince.