This paper presents an approach for detecting the damaged buildings due to earthquake using the watershed segmentation of the post-event aerial images. The approach utilizes the relationship between the buildings and their cast shadows. It is based on an idea that if a building is damaged, it will not produce shadows. The cast shadows of the buildings are detected through an immersion-based watershed segmentation. The boundaries of the buildings are available and stored in a GIS as vector polygons. The vector-building boundaries are used to match the shadow casting edges of the buildings with their corresponding shadows and to perform assessments on a building-specific manner. For each building, a final decision on the damage condition is taken, based on the assessments carried out for that building only. The approach was implemented in Golcuk, one of the urban areas most strongly hit by the 1999 Izmit, Turkey earthquake. To implement the approach, a system called the Building-Based Earthquake Damage Assessment System was developed in MATLAB. Of the 284 buildings processed and analysed, 229 were correctly labelled as damaged and undamaged, providing an overall accuracy of 80.63%.