This letter proposes an approach for building detection from single very high resolution optical satellite images by fusing the knowledge of shadow and urban area information. One of the main contributions of this work is in the integration of urban area information: unlike previous studies, we use such information to substantially revise and improve the initial shadow mask. Additionally, we present an effective way to discriminate dark regions from cast shadows, a task that has continuously been reported to be very difficult. In this letter, we benefit from graph cuts to produce a comprehensive solution for automatic building detection: a flexible multilabel partitioning procedure is proposed, in which the number of optimized classes is automatically selected according to the contents of a scene of interest. The results of the evaluation of 14 demanding test patches confirm the technical merit of the proposed approach, as well as its superiority over three recently developed state-of-the-art methods.