We present a feature-based method to classify salient points as belonging to objects in the face or background classes. We use SURF local descriptors (speeded up robust features) to generate feature vectors and use SVMs (support vector machines) as classifiers. Our system consists of a two-layer hierarchy of SVMs classifiers. On the first layer, a single classifier checks whether feature vectors are from face images or not. On the second layer, component labeling is operated using each component classifier of eye, mouth, and nose. This approach has the advantage about operating time because windows scanning procedure is not needed. Finally, this system performs the procedure to apply geometrical constraints to labeled descriptors. We show experimentally the efficiency of our approach.