Computer vision has long been used for human action/activity recognition and unusual event detection besides video surveillance. Numerous attempts have been made to recognize human activities like sitting, standing, laying down, jumping, walking, etc. Similarly, abnormal event detection has also been an active research area in computer vision. Inspired from these attempts the current research proposes a system for non-invasive detection of patient's discomfort in wards by recognizing different visual clues associated with a patient's pain or other types of discomfort. We use a blob-based method for extracting moving parts of the patient's body and analyze the extracted blobs using decision rules to identify which movement of the body corresponds to discomfort. We created a new data set for evaluating the system and developed a ground truth for images against which we measured the performance of the proposed method. Using this dataset we achieved an overall average accuracy of 90%. This system is equally applicable in homes for taking care of those elderly who prefer to live independently.