Computer VisionProject 2 of 5
Eye Gaze Tracking
Real-time eye gaze estimation that can move your cursor. My minor project, built by a team of 3 and trained on the GazeCapture dataset.
- Kind
- Minor project
- When
- 3rd year
- With
- Team of 3
- My part
- Most of the work
01 / What it does
It watches a face through a camera in real time and works out where the person is looking. It can even move the mouse cursor to that spot.
02 / How it works
MediaPipe finds the face and the eyes. A hybrid model then reads the face, both eyes, the head pose and some metadata: ResNet50 looks at the face and EfficientNetB3 looks at the eyes.
03 / The results
It was trained on the GazeCapture dataset. It reaches 80%+ accuracy at 30 FPS, with a mean error of 5.3°.
04 / Where it came from
It was our minor project in the third year of college. We were a team of 3 and I did most of the work.
Code
# src/model/hybrid_model.py
def create_hybrid_model(
input_shape=(224, 224, 3),
metadata_shape=15,
dropout_rate=0.4,
dense_units=(512, 256),
include_pose_estimation=True,
use_attention=True,
) -> Model:
"""ResNet50 for facial features, EfficientNetB3 (GazeNet)
for eye features and pose estimation."""
face_input = Input(shape=input_shape, name='face_input')
left_eye_input = Input(shape=input_shape, name='left_eye_input')
right_eye_input = Input(shape=input_shape, name='right_eye_input')
metadata_input = Input(shape=(metadata_shape,), name='metadata_input')
resnet_model = ResNet50(weights='imagenet', include_top=False,
pooling='avg', input_shape=input_shape)
face_features_resnet = resnet_model(face_input)
# ...