Sclera Segmentation Benchmarking

Introduction

This page is dedicated to sclera segmentation benchmarking results on the MOBIUS Dataset. The initial results (also included below) were presented at the International Joint Conference on Biometrics (IJCB) 2020, organised by IEEE and IAPR, in the conference paper SSBC 2020: Sclera Segmentation and Benchmarking Competition in the Mobile Environment. The results on this page are presented in a way that they can be easily implemented into your own publications and work as a baseline for comparison. We also provide the necessary code (with detailed instructions) and results from existing solutions to generate these same graphs and tables and include your own segmentation models in them.

If you would like to have your results included on this page, please submit your segmentation masks to [email protected]. When submitting your solution, please follow the Result submission guidelines from the SSBC 2020 instructions. Please note that the pickle files generated by compute.py will not be accepted as valid submissions.

Optionally you can include a single reference for your approach, which will be included in the table next to your submission and will allow future users of these results to properly cite your work. Please submit your reference in both plaintext format and BibTeX format (see the References section for examples).

Code

To obtain, set up, and run the code for the generation of the quantative results and graphs, please see the Code page. The pickle files of the submitted models, which are required to use the code for comparison with existing solutions, are available here.

Results

The current highest-ranking solution is UNet-P with an F1-score of 0.868, as we can see in Table 1. Figure 1 shows the models ranked by their F1-scores, while Figure 2 shows the overall performance of the submitted models in the form of precision-recall curves.

**Table 1:** Comparative assessment on the MOBIUS dataset. The results are ordered according to the achieved mean F1-scores. The F1, Precision, Recall, and IoU scores were computed from the submitted binary masks. The optimal F1-score on the precision-recall curve (F1^opt) and AUC values were calculated from the probabilistic segmentation predictions. The results are reported as μ ± σ, computed over a 5-fold split of the data.
Segmentation model	From binary masks				From probabilistic predictions
Segmentation model	F1	Precision	Recall	IoU	F1^opt	AUC
UNet-P SSBC_2020	0.868 ± 0.003	0.909 ± 0.004	0.831 ± 0.003	0.868 ± 0.003	0.870 ± 0.003	0.930 ± 0.003
FCN8 SSBC_2020	0.854 ± 0.004	0.820 ± 0.004	0.890 ± 0.004	0.853 ± 0.003	0.865 ± 0.004	0.936 ± 0.003
RGB-SS-Eye-MS SSBC_2020, Eye-MMS	0.836 ± 0.004	0.917 ± 0.002	0.769 ± 0.005	0.841 ± 0.003	0.842 ± 0.004	0.872 ± 0.003
Y-SS-Eye-MS SSBC_2020, Eye-MMS	0.823 ± 0.005	0.930 ± 0.004	0.738 ± 0.006	0.830 ± 0.004	0.836 ± 0.005	0.868 ± 0.005
SaSSNet SSBC_2020, SaSSNet	0.821 ± 0.004	0.885 ± 0.004	0.765 ± 0.006	0.827 ± 0.003	0.818 ± 0.004	0.893 ± 0.004
Multi-Deeplab SSBC_2020, MultiDeeplab	0.806 ± 0.005	0.915 ± 0.001	0.719 ± 0.008	0.816 ± 0.004	0.821 ± 0.004	0.896 ± 0.004
CGANs2020CL SSBC_2020	0.803 ± 0.006	0.771 ± 0.009	0.838 ± 0.003	0.810 ± 0.005	0.803 ± 0.006	0.828 ± 0.006
ScleraU-Net SSBC_2020	0.795 ± 0.005	0.941 ± 0.002	0.689 ± 0.007	0.809 ± 0.003	0.805 ± 0.004	0.848 ± 0.003
AB Sclera Net SSBC_2020	0.786 ± 0.008	0.785 ± 0.016	0.787 ± 0.007	0.797 ± 0.006	0.793 ± 0.008	0.878 ± 0.005
Color RITNet SSBC_2020	0.774 ± 0.007	0.898 ± 0.006	0.680 ± 0.011	0.791 ± 0.005	0.783 ± 0.006	0.793 ± 0.012
ScleraMaskRCNN SSBC_2020	0.763 ± 0.011	0.828 ± 0.008	0.707 ± 0.015	0.782 ± 0.008	N/A*	N/A*
Multi-FCN SSBC_2020, MultiFCN	0.760 ± 0.007	0.941 ± 0.003	0.638 ± 0.009	0.782 ± 0.005	0.716 ± 0.006	0.786 ± 0.007
Mask2020CL SSBC_2020, Mask2020CL	0.717 ± 0.010	0.833 ± 0.013	0.629 ± 0.007	0.749 ± 0.006	N/A*	N/A*
MU-Net SSBC_2020	0.651 ± 0.013	0.638 ± 0.012	0.665 ± 0.014	0.698 ± 0.008	0.659 ± 0.013	0.554 ± 0.014
SSIP SSBC_2020, SSIP	0.595 ± 0.007	0.762 ± 0.009	0.489 ± 0.008	0.672 ± 0.005	0.596 ± 0.007	0.524 ± 0.007
S-Net SSBC_2020	0.462 ± 0.008	0.348 ± 0.008	0.687 ± 0.019	0.552 ± 0.005	0.598 ± 0.010	0.662 ± 0.011
* F1^opt and AUC not reported due to issues with the submitted probabilistic results.

\usepackage{multirow}
\usepackage{multicol}
\usepackage{booktabs}

\begin{table}[!t!]
\caption{Caption.}
\label{tab:results}
\centering
\begin{tabular}{lcccc cc}
\hline
\hline
\multirow{2}{*}{Segmentation Model} & \multicolumn{4}{c}{From binary masks} & \multicolumn{2}{c}{From probabilistic predictions}\\\cmidrule(lr){2-5} \cmidrule(lr){6-7}
& $F_1$ & Precision & Recall & IoU & $F_1^{opt}$ & AUC \\
\hline
UNet-P \cite{ssbc20}                          & $0.868 \pm 0.003$ & $0.909 \pm 0.004$ & $0.831 \pm 0.003$ & $0.868 \pm 0.003$ & $0.870 \pm 0.003$ & $0.930 \pm 0.003$\\
FCN8 \cite{ssbc20}                            & $0.854 \pm 0.004$ & $0.820 \pm 0.004$ & $0.890 \pm 0.004$ & $0.853 \pm 0.003$ & $0.865 \pm 0.004$ & $0.936 \pm 0.003$\\
RGB-SS-Eye-MS \cite{ssbc20,boutros2019eyemms} & $0.836 \pm 0.004$ & $0.917 \pm 0.002$ & $0.769 \pm 0.005$ & $0.841 \pm 0.003$ & $0.842 \pm 0.004$ & $0.872 \pm 0.003$\\
Y-SS-Eye-MS \cite{ssbc20,boutros2019eyemms}   & $0.823 \pm 0.005$ & $0.930 \pm 0.004$ & $0.738 \pm 0.006$ & $0.830 \pm 0.004$ & $0.836 \pm 0.005$ & $0.868 \pm 0.005$\\
SaSSNet \cite{ssbc20,wang2019sclerasegnet}    & $0.821 \pm 0.004$ & $0.885 \pm 0.004$ & $0.765 \pm 0.006$ & $0.827 \pm 0.003$ & $0.818 \pm 0.004$ & $0.893 \pm 0.004$\\
Multi-Deeplab \cite{ssbc20,osorio2018visible} & $0.806 \pm 0.005$ & $0.915 \pm 0.001$ & $0.719 \pm 0.008$ & $0.816 \pm 0.004$ & $0.821 \pm 0.004$ & $0.896 \pm 0.004$\\
CGANs2020CL \cite{ssbc20}                     & $0.803 \pm 0.006$ & $0.771 \pm 0.009$ & $0.838 \pm 0.003$ & $0.810 \pm 0.005$ & $0.803 \pm 0.006$ & $0.828 \pm 0.006$\\
ScleraU-Net \cite{ssbc20}                     & $0.795 \pm 0.005$ & $0.941 \pm 0.002$ & $0.689 \pm 0.007$ & $0.809 \pm 0.003$ & $0.805 \pm 0.004$ & $0.848 \pm 0.003$\\
AB Sclera Net \cite{ssbc20}                   & $0.786 \pm 0.008$ & $0.785 \pm 0.016$ & $0.787 \pm 0.007$ & $0.797 \pm 0.006$ & $0.793 \pm 0.008$ & $0.878 \pm 0.005$\\
Color RITNet \cite{ssbc20}                    & $0.774 \pm 0.007$ & $0.898 \pm 0.006$ & $0.680 \pm 0.011$ & $0.791 \pm 0.005$ & $0.783 \pm 0.006$ & $0.793 \pm 0.012$\\
ScleraMaskRCNN$^\dagger$ \cite{ssbc20}        & $0.763 \pm 0.011$ & $0.828 \pm 0.008$ & $0.707 \pm 0.015$ & $0.782 \pm 0.008$ & N/A & N/A\\
Multi-FCN \cite{ssbc20,roig2018iris}          & $0.760 \pm 0.007$ & $0.941 \pm 0.003$ & $0.638 \pm 0.009$ & $0.782 \pm 0.005$ & $0.716 \pm 0.006$ & $0.786 \pm 0.007$\\
Mask2020CL$^\dagger$ \cite{ssbc20,gonzalez2019deepblueberry} & $0.717 \pm 0.010$ & $0.833 \pm 0.013$ & $0.629 \pm 0.007$ & $0.749 \pm 0.006$ & N/A & N/A\\
MU-Net \cite{ssbc20}                          & $0.651 \pm 0.013$ & $0.638 \pm 0.012$ & $0.665 \pm 0.014$ & $0.698 \pm 0.008$ & $0.659 \pm 0.013$ & $0.554 \pm 0.014$\\
SSIP \cite{ssbc20,vyas2019efficient}          & $0.595 \pm 0.007$ & $0.762 \pm 0.009$ & $0.489 \pm 0.008$ & $0.672 \pm 0.005$ & $0.596 \pm 0.007$ & $0.524 \pm 0.007$\\
S-Net \cite{ssbc20}                           & $0.462 \pm 0.008$ & $0.348 \pm 0.008$ & $0.687 \pm 0.019$ & $0.552 \pm 0.005$ & $0.598 \pm 0.010$ & $0.662 \pm 0.011$\\
\hline\hline
\multicolumn{7}{l}{$^\dagger$ \small $F_1^{opt}$ and AUC scores are not reported for Mask2020CL and ScleraMaskRCNN because of issues with the submitted probabilistic results.}
\end{tabular}
\end{table}

**Figure 1:** Overall performance of the submitted segmentation approaches over all test images from the MOBIUS dataset.

**Figure 2:** Performance of the submitted segmentation approaches with different precision/recall tradeoffs (controlled by varying the binarisation threshold in the probabilistic predictions). The operating points denoted with a full circle represent the best possible F1-score (F1^opt), whereas the empty circle denotes the precision-recall point produced by the binary masks. The dotted lines denote the standard deviation.

References

Matej Vitek, Abhijit Das, Yann Pourcenoux, Alexandre Missler, Calvin Paumier, Sumanta Das, Ishita De Ghosh, Diego R. Lucio, Luiz A. Zanlorensi Jr., David Menotti, Fadi Boutros, Naser Damer, Jonas Henry Grebe, Arjan Kuijper, Junxing Hu, Yong He, Caiyong Wang, Hongda Liu, Yunlong Wang, Zhenan Sun, Daile Osorio-Roig, Christian Rathgeb, Christoph Busch, Juan Tapia Farias, Andres Valenzuela, Georgios Zampoukis, Lazaros Tsochatzidis, Ioannis Pratikakis, Sabari Nathan, R Suganya, Vineet Mehta, Abhinav Dhall, Kiran Raja, Gourav Gupta, Jalil Nourmohammadi Khiarak, Mohsen Akbari-Shahper, Farhang Jaryani, Meysam Asgari-Chenaghlu, Ritesh Vyas, Sristi Dakshit, Sagnik Dakshit, Peter Peer, Umapada Pal, and Vitomir Štruc: SSBC 2020: Sclera Segmentation Benchmarking Competition in the Mobile Environment. IEEE International Joint Conference on Biometrics (IJCB), 2020.

@inproceedings{ssbc2020,
	title={{SSBC} 2020: Sclera Segmentation Benchmarking Competition in the Mobile Environment},
	author={Vitek, Matej and Das, Abhijit and Pourcenoux, Yann and Missler, Alexandre and Paumier, Calvin and Das, Sumanta and De Ghosh, Ishita and Lucio, Diego R. and Zanlorensi Jr., Luiz A. and Menotti, David and Boutros, Fadi and Damer, Naser and Grebe, Jonas Henry and Kuijper, Arjan and Hu, Junxing and He, Yong and Wang, Caiyong and Liu, Hongda and Wang, Yunlong and Sun, Zhenan and Osorio-Roig, Daile and Rathgeb, Christian and Busch, Christoph and Tapia Farias, Juan and Valenzuela, Andres and Zampoukis, Georgios and Tsochatzidis, Lazaros and Pratikakis, Ioannis and Nathan, Sabari and Suganya, R and Mehta, Vineet and Dhall, Abhinav and Raja, Kiran and Gupta, Gourav and Khiarak, Jalil Nourmohammadi and Akbari-Shahper, Mohsen and Jaryani, Farhang and Asgari-Chenaghlu, Meysam and Vyas, Ritesh and Dakshit, Sristi and Dakshit, Sagnik and Peer, Peter and Pal, Umapada and \v{S}truc, Vitomir},
	booktitle={IEEE International Joint Conference on Biometrics (IJCB)},
	pages={1--10},
	year={2020},
	month={10},
	doi="10.1109/IJCB48548.2020.9304881"
}

Fadi Boutros, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper: Eye-MMS: Miniature Multi-Scale Segmentation Network of Key Eye-Regions in Embedded Applications. IEEE International Conference on Computer Vision (ICCV) Workshops, 2019.

@inproceedings{boutros2019eyemms,
	author = {Boutros, Fadi and Damer, Naser and Kirchbuchner, Florian and Kuijper, Arjan},
	title={Eye-MMS: Miniature Multi-Scale Segmentation Network of Key Eye-Regions in Embedded Applications},
	booktitle={The IEEE International Conference on Computer Vision (ICCV) Workshops},
	month={10},
	year={2019}
}

Caiyong Wang, Yong He, Yunfan Liu, Zhaofeng He, Ran He, and Zhenan Sun: ScleraSegNet: an Improved U-Net Model with Attention for Accurate Sclera Segmentation. IEEE International Conference on Biometrics (ICB), 1–8, 2019.

@inproceedings{wang2019sclerasegnet,
	title={ScleraSegNet: an Improved U-Net Model with Attention for Accurate Sclera Segmentation},
	author={Wang, Caiyong and He, Yong and Liu, Yunfan and He, Zhaofeng and He, Ran and Sun, Zhenan},
	booktitle={IEEE International Conference on Biometrics (ICB)},
	pages={1--8},
	year={2019}
}

Dailé Osorio-Roig, Christian Rathgeb, Marta Gomez-Barrero, Annette Morales-González, Eduardo Gareal-Llano, and Christoph Busch: Visible wavelength iris segmentation: a multi-class approach using fully convolutional neuronal networks. IEEE International Conference of the Biometrics Special Interest Group (BIOSIG), 1–5, 2018.

@inproceedings{osorio2018visible,
	title={Visible wavelength iris segmentation: a multi-class approach using fully convolutional neuronal networks},
	author={Osorio-Roig, Dail{\'e} and Rathgeb, Christian and Gomez-Barrero, Marta and Morales-Gonz{\'a}lez, Annette and Garea-Llano, Eduardo and Busch, Christoph},
	booktitle={IEEE International Conference of the Biometrics Special Interest Group (BIOSIG)},
	pages={1--5},
	year={2018}
}

Dailé Osorio-Roig, Pawel Drozdowski, Christian Rathgeb, Annette Morales-González, Eduardo Garea-Llano, and Christoph Busch: Iris recognition in visible wavelength: Impact and automated detection of glasses. IEEE Inetrnational Conference on Signal-Image Technology & Internet-Based Systems (SITIS), 542–546, 2018.

@inproceedings{roig2018iris,
	title={Iris recognition in visible wavelength: Impact and automated detection of glasses},
	author={Roig, Dail{\'e} Osorio and Drozdowski, Pawel and Rathgeb, Christian and Quevedo, Annette Morales Gonz{\'a}lez and Llano, Eduardo Garea and Busch, Christoph},
	booktitle={IEEE International Conference on Signal-Image Technology \& Internet-Based Systems (SITIS)},
	pages={542--546},
	year={2018}
}

Sebastian Gonzalez, Claudia Arellano, and Juan Tapia Farias: Deepblueberry: Quantification of blueberries in the wild using instance segmentation. IEEE Access, 7, 105776–105788, 2019.

@article{gonzalez2019deepblueberry,
	title={Deepblueberry: Quantification of blueberries in the wild using instance segmentation},
	author={Gonzalez, Sebastian and Arellano, Claudia and Tapia, Juan E},
	journal={IEEE Access},
	volume={7},
	pages={105776--105788},
	year={2019},
	publisher={IEEE}
}

Ritesh Vyas, Tirupathiraju Kanumuri, Gyanendra Sheoran, and Pawan Dubey: Efficient features for smartphone-based iris recognition. Turkish Journal of Electrical Engineering and Computer Sciences, 27(3), 1589–1602, 2019.

@article{vyas2019efficient,
	title={Efficient features for smartphone-based iris recognition},
	author={Vyas, Ritesh and Kanumuri, Tirupathiraju and Sheoran, Gyanendra and Dubey, Pawan},
	doi={10.3906/elk-1809-98},
	journal={Turkish Journal of Electrical Engineering and Computer Sciences},
	volume={27},
	number={3},
	pages={1589--1602},
	year={2019}
}