Kevin Roitero
Kevin Roitero
Ph.D. Student
University of Udine

Kevin Roitero took the bachelor degree in computer science at University of Udine.
He took his master joint degree in International computer science at University of Udine in collaboration with Alpen-Adria-University of Klagenfurt, with final grade 110/110 cum laude. He presented a Master Thesis on using fewer topics in the effectiveness evaluation of Information Retrieval systems.

He currently is a Ph.D. student in Computer Science at the University of Udine under the supervision of Prof. Stefano Mizzaro.
From April 2017 to June 2017 he visited Gianluca Demartini, at Information School, University Of Sheffield, United Kingdom.

His Research interests include :

  • Information Retrieval.
  • Crowdsourcing for Information Retrieval.
  • Artificial Intelligence for Information Retrieval.
  • Data mining and Analysis.

Here you can Download Kevin Roitero's Curriculum Vitae.

Current Position

November 2016 - Today

PhD student.

University of Udine, Udine (UD), Italy.


Education

November 2016 - Today

Ph.D.

Department of Mathematics, Computer Science, and Physics. University of Udine.

Title of the research project: Economical Evaluation of IR systems (Supervisor Prof. Stefano Mizzaro).
Topics of interest: Information Retrieval, Crowdsourcing for Information Retrieval, Artificial Intelligence for Information Retrieval, Data mining and Analysis.

April 2017 - June 2017

Visiting period at University Of Sheffield.

Information School, University Of Sheffield, United Kingdom.

Visit Gianluca Demartini, at Information School, University Of Sheffield, United Kingdom.

2014 - 2016

Master Joint Degree in International Computer Science.

University of Udine, Udine (UD), Italy,
Alpen-Adria-Universität, Klagenfurt, Austria. (6 months).

Title of the thesis: Improving the Efficiency of Retrieval Effectiveness Evaluation: Finding a Few Good Topics with Clustering and Dimensionality Reduction (Supervisor Prof. Stefano Mizzaro; Co-Supervisor Prof. Klaus Schoeffmann).

September 2015 - February 2016

Visiting period at Alpen-Adria-Universität.

Alpen-Adria-Universität, Klagenfurt, Austria.

2011 - 2014

Bachelor Degree in Computer Science.

University of Udine, Udine (UD), Italy.

Title of the thesis: Una Base di Dati per la Condivisione di Film e Serie Televisive - A database for sharing movies and tv-series informations (Supervisor Prof. Angelo Montanari).

2017

Let's Agree to Disagree: Fixing Agreement Measures for Crowdsourcing.

Alessandro Checco, Kevin Roitero, Eddy Maddalena, Gianluca Demartini and Stefano Mizzaro.

Proceedings of the The fifth AAAI Conference on Human Computation and Crowdsourcing, AAAI HCOMP2017. Quebec City, Canada. October 24-26 2017.
Url: https://aaai.org/ocs/index.php/HCOMP/HCOMP17/paper/viewFile/15927/15258.

Conference Paper.

Abstract

In the context of micro-task crowdsourcing, each task is usually performed by several workers. This allows researchers to leverage measures of the agreement among workers to estimate the reliability of collected data and to better understand answering behaviors of the participants. While many measures of agreement between annotators have been proposed, they are known for suffering from many problems and abnormalities. We identify the main limits of the existing agreement measures in the crowdsourcing context, both by means of toy examples as well as with real-world experimental data, and propose a novel agreement measure based on probabilistic parameter estimation which overcomes such limits. We validate our new agreement measure and show its advantages and disadvantages compared to the existing measures.

Considering Assessor Agreement in IR Evaluation.

Eddy Maddalena, Kevin Roitero, Gianluca Demartini and Stefano Mizzaro.

Proceedings of the 3rd ACM International Conference on the Theory of Information Retrieval, ICTIR2017. Amsterdam, Netherlands. October 1-4 2017.
To Be Published.

Conference Paper.

Abstract

The agreement between relevance assessors is an important but understudied topic in the Information Retrieval literature because of the limited data available about documents assessed by multiple judges. This issue has gained even more importance recently in light of crowdsourced relevance judgments, where it is customary to gather many relevance labels for each topic-document pair. In a crowdsourcing setting, agreement is often even used as a proxy for quality, although without any systematic verification of the conjecture that higher agreement corresponds to higher qualit In this paper we address this issue and we study in particular: the effect of topic on assessor agreement; the relationship between assessor agreement and judgment quality; the effect of agreement on ranking systems according to their effectiveness; and the definition of an agreement-aware effectiveness metric that does not discard information about multiple judgments for the same document as it typically happens in a crowdsourcing setting.

Economic Evaluation of Recommender Systems: A Proposal.

Kevin Roitero, Stefano Mizzaro, and Giuseppe Serra.

Proceedings of the 8th Italian Information Retrieval Workshop, Switzerland. June 05-07, 2017.
Url: http://ceur-ws.org/Vol-1911/8.pdf.

Conference Paper.

Abstract

The evaluation of information retrieval effectiveness by using fewer topics / queries has been studied for some years now: this approach potentially allows to save resources without sacrificing evaluation reliability. We propose to apply it to the evaluation of recommender systems. We describe our proposal and detail what is needed to put it in practice.

Do Easy Topics Predict Effectiveness Better Than Difficult Topics?

Kevin Roitero, Eddy Maddalena, and Stefano Mizzaro.

Proceedings of the 39th European Conference on IR Research, ECIR 2017, Aberdeen, UK, April 8-13, 2017.
pages: 605--611, isbn: 978-3-319-56608-5, doi: 10.1007/978-3-319-56608-5_55.
Url: http://dx.doi.org/10.1007/978-3-319-56608-5_55.

Conference Paper.

Abstract

After a network-based analysis of TREC results, Mizzaro and Robertson [4] found the rather unpleasant result that topic ease (i.e., the average effectiveness of the participating systems, measured with average precision) correlates with the ability of topics to predict system effectiveness (defined as topic hubness). We address this issue by: (i) performing a more detailed analysis, and (ii) using three different datasets. Our results are threefold. First, we confirm that the original result is indeed correct and general across datasets. Second, we show that, however, that result is less worrying than what might seem at first glance, since it depends on considering the least effective systems in the analysis. In other terms, easy topics discriminate most and least effective systems, but when focussing on the most effective systems only this is no longer true. Third, we also clarify what happens when using the GMAP metric.

2016

Improving the Efficiency of Retrieval Effectiveness Evaluation: Finding a Few Good Topics with Clustering?

Kevin Roitero and Stefano Mizzaro.

Proceedings of the 7th Italian Information Retrieval Workshop, Venezia, Italy, May 30-31, 2016.
Url: https://pdfs.semanticscholar.org/c190/c15c57b0a65df061a17c786143b32aa045e9.pdf.

Conference Paper.

Abstract

We consider the issue of using fewer topics in the effectiveness evaluation of information retrieval systems. Previous work has shown that using fewer topics is theoretically possible; one of the main issues that remains to be solved is how to find such a small set of a few good topics. To this aim, in this paper we try a novel approach based on clustering of topics. We consider various algorithms, metrics, and various features of topics that can be helpful in identifying such a set.

  • 6 June 2017

    Economic Evaluation of Recommender Systems: A Proposal.

    IIR2017, Lugano. Switzerland.

  • 10 April 2017

    Do Easy Topics Predict Effectiveness Better Than Difficult Topics?

    ECIR2017, Aberdeen. UK.

  • 30 May 2016

    Improving the Efficiency of Retrieval Effectiveness Evaluation: Finding a Few Good Topics with Clustering?

    IIR 2016, Venezia. Italy.

  • 12 May 2016

    Improving the Efficiency of Retrieval Effectiveness Evaluation: Finding a Few Good Topics with Clustering and Dimensionality Reduction.

    MSc Course (Web Information Retrieval). University of Udine, Udine. Italy.

  • March 2018 - July 2018.

    Allowance to visit RMIT Information Storage, Analysis and Retrieval (ISAR) at RMIT University (the Royal Melbourne Institute of Technology), Melbourne, Victoria, Australia.

    Provided by RMIT Information Storage, Analysis and Retrieval (ISAR).

  • October 2017.

    Student Grant to attend ``ICTIR2017'', Amsterdam, Netherlands.

    Provided by ACM SIGIR.

  • Apr 2017 - June 2017.

    Allowance to visit Information School at University of Sheffield, United Kingdom.

    Provided by ERASMUS+ Traineeship Project.

  • Apr 2017.

    Student Grant to attend ``ECIR2017'', Aberdeen, Scotland.

    Provided by BCS Information Retrieval Specialist Group (IRGS).

  • Sept 2015 - Feb 2016.

    Allowance to visit Alpen-Adria-Universität Kalgenfurt, Klagenfurt, Austria.

    Provided by University of Udine, and ERAMUS Project.

My main topics of interest are:
  • Information Retrieval.
  • Crowdsourcing for Information Retrieval.
  • Artificial Intelligence for Information Retrieval.
  • Data mining and Analysis.

Looking for thesis, projects, collarborations? write me an email.

2016 - 2017

Operating Systems Lab.

Bachelor Degree in Computer Science, University of Udine.

24 hours.

Here you can find some useful resources:

University of Udine
Dept. of Maths, Computer Science, and Physics.
Via delle Scienze 206
Udine (UD), Italy

Office: Rizzi Building, 2° floor, South Node, Room n° 4 (NS04)