Affiliations: Software and Computer System Department, ICT School, Forum 105, 164 40 Kista, Sweden. E-mail: [email protected] | Institute of Computer Science, University of Tartu, J. Liivi 2, 50409, Tartu, Estonia. E-mail: [email protected]
Abstract: In the past years various methods have been developed which require semantic annotations of Web services as an input. Such methods typically leverage discovery, match-making, composition and execution of Web services in dynamic settings. At the same time a number of automated Web service annotation approaches have been proposed for enabling application of these methods in settings where it is not feasible to provide the annotations manually. However, lack of effective automated evaluation frameworks has seriously limited proper assessment of the constructed annotations in settings where the overall annotation quality of large quantities of Web services needs to be evaluated. This paper describes an evaluation framework for measuring the quality of semantic annotations for a large number of real-world Web services from heterogeneous application domains. The evaluation framework is generally based on analyzing properties of Web service networks constructed from semantic annotations of the Web services. More specifically, we measure scale-free, small-world and correlation degree properties of the networks to evaluate the overall quality of annotations. The evaluation is demonstrated using annotations constructed semi-automatically for a set of publicly available WSDL documents containing descriptions of about 200 000 Web service operations.
Keywords: Web service networks, semantic Web services, web service annotation, evaluation of annotation quality, network theory
