Query-by-Gaming: Interactive spatio-temporal querying and retrieval using gaming controller

Inputting spatio-temporal queries is challenging because it requires a sequence of both spatial and temporal information in queries. Despite development of powerful spatio-temporal index structures and querying languages, forming spatio-temporal queries remains difficult for several users. Inspired from games where a user provides a sequence of actions on a spatial layout, we propose the methodology of query-by-gaming for interactive spatio-temporal querying employing a gaming controller. We’ve applied query-by-gaming for spatio-temporal retrieval of content for tennis videos. Queries are built employing a gaming controller instead of a mouse or keyboard. First, for a database of tennis videos, the players, ball, and shot sorts of players are indexed with reference to their positions using our semantic sequence state graph (S3G). The user is allowed to pick the initial positions of objects on the court and a relevant clip is retrieved from the database to create the query. The user starts with this video clip to pick the positions of players and ball including where the ball should be sent after an attempt using the controller. The system searches the database and determines if there's a relevant clip and therefore the partial clip is displayed if found. Then the user is permitted to regulate the opposite player with an option of skipping events and therefore the query may be a built as a sequence of interactions. Meanwhile, the system searches for videos that satisfy the entire query as a conjunction of query segments. At the top of query, the user is provided the list of videos satisfying his or her query. We use linear temporal logic to formally represent our queries. We’ve also performed interface evaluation for comparing gamepad-based interface and mouse interface for forming spatio-temporal queries. Bridging the gap between users and management systems require effective user interfaces. The most challenge of bridging this gap is to organize interfaces that need minimal information to be learned about the system that executes queries. For complex querying systems beyond basic keyword search, menu-based interfaces are popular for querying traditional electronic database systems. However, building spatio-temporal queries is more complicated since these queries may involve associating locations to things and changing positions of objects may have to be specified over a temporal interval or a series of temporal events. A spatio-temporal query because the name implies has spatial and temporal components. For instance, for a soccer video database, the query for retrieving “goals scored by Messi against Real Madrid within the last 10 min outside the box” has both spatial and temporal components. This question features a temporal constraint as “in the last 10 min” and a spatial constraint “outside the box.” Despite being a spatio-temporal query, it doesn't have continuity of events. In other words, this question only checks an occasion that happens almost instantaneously. During this chapter, instead of focusing events instantaneously, the development of queries that have a sequence of events is analyzed. for instance , consider the query for retrieving video clips where “(i) within the mid-field, Messi passes the ball to Neymar, (ii) Neymar passes the ball to Suarez, (iii) Suarez passes the ball to Messi, and iv) Messi scores outside the box to the highest left corner.” to create this question , a simple, interactive, and incremental interface would be necessary. In this chapter, the methodology of “query-by-gaming” is explained as how of getting input from users for constructing spatio-temporal queries. When various user interfaces for inputting data (e.g., keyboard, mouse, gaming controller, voice) is taken into account, it's been observed that gamers provide spatio-temporal information to the gaming engine in many game genres. Gamers control objects, move them, and generate actions based using gaming controllers. Moreover, considering the usage of technology, gaming has been popular environment not just for kids but also for old people. This observation may be a motivation to use gaming controllers for spatio-temporal querying. Especially, for sports games like tennis, soccer, football, basketball (Fig. 1), spatio-temporal querying might be achieved using gaming controllers. Query-by-gaming can effectively be used for building spatio-temporal queries then these queries are often executed using the semantic sequence state graph (S3G). While the user plays a game, the querying system analyzes user inputs on controlling objects then displays the video segment that's closest to what the user has provided for the continuity of the sport. The program processes all user states and eventually presents the particular results of the user query. Moreover, the program also keeps the clips that satisfy the partial user query because the query is made. S3G features a low complexity for executing the next query. Semantic events or displacement of objects determine the new states. Especially, the next query depends on the granularity of semantic events. Semantic transitions should be adapted carefully. Nevertheless, eventual query may always work albeit the granularity of states may differ. As future work, the continuity of user playing experience might be increased. The present system currently pauses the video for the user input. As future work, the querying engine may detect movement of objects and execute queries instantly without user interruption.

Author(s): Ramazan S Aygun

Abstract | PDF

Share This Article