December 21st, 2016

SearchLight: Ad-Hoc, Interactive Search and Exploration over Big Data  


Exploratory data analysis plays a key role in data-driven discovery in a wide range of domains including science, engineering, and business. This project aims to enable data scientists from many domains to search and explore their large data sets far easier and faster than they do today. Rather than spending a lot of time to set up exploration pipelines by combining multiple software tools, users will work with a single, general purpose and more usable system. Overall, this project will enable fundamentally richer means for data exploration and lead to significant productivity improvements; it will accelerate discovery and breakthroughs in many domains such as e-commerce, finance, and science. This research will be incorporated in undergraduate and graduate coursework. The outreach activities include special research and education-focused programs that are geared towards undergraduates and high-school girls.

This research is building a new prototype database system, called Searchlight, that uniquely integrates constraint solving and data management techniques. The result enables rich, highly-efficient means for generic ad hoc search, exploration and mining over large multidimensional data collections. Searchlight allows Constraint Programming (CP) machinery to run efficiently inside a DBMS without the need to extract, transform and move the data. This marriage offers the rich expressiveness and efficiency of constraint-based search and optimization provided by modern CP solvers with the ability of Database Management Systems (DBMSs) to store and query data at scale.

Searchlight is a transformative step in enriching the functionality of database systems towards new data- and search-intensive applications. We are developing novel approaches for synopsis-based in-memory processing, speculative solving, search query optimization, parallel processing and load balancing, which collectively yield performance and usability levels that far improve those of the state of the art.






The Searchlight project is supported by the NSF grant IIS-1526639.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.