DataDiff: User-Interpretable Data Transformation Summaries for Collaborative Data Analysis

ABSTRACT

Interest in collaborative dataset versioning has emerged due to complex, ad-hoc, and collaborative nature of data science, and the need to record and reason about data at various stages of pre-processing, cleaning, and analysis. To support effective collaborative dataset versioning, one critical operation is differentiation : to succinctly describe what has changed from one dataset to the next. Differentiation, or diffing, allows users to understand changes between two versions, to better understand the evolution process, or to support effective merging or conflict detection across versions. We demonstrate DataDiff, a practical and concise data-diff tool that provides human-interpretable explanations of changes between datasets without reliance on the operations that led to the changes.

References

1. {n. d.}. dbv. ({n. d.}). https://dbv.vizuina.com/
2. {n. d.}. Liquibase. ({n. d.}). http://www.liquibase.org/
3. {n. d.}. Noms. ({n. d.}). https://github.com/attic-labs/noms
4. 2017. Towards a Theory of Data-Diff: Optimal Synthesis of Succinct Data Modification Script. Technical Report. http://data-people.cs.illinois.edu/papers/datadiff. pdf
5. Azza Abouzied et al. 2013. Learning and verifying quantified boolean queries by example. In PODS. ACM, 49--60.
6. Ilsoo Ahn et al. 1986. Performance evaluation of a temporal database management system. In ACM SIGMOD Record, Vol. 15. ACM, 96--107.
7. Mohammed Al-Kateb et al. 2013. Temporal query processing in Teradata. In EDBT '13. ACM, 573--578.
8. Anant Bhardwaj, Souvik Bhattacherjee, Amit Chavan, Amol Deshpande, Aaron J Elmore, Samuel Madden, and Aditya G Parameswaran. 2015. Datahub: Collaborative data science &dataset version management at scale. CIDR (2015).
9. Angela Bonifati et al. 2016. Learning join queries from user examples. TODS 40, 4 (2016), 24.
10. Amit Chavan and Amol Deshpande. 2017. DEX: Query Execution in a Delta-based Storage System. In SIGMOD. ACM, 171--186.
11. Anish Das Sarma et al. 2010. Synthesizing view definitions from data. In ICDT. ACM, 89--103.
12. Joseph M Hellerstein et al. 2017. Ground: A Data Context Service.. In CIDR.
13. Silu Huang, Liqi Xu, Jialin Liu, Aaron J. Elmore, and Aditya G. Parameswaran. 2017. OrpheusDB: Bolt-on Versioning for Relational Databases. PVLDB (2017).
14. Christian S Jensen and Richard T Snodgrass. 1999. Temporal data management. IEEE Transactions on Knowledge and Data Engineering 11, 1 (1999), 36--44.
15. Linan Jiang, Betty Salzberg, David B Lomet, and Manuel Barrena García. 2000. The BT-tree: A Branched and Temporal Access Method.. In VLDB. 451--460.
16. Gad M Landau et al. 1995. Historical queries along multiple lines of time evolution. The VLDB Journal 4, 4 (1995), 703--726.
17. Michael Maddox, David Goehring, Aaron J Elmore, Samuel Madden, Aditya Parameswaran, and Amol Deshpande. 2016. Decibel: The relational dataset branching system. VLDB 9, 9 (2016), 624--635.
18. Kiril Panev and Sebastian Michel. 2016. Reverse Engineering Top-k Database Queries with PALEO.. In EDBT. 113--124.
19. Theodoros Rekatsinas et al. 2017. HoloClean: Holistic Data Repairs with Probabilistic Inference. PVLDB 10, 11 (2017), 1190--1201.
20. Betty Joan Salzberg and David B Lomet. 1995. Branched and Temporal Index Structures. College of Computer Science, Northeastern University.
21. Cynthia M Saracco, Matthias Nicola, and Lenisha Gandhi. 2010. A matter of time: Temporal data management in DB2 for z/OS. IBM Corporation, New York (2010).
22. Q. T. Tran, C. Y. Chan, and S. Parthasarathy. 2009. Query by Output.
23. Meihui Zhang et al. 2013. Reverse engineering complex join queries. In SIGMOD. ACM, 809--820.

Index Terms

• Information systems

  • Data management systems

    • Information integration

      • Data cleaning