EFFICIENTLY PROCESSING DATA IN TABLE WITH BILLIONS OF RECORDS
Piotr Bednarczuk
piotr.bednarczuk@wsei.lublin.plUniversity of Economics and Innovation in Lublin, Institute of Computer Science (Poland)
http://orcid.org/0000-0003-1933-7183
Adam Borsuk
University of Economics and Innovation in Lublin, Institute of Computer Science (Poland)
http://orcid.org/0000-0003-2316-1694
Abstract
Over time, systems connected to databases slow down. This is usually due to the increase in the amount of data stored in individual tables, counted even in the billions of records. Nevertheless, there are methods for making the speed of the system independent of the number of records in the database. One of these ways is table partitioning. When used correctly, the solution can ensure efficient operation of very large databases even after several years. However, not everything is predictable because of some undesirable phenomena become apparent only with a very large amount of data. The article presents a study of the execution time of the same queries with increasing number of records in a table. These studies reveal and present the timing and circumstances of the anomaly for a certain number of records.
Keywords:
systems aging, partitioning, efficiently data processing, billions of recordsReferences
Bednarczuk P.: Optimization in very large databases by partitioning tables, Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska 10(3), 2020, 95–98.
DOI: https://doi.org/10.35784/iapgos.2056
Google Scholar
Bandle M., Giceva J., Neumann T.: To Partition, or Not to Partition, That is the Join Question in a Real System. International Conference on Management of Data, 2021.
DOI: https://doi.org/10.1145/3448016.3452831
Google Scholar
Kumar A., Jitendra Singh Y.: A Review on Partitioning Techniques in Database. International Journal of Computer Science and Mobile Computing 13(5), 2014, 342–347.
Google Scholar
Microsoft documentation, Data partitioning guidance, https://learn.microsoft.com/en-us/azure/architecture/best-practices/data-partitioning
Google Scholar
Qi W., Song J., Yu-bin B.: Near-uniform Range Partition Approach for Increased Partitioning in Large Database. 2nd IEEE International Conference on Information Management and Engineering, 2010, 101–106.
DOI: https://doi.org/10.1109/ICIME.2010.5477529
Google Scholar
Song J., Bao Y.: NPA: Increased Partitioning Approach for Massive Data in Real-time Data Warehouse. 2nd International Conference on Information Technology Convergence and Services, 2010, 1–6.
DOI: https://doi.org/10.1109/ITCS.2010.5581277
Google Scholar
Tanvi J., Shivani S.: Refreshing Datawarehouse in Near Real-Time. International Journal of Computer Applications 46(18), 2012, 24–29.
Google Scholar
Zheng K. et al.: Data storage optimization strategy in distributed column-oriented database by considering spatial adjacency. Cluster Computing 20, 2017.
DOI: https://doi.org/10.1007/s10586-017-1081-3
Google Scholar
Authors
Piotr Bednarczukpiotr.bednarczuk@wsei.lublin.pl
University of Economics and Innovation in Lublin, Institute of Computer Science Poland
http://orcid.org/0000-0003-1933-7183
Authors
Adam BorsukUniversity of Economics and Innovation in Lublin, Institute of Computer Science Poland
http://orcid.org/0000-0003-2316-1694
Statistics
Abstract views: 1698PDF downloads: 116
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Most read articles by the same author(s)
- Tomasz Rymarczyk, Bartek Przysucha, Marcin Kowalski, Piotr Bednarczuk, ANALYSIS OF DATA FROM MEASURING SENSORS FOR PREDICTION IN PRODUCTION PROCESS CONTROL SYSTEMS , Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska: Vol. 9 No. 4 (2019)
- Piotr Bednarczuk, OPTIMIZATION IN VERY LARGE DATABASES BY PARTITIONING TABLES , Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska: Vol. 10 No. 3 (2020)
- Piotr Bednarczuk, CUSTOMIZATION BASED ON CAD AUTOMATION IN PRODUCTION OF MEDICAL SCREWS BY 3D PRINTING , Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska: Vol. 11 No. 3 (2021)
