EFFICIENTLY PROCESSING DATA IN TABLE WITH BILLIONS OF RECORDS


Abstract

Over time, systems connected to databases slow down. This is usually due to the increase in the amount of data stored in individual tables, counted even in the billions of records. Nevertheless, there are methods for making the speed of the system independent of the number of records in the database. One of these ways is table partitioning. When used correctly, the solution can ensure efficient operation of very large databases even after several years. However, not everything is predictable because of some undesirable phenomena become apparent only with a very large amount of data. The article presents a study of the execution time of the same queries with increasing number of records in a table. These studies reveal and present the timing and circumstances of the anomaly for a certain number of records.


Keywords

systems aging; partitioning; efficiently data processing; billions of records

Bednarczuk P.: Optimization in very large databases by partitioning tables, Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska 10(3), 2020, 95–98. DOI: https://doi.org/10.35784/iapgos.2056

Bandle M., Giceva J., Neumann T.: To Partition, or Not to Partition, That is the Join Question in a Real System. International Conference on Management of Data, 2021. DOI: https://doi.org/10.1145/3448016.3452831

Kumar A., Jitendra Singh Y.: A Review on Partitioning Techniques in Database. International Journal of Computer Science and Mobile Computing 13(5), 2014, 342–347.

Microsoft documentation, Data partitioning guidance, https://learn.microsoft.com/en-us/azure/architecture/best-practices/data-partitioning

Qi W., Song J., Yu-bin B.: Near-uniform Range Partition Approach for Increased Partitioning in Large Database. 2nd IEEE International Conference on Information Management and Engineering, 2010, 101–106. DOI: https://doi.org/10.1109/ICIME.2010.5477529

Song J., Bao Y.: NPA: Increased Partitioning Approach for Massive Data in Real-time Data Warehouse. 2nd International Conference on Information Technology Convergence and Services, 2010, 1–6. DOI: https://doi.org/10.1109/ITCS.2010.5581277

Tanvi J., Shivani S.: Refreshing Datawarehouse in Near Real-Time. International Journal of Computer Applications 46(18), 2012, 24–29.

Zheng K. et al.: Data storage optimization strategy in distributed column-oriented database by considering spatial adjacency. Cluster Computing 20, 2017. DOI: https://doi.org/10.1007/s10586-017-1081-3

Download

Published : 2022-12-30


Bednarczuk, P., & Borsuk, A. (2022). EFFICIENTLY PROCESSING DATA IN TABLE WITH BILLIONS OF RECORDS. Informatyka, Automatyka, Pomiary W Gospodarce I Ochronie Środowiska, 12(4), 17-20. https://doi.org/10.35784/iapgos.3058

Piotr Bednarczuk  piotr.bednarczuk@wsei.lublin.pl
University of Economics and Innovation in Lublin, Institute of Computer Science  Poland
http://orcid.org/0000-0003-1933-7183
Adam Borsuk 
University of Economics and Innovation in Lublin, Institute of Computer Science  Poland
http://orcid.org/0000-0003-2316-1694