EFFICIENTLY PROCESSING DATA IN TABLE WITH BILLIONS OF RECORDS
Abstract
Over time, systems connected to databases slow down. This is usually due to the increase in the amount of data stored in individual tables, counted even in the billions of records. Nevertheless, there are methods for making the speed of the system independent of the number of records in the database. One of these ways is table partitioning. When used correctly, the solution can ensure efficient operation of very large databases even after several years. However, not everything is predictable because of some undesirable phenomena become apparent only with a very large amount of data. The article presents a study of the execution time of the same queries with increasing number of records in a table. These studies reveal and present the timing and circumstances of the anomaly for a certain number of records.
Keywords
systems aging; partitioning; efficiently data processing; billions of records
References
Bednarczuk P.: Optimization in very large databases by partitioning tables, Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska 10(3), 2020, 95–98. DOI: https://doi.org/10.35784/iapgos.2056
Bandle M., Giceva J., Neumann T.: To Partition, or Not to Partition, That is the Join Question in a Real System. International Conference on Management of Data, 2021. DOI: https://doi.org/10.1145/3448016.3452831
Kumar A., Jitendra Singh Y.: A Review on Partitioning Techniques in Database. International Journal of Computer Science and Mobile Computing 13(5), 2014, 342–347.
Microsoft documentation, Data partitioning guidance, https://learn.microsoft.com/en-us/azure/architecture/best-practices/data-partitioning
Qi W., Song J., Yu-bin B.: Near-uniform Range Partition Approach for Increased Partitioning in Large Database. 2nd IEEE International Conference on Information Management and Engineering, 2010, 101–106. DOI: https://doi.org/10.1109/ICIME.2010.5477529
Song J., Bao Y.: NPA: Increased Partitioning Approach for Massive Data in Real-time Data Warehouse. 2nd International Conference on Information Technology Convergence and Services, 2010, 1–6. DOI: https://doi.org/10.1109/ITCS.2010.5581277
Tanvi J., Shivani S.: Refreshing Datawarehouse in Near Real-Time. International Journal of Computer Applications 46(18), 2012, 24–29.
Zheng K. et al.: Data storage optimization strategy in distributed column-oriented database by considering spatial adjacency. Cluster Computing 20, 2017. DOI: https://doi.org/10.1007/s10586-017-1081-3
University of Economics and Innovation in Lublin, Institute of Computer Science Poland
http://orcid.org/0000-0003-1933-7183
University of Economics and Innovation in Lublin, Institute of Computer Science Poland
http://orcid.org/0000-0003-2316-1694
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
