Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADBDEV-4648: Limit diskquota hash table's size according initial request #28

Merged
merged 34 commits into from
Dec 29, 2023

Conversation

KnightMurloc
Copy link

@KnightMurloc KnightMurloc commented Nov 27, 2023

Diskquota did not control the size of its hash tables in shared memory and could
have consumed shared memory not intended for it, potentially impacting the other
database subsystems. Hash tables can also grow indefinitely if the background
process on the coordinator has not started, which can happen for a number of
reasons: gone done with error, pause, isn’t started. In this case, data is not
collected from segments and some hash tables (active_tables_map, relation_cache,
relid_cache) would not be cleared and would overflow.

This patch adds a limit on the size of all hash tables in shared memory by
adding a function that checks whether the hash table is full. The function
returns HASH_FIND if the map is full and HASH_ENTER otherwise. It also report a
warning if the table is full. Implemented a GUC that controls how frequently the
warning will be reported, as it could be reported too frequently. Also added a
GUC to control size of local reject map. The size of global reject map is set
as diskquota_max_local_reject_entries * diskquota_max_monitored_databases.

The test_active_table_limit test has been changed. Firstly, the value of
max_active_tables was changed from 2 to 5, since tables from all databases
processed by diskquota are placed in active_tables_map and with a limit of 2
tables overflow occurs even when the extension is created. Secondly, now a
table with 10 partitions is created to overflow active_tables_map, after which
another table is created into which data is inserted that should exhaust the
quota, but since this table does not inserted into active_tables_map, its size
is not taken into account and we can insert into the table after that. At the
end, vacuum full is done to achieve the overflow of altered_reloid_cache.

red1452 and others added 2 commits November 23, 2023 09:17
Diskquota does not control the size of its hash tables in shared memory and may
consume shared memory that is not intended for it, which may affect the other
database subsystems. Hash tables can also grow indefinitely if the background
process on the coordinator has not started, which can happen for a number of
reasons: gone done with error, pause, isn’t started.

This patch adds a limit on the size of hash tables in shared memory by adding a
wrapper over hash_search that controls the size of the table and stops adding
elements when the number of elements allocated at the beginning is reached and
report a warning. GUC is also added, which limits the frequency of report of the
warning, since in some cases it may be report too often.
src/diskquota.c Outdated Show resolved Hide resolved
src/diskquota_utility.c Outdated Show resolved Hide resolved
src/gp_activetable.c Outdated Show resolved Hide resolved
src/diskquota_utility.c Outdated Show resolved Hide resolved
src/gp_activetable.c Outdated Show resolved Hide resolved
src/diskquota_utility.c Outdated Show resolved Hide resolved
src/gp_activetable.c Outdated Show resolved Hide resolved
src/diskquota_utility.c Outdated Show resolved Hide resolved
… HASH_FIXED_SIZE to all hash tables in shared memory.
src/diskquota_utility.c Outdated Show resolved Hide resolved
@RekGRpth

This comment was marked as resolved.

src/quotamodel.c Outdated Show resolved Hide resolved
@RekGRpth

This comment was marked as resolved.

@RekGRpth

This comment was marked as resolved.

@KnightMurloc
Copy link
Author

we don't need them. I will remake all the tables in shared memory in the same style.

src/diskquota.c Show resolved Hide resolved
src/diskquota_utility.c Outdated Show resolved Hide resolved
src/diskquota_utility.c Outdated Show resolved Hide resolved
src/gp_activetable.c Outdated Show resolved Hide resolved
RekGRpth

This comment was marked as resolved.

src/diskquota_utility.c Outdated Show resolved Hide resolved
src/diskquota_utility.c Outdated Show resolved Hide resolved
src/gp_activetable.c Outdated Show resolved Hide resolved
src/quotamodel.c Outdated Show resolved Hide resolved
src/quotamodel.c Outdated Show resolved Hide resolved
src/relation_cache.c Outdated Show resolved Hide resolved
Base automatically changed from 2.3.0-sync to gpdb December 4, 2023 21:27
@RekGRpth

This comment was marked as resolved.

src/quotamodel.c Outdated Show resolved Hide resolved
src/quotamodel.c Outdated Show resolved Hide resolved
src/diskquota_utility.c Outdated Show resolved Hide resolved
src/quotamodel.c Outdated Show resolved Hide resolved
src/relation_cache.c Outdated Show resolved Hide resolved
@RekGRpth

This comment was marked as resolved.

RekGRpth
RekGRpth previously approved these changes Dec 27, 2023
red1452
red1452 previously approved these changes Dec 27, 2023
@KnightMurloc KnightMurloc dismissed stale reviews from red1452 and RekGRpth via 4508501 December 29, 2023 10:28
src/gp_activetable.c Outdated Show resolved Hide resolved
@andr-sokolov andr-sokolov merged commit 2862042 into gpdb Dec 29, 2023
3 checks passed
@andr-sokolov andr-sokolov deleted the ADBDEV-4648 branch December 29, 2023 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants