forked from greenplum-db/diskquota-archive
-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ADBDEV-4648: Limit diskquota hash table's size according initial request #28
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Diskquota does not control the size of its hash tables in shared memory and may consume shared memory that is not intended for it, which may affect the other database subsystems. Hash tables can also grow indefinitely if the background process on the coordinator has not started, which can happen for a number of reasons: gone done with error, pause, isn’t started. This patch adds a limit on the size of hash tables in shared memory by adding a wrapper over hash_search that controls the size of the table and stops adding elements when the number of elements allocated at the beginning is reached and report a warning. GUC is also added, which limits the frequency of report of the warning, since in some cases it may be report too often.
KnightMurloc
force-pushed
the
ADBDEV-4648
branch
from
November 27, 2023 08:56
64ae8db
to
f7b0ca1
Compare
RekGRpth
reviewed
Nov 28, 2023
RekGRpth
reviewed
Nov 28, 2023
… HASH_FIXED_SIZE to all hash tables in shared memory.
RekGRpth
reviewed
Nov 29, 2023
This comment was marked as resolved.
This comment was marked as resolved.
RekGRpth
reviewed
Nov 29, 2023
This comment was marked as resolved.
This comment was marked as resolved.
…add flag HASH_FIXED_SIZE to all hash tables in shared memory." This reverts commit 705d2ca.
This comment was marked as resolved.
This comment was marked as resolved.
we don't need them. I will remake all the tables in shared memory in the same style. |
RekGRpth
reviewed
Nov 30, 2023
RekGRpth
reviewed
Nov 30, 2023
RekGRpth
reviewed
Nov 30, 2023
RekGRpth
reviewed
Nov 30, 2023
This comment was marked as resolved.
This comment was marked as resolved.
KnightMurloc
force-pushed
the
ADBDEV-4648
branch
from
December 25, 2023 09:51
99cd01f
to
f650b93
Compare
RekGRpth
reviewed
Dec 25, 2023
RekGRpth
reviewed
Dec 26, 2023
RekGRpth
reviewed
Dec 26, 2023
RekGRpth
reviewed
Dec 26, 2023
RekGRpth
reviewed
Dec 26, 2023
…ota_worker_shmem_size
This comment was marked as resolved.
This comment was marked as resolved.
RekGRpth
reviewed
Dec 27, 2023
RekGRpth
reviewed
Dec 27, 2023
RekGRpth
previously approved these changes
Dec 27, 2023
red1452
previously approved these changes
Dec 27, 2023
RekGRpth
reviewed
Dec 29, 2023
RekGRpth
approved these changes
Dec 29, 2023
red1452
approved these changes
Dec 29, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Diskquota did not control the size of its hash tables in shared memory and could
have consumed shared memory not intended for it, potentially impacting the other
database subsystems. Hash tables can also grow indefinitely if the background
process on the coordinator has not started, which can happen for a number of
reasons: gone done with error, pause, isn’t started. In this case, data is not
collected from segments and some hash tables (active_tables_map, relation_cache,
relid_cache) would not be cleared and would overflow.
This patch adds a limit on the size of all hash tables in shared memory by
adding a function that checks whether the hash table is full. The function
returns HASH_FIND if the map is full and HASH_ENTER otherwise. It also report a
warning if the table is full. Implemented a GUC that controls how frequently the
warning will be reported, as it could be reported too frequently. Also added a
GUC to control size of local reject map. The size of global reject map is set
as diskquota_max_local_reject_entries * diskquota_max_monitored_databases.
The test_active_table_limit test has been changed. Firstly, the value of
max_active_tables was changed from 2 to 5, since tables from all databases
processed by diskquota are placed in active_tables_map and with a limit of 2
tables overflow occurs even when the extension is created. Secondly, now a
table with 10 partitions is created to overflow active_tables_map, after which
another table is created into which data is inserted that should exhaust the
quota, but since this table does not inserted into active_tables_map, its size
is not taken into account and we can insert into the table after that. At the
end, vacuum full is done to achieve the overflow of altered_reloid_cache.