Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend testing of JS runtime limits #5594

Merged
merged 12 commits into from
Sep 19, 2023

Conversation

eddyashton
Copy link
Member

Trying to improve confidence in JS runtime limits following interpreter reuse investigation. This PR extends the existing tests to set a too-low value for each limit, and confirm that causes an isolated failure, as well as testing a too-large-for-the-default value. I've also moved the execution time limit to the limit tests, rather than testing it separately in modules.

Verified

This commit was signed with the committer’s verified signature.
mszostok Mateusz Szostok

Verified

This commit was signed with the committer’s verified signature.
mszostok Mateusz Szostok
@eddyashton eddyashton requested a review from a team August 29, 2023 10:24
@ghost
Copy link

ghost commented Aug 29, 2023

test_js_runtime_limits@75982 aka 20230919.7 vs main ewma over 20 builds from 75578 to 75972

Click to see table

main

build_id build_number pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem Commit latency factor pi_basic_mt_virtual_cft^ ls_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_ls_virtual_cft^ pi_ls_sgx_cft^ pi_ls_sgx_cft_mem pi_basic_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem pi_basic_js_virtual_cft^ ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ pi_basic_js_sgx_cft^ pi_basic_js_sgx_cft_mem ls_full_js_virtual_cft^ ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_jwt_virtual_cft^ pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^ tpcc_virtual_cft^ tpcc_sgx_cft^ tpcc_sgx_cft_mem
75578 20230908.2 32091.1 2.51822e+07 0.828259 78823.5 43942.1 16699.5 1.88908e+07 48082.5 16847.4 1.05021e+07 54705.4 18400.6 1.46964e+07 4443.4 17216 18878.3 17041.1 1654.5 1.25993e+07 16926.4 8136.67 1.88908e+07 9811.7 8377.3 6.30784e+06 7061.63 1.67936e+07 6591.31 1.67936e+07 4554.19 1.88908e+07 48501.8 833497 1.1874e+06 8.15413e+06 3.08021e+07 nan nan nan
75585 20230908.5 32334.7 2.51822e+07 0.797482 74515.3 43449.2 16880.3 1.88908e+07 48049.2 16912.1 1.05021e+07 54134.4 18498.4 1.25993e+07 4445.7 16982.9 19441.6 17178.3 1654.4 1.25993e+07 14747.2 8214.32 1.88908e+07 10169.7 8178.3 6.30784e+06 6689.4 1.67936e+07 6602.85 1.67936e+07 4556.48 1.88908e+07 48501 840060 1.18261e+06 8.15384e+06 3.08624e+07 nan nan nan
75619 20230908.14 27206.7 2.51822e+07 0.792552 71512.6 45711.8 13892.6 1.88908e+07 47721.1 13918.6 1.05021e+07 53565.9 15383.3 1.46964e+07 4169.8 17033 19433.3 17333.6 1447 1.25993e+07 14800.1 6791.3 1.67936e+07 9828.73 6879.8 6.30784e+06 5749.43 1.67936e+07 5721.31 1.67936e+07 3985.08 1.67936e+07 46035.4 831032 1.18459e+06 8.14194e+06 3.08067e+07 nan nan nan
75633 20230908.19 26426.9 2.30851e+07 0.814514 87196.1 43679.4 13841.4 1.88908e+07 47559.5 13952.4 1.05021e+07 53848.3 15293 1.46964e+07 4427.3 17198.8 19457.1 17581.4 1426.1 1.25993e+07 14768.9 6803.4 1.67936e+07 10298.4 6796.8 6.30784e+06 5780.48 1.67936e+07 5455.24 1.67936e+07 3984.61 1.67936e+07 40163.3 833682 1.18594e+06 8.13551e+06 3.05845e+07 nan nan nan
75639 20230908.21 27149.3 2.51822e+07 0.780733 86236.2 45714.2 13880 1.88908e+07 47797.3 13884.3 1.05021e+07 54112.7 15315.7 1.46964e+07 4489.1 17346.3 19833.8 17596.5 1435.7 1.25993e+07 14765.9 7182.95 1.67936e+07 9895.98 6940.1 6.30784e+06 5791.25 1.67936e+07 5708.59 1.67936e+07 3967.22 1.67936e+07 46797.6 836115 1.1819e+06 8.14551e+06 3.06335e+07 nan nan nan
75654 20230908.26 27195 2.51822e+07 0.800096 89588.7 43869.4 13871.4 1.67936e+07 47485.2 13956.2 1.05021e+07 54275.4 15367 1.46964e+07 4469.5 17192.2 19753.5 17307.3 1435.2 1.25993e+07 15019.4 6841.62 1.88908e+07 9817.01 6895.5 6.30784e+06 5790.81 1.67936e+07 5688.99 1.67936e+07 3967.13 1.67936e+07 43470.8 831294 1.18063e+06 8.16519e+06 3.08276e+07 nan nan nan
75662 20230911.2 27181.8 2.51822e+07 0.808688 83834.8 45601.5 13935.4 1.88908e+07 47690.4 13957 1.05021e+07 53934.4 15499.6 1.46964e+07 4406 17267 18849.8 17581 1440.2 1.25993e+07 14898.7 6826.9 1.67936e+07 9856.76 7172.6 6.30784e+06 5746.37 1.67936e+07 5756.56 1.67936e+07 3997.11 1.67936e+07 44575.1 836042 1.18753e+06 8.15413e+06 3.15135e+07 nan nan nan
75672 20230911.6 26990.3 2.30851e+07 0.819773 75501.5 43845.5 13880 1.67936e+07 47682.4 13947.8 1.05021e+07 54141 15361.7 1.46964e+07 4457.3 17348 19305 15082.7 1436.3 1.25993e+07 14637.5 6809.97 1.88908e+07 10070.8 6953.1 6.30784e+06 5793.21 1.67936e+07 5761.62 1.67936e+07 3995.63 1.67936e+07 43290.9 837710 1.17811e+06 8.15426e+06 3.07175e+07 nan nan nan
75689 20230911.11 27924.5 2.51822e+07 0.823735 71344 44179.5 14021.1 1.67936e+07 48569.2 14105.4 1.05021e+07 55054.6 15658.3 1.25993e+07 4427.2 16956.5 19235.2 16861.3 1439.1 1.25993e+07 14888.8 6789.15 1.88908e+07 10027.2 6886.5 6.30784e+06 5751.1 1.67936e+07 5791.93 1.67936e+07 3990.84 1.67936e+07 44569.5 828507 1.1814e+06 8.15254e+06 3.08067e+07 nan nan nan
75737 20230912.2 28245.3 2.30851e+07 0.799989 66593.6 45828 14029.7 1.67936e+07 48672.7 14130 1.05021e+07 55157 15650.9 1.46964e+07 4401.2 17455.8 19046.9 17397.5 1439.4 1.25993e+07 14976.6 6851.6 1.67936e+07 9848.82 6952.3 6.30784e+06 5810.95 1.67936e+07 5776.87 1.67936e+07 4003.93 1.67936e+07 45150.1 829130 1.18145e+06 8.15413e+06 3.08165e+07 nan nan nan
75772 20230913.11 28279.8 2.30851e+07 0.788811 88377.3 45669.9 14062.8 1.67936e+07 48638.1 14175.1 1.05021e+07 54661.6 15715.1 1.25993e+07 4470.9 17245.1 19200.5 17151.2 1442.1 1.25993e+07 16866.9 6915.48 1.67936e+07 9889.62 7083 6.30784e+06 5767.63 1.67936e+07 5777.31 1.67936e+07 3997.95 1.67936e+07 44998.8 832119 1.18482e+06 8.15241e+06 3.12433e+07 nan nan nan
75799 20230914.3 28106.1 2.30851e+07 0.806205 87358.3 45522.9 14099.3 1.88908e+07 47807.1 14222.8 1.05021e+07 54260.2 15712.5 1.25993e+07 4440.7 17055.8 19494.3 17556.9 1443.8 1.25993e+07 15082 6861.85 1.67936e+07 10417.4 7063 6.30784e+06 5764.98 1.67936e+07 5779.31 1.67936e+07 4011.33 1.67936e+07 39042 835858 1.18377e+06 8.15563e+06 3.10359e+07 nan nan nan
75807 20230914.6 27691.2 2.72794e+07 0.78551 67424.6 45657.1 14039.5 1.67936e+07 42493.5 14143.9 1.05021e+07 55173.4 15706.6 1.46964e+07 4485.6 17097 19237.8 17069 1444.6 1.25993e+07 14955.4 6835.29 1.67936e+07 9815.92 7167.6 6.30784e+06 5814.42 1.67936e+07 5740.11 1.67936e+07 4006.15 1.67936e+07 47030.4 832713 1.1827e+06 8.15501e+06 3.08587e+07 17283.2 5625.94 8.59996e+07
75836 20230915.2 28225.2 2.51822e+07 0.806175 71131.7 45882.2 14058.9 1.67936e+07 48155.9 14137 1.05021e+07 55533.3 15692.2 1.46964e+07 4509.5 17429.7 19296.8 17117.5 1446.3 1.25993e+07 14633.5 6865.75 1.88908e+07 9896.12 6927.4 6.30784e+06 5761.62 1.67936e+07 5745.27 1.67936e+07 4006.31 1.67936e+07 41678 833529 1.17526e+06 8.15209e+06 3.07549e+07 17210.5 5618.1 8.59996e+07
75842 20230915.5 27975.1 2.51822e+07 0.812338 66045.1 45522.7 14057.6 1.67936e+07 47870.5 14097.1 1.05021e+07 54521.4 15647 1.46964e+07 4429.9 17019.5 19290.9 17063.7 1438.9 1.25993e+07 14987.3 6863.63 1.67936e+07 10237.6 6939.8 6.30784e+06 5785.28 1.67936e+07 5782.56 1.67936e+07 4007.85 1.67936e+07 41341.1 830832 1.18103e+06 8.1391e+06 3.03488e+07 17070.3 5598.47 8.59996e+07
75882 20230915.14 27598.1 2.51822e+07 0.79396 82541.5 45745 14070.2 1.88908e+07 47701.7 14174.4 1.05021e+07 54950.9 15653.1 1.46964e+07 4451.4 17193.1 19974.4 17175.9 1446.3 1.05021e+07 14859.4 6863.13 1.88908e+07 10059.6 7074 6.30784e+06 5781.81 1.67936e+07 5780.46 1.67936e+07 4000.23 1.67936e+07 41264.9 831777 1.18327e+06 8.17288e+06 3.10618e+07 17347.4 5657.84 8.59996e+07
75891 20230915.17 27847.3 2.30851e+07 0.812985 81063.5 45795.3 13993.8 1.67936e+07 48452.4 14041.4 1.05021e+07 54635 15427.5 1.46964e+07 4439.8 17018.8 19367.8 17619.6 1429.7 1.25993e+07 14952.2 6863.77 1.88908e+07 9842.66 7026.7 6.30784e+06 5820.89 1.67936e+07 5489.42 1.67936e+07 3992.44 1.67936e+07 40064.4 839722 1.17713e+06 8.17392e+06 3.07494e+07 17178.3 5497.14 8.59996e+07
75905 20230915.21 28013 2.51822e+07 0.794597 84235.8 46018.7 13971 1.88908e+07 47955.2 14089.9 1.05021e+07 55480.7 15562.1 1.46964e+07 4446.8 17077.2 19101.1 17045.6 1430.6 1.25993e+07 16704.8 6840 1.88908e+07 10411.6 6841 6.30784e+06 5780.32 1.67936e+07 5754.13 1.67936e+07 3966.24 1.67936e+07 39534.4 838322 1.18234e+06 8.14897e+06 3.10045e+07 17245 5560.01 8.59996e+07
75919 20230918.1 28037.7 2.51822e+07 0.807726 80251.8 45705.8 14069.4 1.88908e+07 48106.5 14100.3 1.05021e+07 54799.3 15624.7 1.46964e+07 4473.2 17309.5 19191.3 17711.8 1447.6 1.25993e+07 15013 7246.59 1.67936e+07 9870.13 6938.6 6.30784e+06 5817.38 1.67936e+07 5750.83 1.88908e+07 4001.82 1.67936e+07 45085.8 836473 1.18406e+06 8.15462e+06 3.0718e+07 17123.9 5617.99 8.59996e+07
75972 20230919.3 28013.6 2.51822e+07 0.797908 89712 45907.5 13979.5 1.88908e+07 48087.1 14131.3 1.05021e+07 54724 15590.4 1.46964e+07 4440.1 17023.5 19581.7 17397.5 1440.2 1.25993e+07 14717.5 6847.99 1.88908e+07 10069 7025.4 6.30784e+06 5812.09 1.67936e+07 5786.4 1.67936e+07 3974.68 1.67936e+07 46322.7 826226 1.18058e+06 8.15186e+06 3.07355e+07 17443 5600.61 8.59996e+07

test_js_runtime_limits

build_id build_number pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem Commit latency factor pi_basic_mt_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_virtual_cft^ pi_ls_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem pi_basic_virtual_cft^ pi_basic_js_virtual_cft^ ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ pi_basic_js_sgx_cft^ pi_basic_js_sgx_cft_mem ls_js_virtual_cft^ ls_full_js_virtual_cft^ ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_jwt_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^ tpcc_sgx_cft^ tpcc_sgx_cft_mem tpcc_virtual_cft^
75192 20230901.6 31802.4 2.51822e+07 0.776473 85238.3 16603 1.88908e+07 16610.8 1.05021e+07 45814.7 48336.7 18149.6 1.46964e+07 53887.8 4512.5 17245.9 19169.2 1652.2 1.25993e+07 17089 16909.2 8176.55 1.88908e+07 8088.3 6.30784e+06 9891.41 7064.89 1.67936e+07 6611.01 1.67936e+07 4583.6 1.88908e+07 47651.7 832329 1.18011e+06 8.15413e+06 3.0973e+07 nan nan nan
75228 20230901.22 31817.3 2.51822e+07 0.775781 73013.6 16458.4 1.88908e+07 16583.2 1.05021e+07 45651.2 47374.1 18020.8 1.46964e+07 52457.9 4513.4 17332.2 19938.6 1647.4 1.25993e+07 17730.7 14911.3 8176.02 1.88908e+07 8294.3 6.30784e+06 9830.51 7044.49 1.67936e+07 6643.54 1.67936e+07 4541.33 1.67936e+07 45113.9 823943 1.17776e+06 8.15352e+06 3.31413e+07 nan nan nan
75319 20230905.9 32066.9 2.51822e+07 0.828782 71536.9 16726.3 1.88908e+07 16773.3 1.05021e+07 45798.9 48221.6 18386.4 1.46964e+07 52403.5 4473.5 17375.5 19262.3 1659.6 1.25993e+07 17264.5 14963.1 8146.6 1.88908e+07 8041.4 6.30784e+06 10037.2 7076.23 1.67936e+07 6647.46 1.67936e+07 4549.2 1.88908e+07 45170.5 830881 1.17631e+06 8.13573e+06 3.07947e+07 nan nan nan
75982 20230919.7 28295.7 2.30851e+07 0.840105 80861.5 14089.1 1.88908e+07 14135.1 1.05021e+07 45709 48360.6 15735.7 1.25993e+07 54236.9 4365.6 17354.5 19313.6 1446.1 1.25993e+07 17132.2 14910.1 6835.89 1.67936e+07 7042.2 6.30784e+06 10279.6 5812.75 1.67936e+07 5495.53 1.67936e+07 4008.03 1.67936e+07 42792 840073 1.18327e+06 8.13421e+06 3.09193e+07 5603.04 8.59996e+07 17087.3

images

@eddyashton eddyashton merged commit d94b3f5 into microsoft:main Sep 19, 2023
eddyashton added a commit that referenced this pull request Sep 21, 2023
achamayou pushed a commit to achamayou/CCF that referenced this pull request Oct 24, 2023
(cherry picked from commit d94b3f5)

# Conflicts:
#	tests/js-custom-authorization/custom_authorization.py
@achamayou achamayou added the backported This PR was successfully backported to LTS branch label Oct 24, 2023
achamayou added a commit that referenced this pull request Oct 24, 2023
#5766)

Co-authored-by: Eddy Ashton <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backported This PR was successfully backported to LTS branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants