Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding TLC stats to cimetrics #5807

Merged
merged 26 commits into from
Nov 7, 2023
Merged

Conversation

heidihoward
Copy link
Member

@heidihoward heidihoward commented Nov 1, 2023

This PR adds some TLC stats to the cimetrics. To keep things simple, this PR just adds the duration and number of distinct states for MCccfraft.tla but we can easily extended this.

@heidihoward heidihoward added the tla TLA+ specifications label Nov 1, 2023
@ghost
Copy link

ghost commented Nov 1, 2023

tla-cimetrics@78481 aka 20231107.14 vs main ewma over 20 builds from 78115 to 78447

Click to see table

main

build_id build_number Commit latency factor tpcc_sgx_cft^ tpcc_sgx_cft_mem pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem pi_basic_sgx_cft^ pi_basic_sgx_cft_mem tpcc_virtual_cft^ ls_virtual_cft^ pi_ls_virtual_cft^ pi_basic_virtual_cft^ pi_basic_js_sgx_cft^ pi_basic_js_sgx_cft_mem pi_basic_js_virtual_cft^ ls_jwt_virtual_cft^ ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_virtual_cft^ pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ pi_basic_mt_virtual_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
78115 20231031.5 0.788781 5597.14 8.59996e+07 28096.6 2.51822e+07 14046.4 1.88908e+07 14149.3 1.05021e+07 15573.1 1.46964e+07 17153.5 45717.6 47743.9 55284.5 1438.8 1.25993e+07 4374.5 17204.3 6857.73 1.67936e+07 19257.8 6984.1 6.30784e+06 17318.7 5786.81 1.67936e+07 14893.2 9790.65 5770.07 1.67936e+07 3994.27 1.67936e+07 44365.6 90398.6 818357 1.17717e+06 8.15176e+06 3.07813e+07
78128 20231031.10 0.75716 5630.66 8.59996e+07 27738.1 2.51822e+07 14058.5 1.88908e+07 14160.9 1.05021e+07 15631.8 1.46964e+07 17076.3 43862 47267.7 55100.9 1441.7 1.25993e+07 4371.6 17200.1 7284.99 1.67936e+07 19235.2 7044.7 6.30784e+06 17122 5805.39 1.67936e+07 14762.1 10396.2 5776.39 1.67936e+07 3998.07 1.67936e+07 42934.8 71982.3 837504 1.18196e+06 8.15251e+06 3.15981e+07
78146 20231031.17 0.780638 5598.38 8.59996e+07 27901 2.51822e+07 14060.8 1.88908e+07 14179 1.05021e+07 15535.7 1.46964e+07 17229.5 43813 47595.9 55096.1 1433.1 1.25993e+07 4355.7 17361 6790.28 1.67936e+07 19744.5 6889.6 6.30784e+06 17356.3 5772.45 1.67936e+07 14955.7 9799.34 5488.66 1.67936e+07 3966.05 1.67936e+07 43021.9 76464.7 820080 1.18077e+06 8.15047e+06 3.07762e+07
78161 20231031.23 0.782488 5605.13 8.59996e+07 28091.4 2.51822e+07 14058.7 1.88908e+07 14092.9 1.05021e+07 15472.1 1.46964e+07 17341.2 43725.6 47396.3 54809.3 1434.3 1.25993e+07 4373.5 17176.1 6855.37 1.67936e+07 19492.8 7006.3 6.30784e+06 17576.4 5771.02 1.67936e+07 14802.5 9795.27 5735.27 1.67936e+07 3972.38 1.67936e+07 46424.3 62893.4 831500 1.17237e+06 8.17199e+06 3.13663e+07
78188 20231031.34 0.777414 5602.33 8.59996e+07 27969.8 2.30851e+07 14059.1 1.88908e+07 14173.3 1.05021e+07 15545.1 1.46964e+07 17118.8 43790 47386.9 54591.4 1434.6 1.05021e+07 4355.1 17093.8 7223.17 1.67936e+07 17419.8 6938.2 6.30784e+06 17431.3 5775.87 1.67936e+07 14898.7 9859.67 5491.97 1.67936e+07 3979.22 1.67936e+07 39990.1 69302.6 830634 1.18149e+06 8.13241e+06 3.07346e+07
78196 20231101.2 0.832989 5623.15 8.59996e+07 28088.2 2.30851e+07 14070.4 1.88908e+07 14239.9 1.05021e+07 15638 1.46964e+07 17212.9 43839.1 47046.4 50979.6 1437.9 1.25993e+07 4393 17147.3 6871.58 1.88908e+07 19277.1 7054.3 6.30784e+06 17624.4 5818.22 1.67936e+07 14959.6 10005.4 5735.09 1.67936e+07 3977.77 1.67936e+07 44220.3 88490.7 834004 1.18027e+06 8.15222e+06 3.09693e+07
78218 20231101.11 0.794466 5594.46 8.59996e+07 27482.4 2.30851e+07 14033 1.88908e+07 14051.8 1.05021e+07 15531.3 1.46964e+07 17267 43523.4 39919.2 54609.1 1428.2 1.25993e+07 4338.1 17157.6 6867.98 1.67936e+07 18949.1 6933.2 6.30784e+06 17376.4 5807.77 1.67936e+07 14871.1 10180.4 5469.02 1.67936e+07 3994.19 1.67936e+07 45431.3 83063.2 833473 1.18618e+06 8.15072e+06 3.08689e+07
78226 20231101.13 0.798772 5625.43 8.59996e+07 27521.2 2.51822e+07 14043.1 1.88908e+07 14097.8 1.05021e+07 15604.3 1.46964e+07 17229.8 43650.8 46528.9 47448.2 1414.9 1.25993e+07 4341.1 17155.8 7241.73 1.67936e+07 18953.8 6889 6.30784e+06 14940.7 5756.54 1.67936e+07 14721.7 10072.6 5445.43 1.67936e+07 3965.05 1.67936e+07 46340.6 73492.7 835263 1.18362e+06 8.1689e+06 3.06211e+07
78246 20231101.20 0.784451 5643.71 8.59996e+07 27565 2.30851e+07 14099.1 1.88908e+07 14165.3 1.05021e+07 15684.7 1.46964e+07 17123.7 45792.5 46974.8 53423.6 1423.7 1.25993e+07 4408.4 17022.2 7247.5 1.67936e+07 19427.7 6996.2 6.30784e+06 15054.6 5794.78 1.67936e+07 14945.9 10275.3 5485.63 1.67936e+07 3992.72 1.67936e+07 45075.5 70500.2 837584 1.17997e+06 8.17199e+06 3.10982e+07
78261 20231101.25 0.81842 5612.19 8.59996e+07 28224.5 2.51822e+07 14117.8 1.88908e+07 14186 1.05021e+07 15647.1 1.25993e+07 17440.2 45894.6 47638.6 53299.1 1436.8 1.25993e+07 4380.3 17042 7264.49 1.67936e+07 19569.5 7000.4 6.30784e+06 17247.3 5797.39 1.67936e+07 14729.7 10279.9 5486.18 1.67936e+07 3994.48 1.67936e+07 43354.3 69365.6 835866 1.1715e+06 8.14353e+06 3.07998e+07
78284 20231102.4 0.775601 5607 8.59996e+07 27902.1 2.30851e+07 14039.4 1.88908e+07 14144.7 1.05021e+07 15534.3 1.46964e+07 17090 43810.2 47171.4 53039.6 1433.6 1.25993e+07 4380.6 16902.4 6870.67 1.88908e+07 19632.5 7007.8 6.30784e+06 14945.6 5753.55 1.67936e+07 14982 10242.7 5431.34 1.67936e+07 3982.9 1.67936e+07 40950.5 74128.5 830357 1.18333e+06 8.13473e+06 3.08373e+07
78305 20231102.12 0.78799 5520.15 8.59996e+07 27661.5 2.30851e+07 14034 1.88908e+07 14105 1.05021e+07 15532.4 1.25993e+07 17322.6 45645.4 48989.3 53847.3 1421.1 1.25993e+07 4386.2 17291.3 6844.95 1.88908e+07 19701.3 7082.4 6.30784e+06 17082.4 5797.56 1.67936e+07 14871.8 10051.9 5725.47 1.88908e+07 4001.42 1.67936e+07 40964.4 88216.8 822024 1.17786e+06 8.15592e+06 3.10807e+07
78319 20231102.17 0.793434 5616.9 8.59996e+07 27989.4 2.30851e+07 13956 1.88908e+07 14087.7 1.05021e+07 15500.2 1.46964e+07 17142.8 45692.3 48014.1 53896.9 1432 1.25993e+07 4359.7 17057.3 6839.82 1.67936e+07 19245.9 6924.1 6.30784e+06 17127.5 5801.1 1.67936e+07 14800.7 10296.3 5739.65 1.67936e+07 3972.71 1.67936e+07 45008.6 83397.2 834976 1.18569e+06 8.12273e+06 3.07789e+07
78358 20231103.3 0.80868 5613.85 8.59996e+07 27874.4 2.51822e+07 14024 1.88908e+07 14173.7 1.05021e+07 15658 1.46964e+07 17311.7 43799.2 48061.9 53988.3 1431 1.25993e+07 4395.8 17328 6848.6 1.88908e+07 19780 6936 6.30784e+06 17379.4 5795.98 1.67936e+07 14648.9 10273.9 5450.88 1.67936e+07 3999.15 1.67936e+07 45285.7 73888.9 829705 1.18299e+06 8.14787e+06 3.0569e+07
78365 20231103.6 0.787105 5629.97 8.59996e+07 28034 2.30851e+07 14078.6 1.88908e+07 14064.8 1.05021e+07 15659.9 1.25993e+07 17440.6 45729.3 48139.6 54762.3 1424 1.25993e+07 4342.9 17317.7 6812 1.67936e+07 19576 6988 6.30784e+06 15085.7 5761.67 1.67936e+07 14683.7 9954.18 5755.45 1.67936e+07 3977.18 1.67936e+07 43491.4 72764.7 829913 1.18447e+06 8.14194e+06 3.07286e+07
78372 20231103.8 0.811727 5578.73 8.59996e+07 27795.9 2.30851e+07 13977.9 1.88908e+07 14087 1.05021e+07 15549.8 1.25993e+07 17281.7 45628 48368.5 53990.3 1424.7 1.25993e+07 4355.3 17137.4 6788.55 1.88908e+07 19422.4 6925.3 6.30784e+06 14922.2 5793.13 1.67936e+07 14674.6 10171.6 5490.55 1.88908e+07 3975.67 1.67936e+07 44113 90924.9 838885 1.18174e+06 8.15439e+06 3.16504e+07
78384 20231103.12 0.802159 5517.44 8.59996e+07 27988.7 2.30851e+07 13977 1.88908e+07 14136.3 1.05021e+07 15467.4 1.46964e+07 17108.5 43878.6 47871.9 52246 1436.3 1.05021e+07 4407.6 17039.1 6845.18 1.88908e+07 19823.6 6873 6.30784e+06 17407.9 5794.5 1.67936e+07 14732.2 10283.4 5481.25 1.67936e+07 3986.29 1.67936e+07 45856.3 74382.9 827005 1.1842e+06 8.15429e+06 3.09913e+07
78407 20231103.18 0.797156 5591.14 8.59996e+07 28323.1 2.30851e+07 13982.6 1.88908e+07 14162.3 1.05021e+07 15482.2 1.25993e+07 17161.2 43521.6 46519.9 53186.1 1429.7 1.25993e+07 4343 16972 6854.13 1.67936e+07 19625.2 7087.8 6.30784e+06 14970 5753.69 1.67936e+07 14529.4 10096.1 5469.17 1.67936e+07 3986.36 1.67936e+07 40541.7 59381.6 829355 1.18365e+06 8.17382e+06 3.16191e+07
78430 20231106.1 0.785044 5655.91 8.59996e+07 27904 2.30851e+07 14049.4 1.88908e+07 14103.2 1.05021e+07 15592 1.46964e+07 17142.7 45699 47880.3 54141.9 1438.6 1.05021e+07 4390.8 17165.6 6904.68 1.88908e+07 19773.4 7080.2 6.30784e+06 15067.1 5768.13 1.67936e+07 14805.8 10259.9 5444.99 1.67936e+07 4003.81 1.67936e+07 45223.5 75314.2 830876 1.1802e+06 8.13838e+06 3.08745e+07
78447 20231107.4 0.803997 5657.58 8.59996e+07 27883.9 2.30851e+07 14082.8 1.67936e+07 14226.4 1.05021e+07 15619.5 1.46964e+07 17233.7 43850.9 46545.8 53892.3 1439.7 1.25993e+07 4342.9 16914.1 7269.63 1.67936e+07 19725.8 7062.6 6.30784e+06 17400.6 5804.83 1.67936e+07 14863.8 10109.8 5742.73 1.67936e+07 3981.14 1.67936e+07 43312.1 63226 834561 1.18331e+06 8.16776e+06 3.07268e+07

tla-cimetrics

build_id build_number pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem Commit latency factor tpcc_sgx_cft^ tpcc_sgx_cft_mem tpcc_virtual_cft^ ls_virtual_cft^ pi_ls_virtual_cft^ pi_basic_mt_virtual_cft^ pi_basic_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem pi_basic_js_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ pi_basic_js_sgx_cft^ pi_basic_js_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem hist_sgx_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^ tlc_3node_fixed_duration_s tlc_3node_fixed_states
78471 20231107.11 27991.2 2.51822e+07 0.813703 5509.02 8.59996e+07 17320.7 43874 47953.4 76520.5 54117.3 13934.4 1.88908e+07 14078.3 1.05021e+07 4429.1 15396.6 1.46964e+07 17213.1 18861.9 17553.3 14820.7 10183.4 1424.9 1.25993e+07 6818.51 1.67936e+07 6884.7 6.30784e+06 5765.88 1.67936e+07 44018.3 5480.58 1.88908e+07 3973.76 1.67936e+07 829471 1.18464e+06 8.12656e+06 3.06051e+07 nan nan
78478 20231107.13 28169.9 2.30851e+07 0.814503 5637.2 8.59996e+07 17239 45852.7 48506.7 74197.7 51902.9 14112.5 1.88908e+07 14179.2 1.05021e+07 4389.5 15599.5 1.46964e+07 17014.9 19450.3 17369.3 14752.3 10021.6 1436.6 1.25993e+07 7273.86 1.67936e+07 7006.9 6.30784e+06 5762.79 1.67936e+07 38873.3 5718.85 1.67936e+07 3975.74 1.67936e+07 815274 1.17466e+06 8.13596e+06 3.08165e+07 126 1.97092e+06
78481 20231107.14 28057.7 2.51822e+07 0.795 5554 8.59996e+07 17699.6 43933.9 47745.7 85400.7 53798.5 14058.8 1.88908e+07 14168.4 1.05021e+07 4413 15581 1.25993e+07 17202.1 18938.7 17294.7 14965.5 10291 1431.8 1.25993e+07 6887.13 1.88908e+07 6932 6.30784e+06 5789.38 1.67936e+07 38845.2 5487 1.67936e+07 3998.92 1.67936e+07 830279 1.17892e+06 8.1555e+06 3.06844e+07 122 1.97092e+06

images

lemmy added a commit to tlaplus/CommunityModules that referenced this pull request Nov 1, 2023
@heidihoward
Copy link
Member Author

@eddyashton @achamayou Still a WIP but I'd appreciate feedback at this stage. I've deliberately omitted METRICS_MONGO_CONNECTION from the gh action to avoid actually posting data until we are ready

RonSheely pushed a commit to RonSheely/tlaplus that referenced this pull request Nov 3, 2023
@achamayou
Copy link
Member

@heidihoward if you would like a different category, you can add a line in https://github.com/microsoft/CCF/blob/main/metrics.yml, something like "TLC": "tlc_.*", before the "Others" catch-all (they are evaluated in order).

@heidihoward
Copy link
Member Author

@heidihoward if you would like a different category, you can add a line in https://github.com/microsoft/CCF/blob/main/metrics.yml, something like "TLC": "tlc_.*", before the "Others" catch-all (they are evaluated in order).

done in e1e3cc4

@heidihoward heidihoward changed the title WIP: Adding TLC stats to cimetrics Adding TLC stats to cimetrics Nov 7, 2023
@heidihoward heidihoward marked this pull request as ready for review November 7, 2023 13:56
@heidihoward heidihoward requested a review from a team November 7, 2023 13:56
@heidihoward
Copy link
Member Author

Yay, this is now working!
image

@achamayou achamayou enabled auto-merge (squash) November 7, 2023 21:39
@heidihoward
Copy link
Member Author

Looking at the CI output wheel is complaining a bit but it seems to work anyways. I think actually it might be best to switch to cimetrics_env.sh so only cimetrics is installed. I did spot this handy script the first time around but did not use it as I wrongly assumed we didn't need a virtual env.

PS: Sorry, I don't know how to post a link to ADO pipelines, copying and pasting the URL gives this fun error
image

@achamayou achamayou merged commit 6e0f675 into microsoft:main Nov 7, 2023
29 of 34 checks passed
@achamayou
Copy link
Member

@heidihoward you're absolutely right, I'd completely forgotten about that script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tla TLA+ specifications
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants