Add failing scenario to test CI is red on trace validation failure

[achamayou]

No description provided.

[ghost]

test_trace_failure_is_red_in_ci@77103 aka 20231011.13 vs main ewma over 20 builds from 76737 to 77087

Click to see table

main

build_id	build_number	Commit latency factor	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	pi_basic_mt_sgx_cft^	pi_basic_mt_sgx_cft_mem	pi_ls_sgx_cft^	pi_ls_sgx_cft_mem	pi_basic_sgx_cft^	pi_basic_sgx_cft_mem	pi_basic_mt_virtual_cft^	pi_basic_js_sgx_cft^	pi_basic_js_sgx_cft_mem	tpcc_virtual_cft^	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	pi_ls_jwt_sgx_cft^	pi_ls_jwt_sgx_cft_mem	ls_virtual_cft^	pi_ls_virtual_cft^	ls_js_sgx_cft^	ls_js_sgx_cft_mem	pi_basic_virtual_cft^	pi_basic_js_virtual_cft^	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_jwt_virtual_cft^	pi_ls_jwt_virtual_cft^	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	ls_js_virtual_cft^	ls_full_js_virtual_cft^	ls_js_jwt_virtual_cft^	hist_sgx_cft^	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
76737	20231003.4	0.819855	5602.23	8.59996e+07	14099.4	1.67936e+07	27593.5	2.30851e+07	14185.4	1.05021e+07	15720.3	1.46964e+07	64150.5	1442.7	1.25993e+07	17191.7	6888.2	1.88908e+07	6947.2	6.30784e+06	45892.7	48119	5809.12	1.67936e+07	54628	4416	5745.68	1.67936e+07	17162	19516.8	3987.27	1.67936e+07	17353	14663.4	9885.05	45195.8	833711	1.17516e+06	8.15378e+06	3.0875e+07
76798	20231003.31	0.832841	5507.34	8.59996e+07	13926.2	1.88908e+07	27510.7	2.51822e+07	14064.4	1.05021e+07	15503.8	1.25993e+07	81597	1426.1	1.25993e+07	17353.1	6835.05	1.88908e+07	6934.2	6.30784e+06	45905.3	48154	5797.3	1.67936e+07	53871	4439.9	5480.27	1.67936e+07	17065.3	19658.3	3989.1	1.67936e+07	17561	15029.1	10165.7	43728.3	828433	1.17822e+06	8.15582e+06	3.11739e+07
76817	20231004.1	0.780939	5626.78	8.59996e+07	13971.1	1.88908e+07	27941.5	2.51822e+07	14077.8	1.05021e+07	15496.4	1.46964e+07	69954.7	1437	1.05021e+07	17068.1	7251.7	1.67936e+07	6941.9	6.30784e+06	43943.3	47753	5773.46	1.67936e+07	54465.2	4457.9	5738.5	1.67936e+07	17248	19168.1	3993.81	1.67936e+07	17214.9	16923.5	9820.54	41295.8	841933	1.17864e+06	8.15569e+06	3.07938e+07
76832	20231004.7	0.791761	5615.29	8.59996e+07	13975.4	1.67936e+07	27629.2	2.30851e+07	14056.3	1.05021e+07	15601.7	1.25993e+07	68893.5	1434	1.25993e+07	17248.6	6873.72	1.88908e+07	7089.4	6.30784e+06	43721.4	48138.9	5806.39	1.67936e+07	46993.4	4444.2	5726.45	1.67936e+07	17038.4	19713.4	3976.51	1.67936e+07	17646.2	14882.5	9901.95	42527.8	835979	1.17914e+06	8.15303e+06	3.17647e+07
76869	20231004.22	0.806157	5552.49	8.59996e+07	14003.1	1.88908e+07	27762.1	2.51822e+07	14065.4	1.05021e+07	15372.4	1.46964e+07	70744.9	1427	1.25993e+07	17245.9	6868.91	1.88908e+07	7022	6.30784e+06	45658.4	48077.4	5804.63	1.67936e+07	54561.1	4464.3	5756.12	1.67936e+07	17322.8	18923.3	3991.19	1.67936e+07	17202.1	15082.4	9913.2	45382.2	834342	1.18098e+06	8.14978e+06	3.08076e+07
76888	20231004.30	0.810034	5604.3	8.59996e+07	13985	1.88908e+07	27780.7	2.30851e+07	14115.9	1.05021e+07	15556.5	1.46964e+07	67387.1	1434.9	1.25993e+07	17390.9	6863.79	1.67936e+07	7032.1	6.30784e+06	45776.9	48301	5815.76	1.67936e+07	51883.1	4390.1	5751.88	1.67936e+07	17189.2	20048.9	3969.63	1.67936e+07	15378.1	14886.4	10356.8	42002.2	823286	1.18056e+06	8.1555e+06	3.03318e+07
76915	20231004.42	0.774478	5624.97	8.59996e+07	14042.4	1.88908e+07	27485.2	2.51822e+07	14174.9	1.05021e+07	15627.4	1.25993e+07	80712.1	1433.2	1.25993e+07	17274.8	6910.89	1.88908e+07	6998.2	6.30784e+06	46033.9	48315.9	5784.96	1.67936e+07	53167.5	4402.4	5748.49	1.67936e+07	17273.2	19310.2	4003.02	1.67936e+07	17384.9	14995.5	10159.5	40358.8	835408	1.17924e+06	8.12898e+06	3.07817e+07
76922	20231005.1	0.810974	5626.69	8.59996e+07	14039	1.88908e+07	28029.7	2.51822e+07	14080.5	1.05021e+07	15577.1	1.46964e+07	65412.6	1432.3	1.25993e+07	17459.2	6863.93	1.67936e+07	7085.2	6.30784e+06	45898.2	48742	5810.75	1.67936e+07	54118.6	4456.3	5719.35	1.67936e+07	17231.8	20034.9	3992.52	1.67936e+07	17323.3	14832.9	10275.8	42574.2	829315	1.17285e+06	8.14651e+06	3.07836e+07
76933	20231005.5	0.805698	5569.11	8.59996e+07	14005.2	1.88908e+07	27882.8	2.30851e+07	14146.2	1.05021e+07	15512.2	1.46964e+07	81465	1432.2	1.25993e+07	17274.6	6865.95	1.88908e+07	6977.1	6.30784e+06	45946.3	48589.4	5765.37	1.67936e+07	54166.2	4463.6	5766.41	1.67936e+07	17329.5	19980	3994.82	1.67936e+07	17094.5	16949.3	9763.9	39270.6	837034	1.18428e+06	8.10377e+06	3.25923e+07
76941	20231006.1	0.799935	5581.89	8.59996e+07	14055.4	1.88908e+07	28097.1	2.30851e+07	14129.3	1.05021e+07	15654.4	1.25993e+07	65485.1	1434.8	1.25993e+07	17296.4	6890.62	1.67936e+07	6976.6	6.30784e+06	45618.5	48175.1	5781.49	1.67936e+07	53457.3	4479	5744.04	1.67936e+07	17334.7	19718.8	4009.45	1.67936e+07	17075.6	14603.4	9842.12	39294.9	839192	1.18477e+06	8.15264e+06	3.23033e+07
76953	20231006.6	0.797479	5610.73	8.59996e+07	14024	1.88908e+07	27710.2	2.51822e+07	14122.5	1.05021e+07	15585.3	1.25993e+07	78324.4	1444.6	1.25993e+07	16945.5	7252.76	1.67936e+07	7058.2	6.30784e+06	43917.9	48958.9	5820.86	1.67936e+07	53263.2	4423.9	5773.73	1.67936e+07	17245.1	18721.7	4008.93	1.67936e+07	17292	14773.2	9748.83	45657	835912	1.17953e+06	8.07625e+06	2.6895e+07
76967	20231006.11	0.799426	5542.39	8.59996e+07	13951.2	1.88908e+07	28032.2	2.30851e+07	14077.9	1.05021e+07	15519.1	1.25993e+07	64719	1421.7	1.25993e+07	17341.4	6820.04	1.88908e+07	6884.1	6.30784e+06	45691.3	48875.1	5770.08	1.67936e+07	54004.5	4469.9	5760.09	1.88908e+07	17300.7	19158.2	4004.62	1.67936e+07	16934.3	14583.1	9795.4	47130.7	835081	1.17659e+06	8.17304e+06	3.09852e+07
76976	20231007.2	0.797827	5640.74	8.59996e+07	14032.5	1.88908e+07	27833	2.51822e+07	14190	1.05021e+07	15631.1	1.46964e+07	69696.2	1433.9	1.25993e+07	17276.4	7287.64	1.67936e+07	7044.3	6.30784e+06	45702.6	48619	5786.37	1.67936e+07	53436.8	4409.7	5747.56	1.67936e+07	17184.3	19733.2	4006.09	1.67936e+07	17169.5	14871.6	9779.02	45049.2	830418	1.18267e+06	8.17291e+06	3.07891e+07
76982	20231009.2	0.792855	5580.01	8.59996e+07	13963.1	1.88908e+07	27935.7	2.51822e+07	14141.8	1.05021e+07	15380.7	1.25993e+07	81026	1439.9	1.25993e+07	17231.2	7196.39	1.67936e+07	7023.9	6.30784e+06	45523.7	48581	5773.77	1.67936e+07	54131	4253.1	5474.35	1.67936e+07	17397.5	19156	3981.48	1.67936e+07	17135.7	16869.3	9793.25	46259.3	832879	1.18218e+06	8.14943e+06	3.15184e+07
76994	20231009.5	0.799524	5591.29	8.59996e+07	13986.8	1.88908e+07	27792.2	2.30851e+07	14144	1.05021e+07	15585.3	1.25993e+07	88389.5	1431.3	1.25993e+07	17106.4	6852.61	1.67936e+07	6940.5	6.30784e+06	45848	48779.1	5808.48	1.67936e+07	53666.3	4397.6	5783.34	1.67936e+07	17242	19596.7	3987.05	1.67936e+07	16914.5	16715.5	9855.69	44311.9	831717	1.18619e+06	8.15293e+06	3.07093e+07
77015	20231010.4	0.83228	5592.78	8.59996e+07	14023.3	1.88908e+07	27777.8	2.30851e+07	14107	1.05021e+07	15597.5	1.25993e+07	62049.7	1436	1.25993e+07	17372.9	6898.77	1.88908e+07	7117.8	6.30784e+06	45785.1	48171.9	5804.14	1.67936e+07	53651.8	4441.2	5786.43	1.67936e+07	17407.2	19739.4	4016.78	1.67936e+07	17105.5	16857.9	9954.93	41221.6	825345	1.18742e+06	8.12844e+06	3.06711e+07
77052	20231010.17	0.805827	5614.03	8.59996e+07	14004.1	1.88908e+07	27851.7	2.51822e+07	14072.2	1.05021e+07	15532.7	1.25993e+07	87071.8	1434.8	1.25993e+07	17123.9	6879.16	1.88908e+07	6931.4	6.30784e+06	45707.2	48521.1	5811.67	1.67936e+07	53548.8	4435.4	5725.25	1.88908e+07	17238.9	19725.4	3978.87	1.67936e+07	17064.9	16932.1	9900.42	39661.9	835356	1.17508e+06	8.12795e+06	3.02453e+07
77065	20231010.21	0.805254	5593.79	8.59996e+07	13984.2	1.88908e+07	27866.2	2.51822e+07	14076.6	1.05021e+07	15457.8	1.46964e+07	76791.9	1431.7	1.25993e+07	17453.5	6839.77	1.67936e+07	6882.8	6.30784e+06	45981.3	47639.7	5802.59	1.67936e+07	53782.5	4471.3	5713.9	1.67936e+07	17159.7	19727	3982.91	1.67936e+07	17221.9	16835.2	9888.21	47354.4	831464	1.17023e+06	8.14609e+06	3.06028e+07
77080	20231011.6	0.805739	5614.98	8.59996e+07	14016.4	1.67936e+07	28231.3	2.30851e+07	14058.7	1.05021e+07	15607.1	1.25993e+07	77100.4	1435.6	1.25993e+07	17356.8	6865.39	1.88908e+07	6917	6.30784e+06	45541.3	47443.3	5768.86	1.67936e+07	53459.3	4480.6	5749.11	1.67936e+07	17192.9	19626.7	3974.5	1.67936e+07	16932.3	16704.5	9841.59	44338.6	839572	1.1863e+06	8.14557e+06	3.07489e+07
77087	20231011.8	0.821071	5620.28	8.59996e+07	14043.4	1.88908e+07	27891.2	2.51822e+07	14129.4	1.05021e+07	15619	1.46964e+07	83135	1427.7	1.25993e+07	17648.9	6872.29	1.67936e+07	7079.7	6.30784e+06	45716.6	48214.1	5780.82	1.67936e+07	53475.5	4431.6	5765.08	1.67936e+07	17316.9	19004.2	3974.44	1.67936e+07	17758.2	16880.7	9849.81	40652.5	827510	1.16961e+06	8.14036e+06	3.30899e+07

test_trace_failure_is_red_in_ci

build_id	build_number	Commit latency factor	tpcc_sgx_cft^	tpcc_sgx_cft_mem	pi_basic_mt_sgx_cft^	pi_basic_mt_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	pi_basic_mt_virtual_cft^	pi_ls_sgx_cft^	pi_ls_sgx_cft_mem	pi_basic_sgx_cft^	pi_basic_sgx_cft_mem	tpcc_virtual_cft^	ls_virtual_cft^	pi_ls_virtual_cft^	pi_basic_virtual_cft^	pi_basic_js_sgx_cft^	pi_basic_js_sgx_cft_mem	pi_basic_js_virtual_cft^	ls_jwt_virtual_cft^	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	pi_ls_jwt_virtual_cft^	pi_ls_jwt_sgx_cft^	pi_ls_jwt_sgx_cft_mem	ls_js_virtual_cft^	ls_full_js_virtual_cft^	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_js_jwt_virtual_cft^	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	hist_sgx_cft^	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
77085	20231011.7	0.761125	5632.94	8.59996e+07	27994.9	2.51822e+07	14014.2	1.88908e+07	64963.3	14143.9	1.05021e+07	15575	1.25993e+07	17373	45591.7	44733.3	52712	1433.9	1.25993e+07	4417.1	17306.4	6868.31	1.88908e+07	19235.6	7033.2	6.30784e+06	17147.1	16744.1	5773.86	1.67936e+07	9808.71	5752.37	1.67936e+07	4005.3	1.67936e+07	45424.4	838414	1.18094e+06	8.12557e+06	3.07748e+07
77103	20231011.13	0.825877	5586.21	8.59996e+07	27609.4	2.51822e+07	14000.1	1.67936e+07	59046.8	14130.7	1.05021e+07	15592.6	1.46964e+07	17339.1	45810	48478.5	53630.8	1428.7	1.25993e+07	4428.3	17395.5	6838.14	1.67936e+07	16146.5	6452.5	6.30784e+06	17081.4	16946.4	5788.72	1.67936e+07	9855.83	5495.61	1.67936e+07	4012.66	1.67936e+07	43746.5	831768	1.18156e+06	8.13573e+06	3.06688e+07

[achamayou]

CI is not red, despite trace validation clearly failing:

<<"Trace:", "../build/election_while_reconfiguration.ndjson", "Length:", 158>>
Computing initial states...
Finished computing initial states: 1 distinct state generated at 2023-10-11 08:27:02.
Error: Temporal properties were violated.

Error: The following behavior constitutes a counter-example:

State 1: <Initial predicate>

[achamayou]

I always believed that the script: | task would fail if a single line returned non-zero, but that's clearly not the case. I can't find any mention of whether this is intentional under https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps-script?view=azure-pipelines

The lesson seems to be that every time we have a script: | task, we need to set -e in the first line.