You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Precentage of problems each solver is able to solve:
default
high_accuracy
low_accuracy
mid_accuracy
clarabel
94
94
94
94
cvxopt
94
0
91
47
daqp
94
94
94
94
ecos
58
0
25
5
highs
98
0
94
0
osqp
100
86
88
73
piqp
97
95
100
97
qpalm
100
97
59
98
quadprog
94
94
94
94
scs
100
100
100
100
Rows are solvers and columns are settings. We consider that a solver successfully solved a problem when (1) it returned with a success status and (2) its solution satisfies optimality conditions within tolerance. The second table below summarizes the frequency at which solvers return success (1) and the corresponding solution did indeed pass tolerance checks.
Percentage of problems where "solved" return codes are correct:
default
high_accuracy
low_accuracy
mid_accuracy
clarabel
100
100
100
100
cvxopt
100
6
97
53
daqp
100
100
100
100
ecos
100
42
67
47
highs
100
2
94
2
osqp
100
94
88
80
piqp
100
100
100
100
qpalm
100
98
59
98
quadprog
100
100
100
100
scs
100
100
100
100
Computation time
We compare solver computation times over the whole test set using the shifted geometric mean. Intuitively, a solver with a shifted-geometric-mean runtime of Y is Y times slower than the best solver over the test set. See Metrics for details.
Shifted geometric mean of solver computation times (1.0 is the best):
default
high_accuracy
low_accuracy
mid_accuracy
clarabel
782.0
12.7
834.7
632.9
cvxopt
791.8
12.8
845.1
640.5
daqp
780.9
12.7
833.7
632.0
ecos
5992.5
97.1
6397.5
4850.0
highs
217.3
3.5
27.0
173.7
osqp
1.3
15.9
1.8
633.5
piqp
388.8
9.5
10.8
314.5
qpalm
1.0
3.2
1.0
1.0
quadprog
780.8
12.7
833.6
632.0
scs
3.8
1.0
2.0
5.4
Rows are solvers and columns are solver settings. The shift is $sh = 10$. As in the OSQP and ProxQP benchmarks, we assume a solver's run time is at the time limit when it fails to solve a problem.
Optimality conditions
Primal residual
The primal residual measures the maximum (equality and inequality) constraint violation in the solution returned by a solver. We use the shifted geometric mean to compare solver primal residuals over the whole test set. Intuitively, a solver with a shifted-geometric-mean primal residual of Y is Y times less precise on constraints than the best solver over the test set. See Metrics for details.
Shifted geometric means of primal residuals (1.0 is the best):
default
high_accuracy
low_accuracy
mid_accuracy
clarabel
1636.0
1.4
2513051638.0
3.9
cvxopt
1636.0
1.4
2513051638.0
3.9
daqp
1636.0
1.4
2874957038.4
3.9
ecos
11234.4
9.7
16963403349.0
26.6
highs
408.1
9.5
1.0
1.0
osqp
10.0
2.2
4251530245.6
8.3
piqp
816.8
1.1
1627.2
2.0
qpalm
1.0
1.0
1551238476.9
1.5
quadprog
1636.0
1.4
2513051638.0
3.9
scs
1.3
1.3
3849713756.5
5.1
Rows are solvers and columns are solver settings. The shift is $sh = 10$. A solver that fails to find a solution receives a primal residual equal to the full primal tolerance.
Dual residual
The dual residual measures the maximum violation of the dual feasibility condition in the solution returned by a solver. We use the shifted geometric mean to compare solver dual residuals over the whole test set. Intuitively, a solver with a shifted-geometric-mean dual residual of Y is Y times less precise on the dual feasibility condition than the best solver over the test set. See Metrics for details.
Shifted geometric means of dual residuals (1.0 is the best):
default
high_accuracy
low_accuracy
mid_accuracy
clarabel
660.5
2.9
99.2
3.1
cvxopt
660.5
2.8
99.2
3.1
daqp
660.5
2.8
99.2
3.1
ecos
4541.4
23212705.5
1478.7
25543.9
highs
164.8
23237.4
1.0
26.5
osqp
485.1
10.7
172.0
15.0
piqp
329.7
2.3
23.4
1.8
qpalm
1.0
1.0
147.4
1.0
quadprog
660.5
2.8
99.2
3.1
scs
116.7
1.0
54.3
2.9
Rows are solvers and columns are solver settings. The shift is $sh = 10$. A solver that fails to find a solution receives a dual residual equal to the full dual tolerance.
Duality gap
The duality gap measures the consistency of the primal and dual solutions returned by a solver. A duality gap close to zero ensures that the complementarity slackness optimality condition is satisfied. We use the shifted geometric mean to compare solver duality gaps over the whole test set. Intuitively, a solver with a shifted-geometric-mean duality gap of Y is Y times less precise on the complementarity slackness condition than the best solver over the test set. See Metrics for details.
Shifted geometric means of duality gaps (1.0 is the best):
default
high_accuracy
low_accuracy
mid_accuracy
clarabel
13.3
3.6
5.6
4.5
cvxopt
13.4
2690664.0
3.7
2736.9
daqp
13.3
1.0
1.0
1.0
ecos
96.0
295837766.0
307.6
300814.0
highs
3.3
579590.4
317.7
618.1
osqp
8.1
4.9
20.7
11.7
piqp
6.7
4.3
3.9
3.2
qpalm
1.0
1.2
71.9
2.3
quadprog
13.3
1.0
1.0
1.0
scs
8.6
1.2
2.6
2.8
Rows are solvers and columns are solver settings. The shift is $sh = 10$. A solver that fails to find a solution receives a duality gap equal to the full gap tolerance.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Model predictive control test set
Benchmark reports are copious as we aim to document comparison factors as much as possible. You can also jump to results directly.
Contents
Description
Problems arising from model predictive control in robotics.
Solvers
All solvers were called via qpsolvers v4.4.0.
CPU info
archarch_string_rawbitsbrand_rawcountcpuinfo_version_stringfamilyflags3dnow,3dnowprefetch,abm,adx,aes,apic,avx,avx2,bmi1,bmi2,clflush,clflushopt,clwb,cmov,cmp_legacy,cr8_legacy,cx16,cx8,de,dts,erms,f16c,fma,fpu,fxsr,ht,hypervisor,ia64,lahf_lm,lm,mca,mce,misalignsse,mmx,movbe,msr,mtrr,osvw,osxsave,pae,pat,pclmulqdq,pge,pni,popcnt,pqe,pqm,pse,pse36,rdpid,rdrnd,rdseed,sep,sepamd,serial,sha,smap,smep,ss,sse,sse2,sse4_1,sse4_2,sse4a,ssse3,tm,topoext,tsc,umip,vaes,vme,vpclmulqdq,wdt,xsavehz_actual_friendlyhz_advertised_friendlyl2_cache_associativityl2_cache_line_sizel2_cache_sizel3_cache_sizemodelpython_versionvendor_id_rawSettings
There are 4 settings: default, high_accuracy, low_accuracy and mid_accuracy. They validate solutions using the following tolerances:
dualgapprimalruntimeSolvers for each settings are configured as follows:
tol_feastol_gap_abstol_gap_relfeastoldual_tolprimal_tolfeastoldual_feasibility_toleranceprimal_feasibility_tolerancetime_limiteps_abseps_reltime_limitcheck_duality_gapeps_abseps_duality_gap_abseps_duality_gap_releps_releps_abseps_reltime_limiteps_abseps_reltime_limit_secsKnown limitations
The following issues have been identified as impacting the fairness of this benchmark. Keep them in mind when drawing conclusions from the results.
Results by settings
Default
Solvers are compared over the whole test set by shifted geometric mean (shm). Lower is better, 1.0 is the best.
High accuracy
Solvers are compared over the whole test set by shifted geometric mean (shm). Lower is better, 1.0 is the best.
Low accuracy
Solvers are compared over the whole test set by shifted geometric mean (shm). Lower is better, 1.0 is the best.
Mid accuracy
Solvers are compared over the whole test set by shifted geometric mean (shm). Lower is better, 1.0 is the best.
Results by metric
Success rate
Precentage of problems each solver is able to solve:
Rows are solvers and columns are settings. We consider that a solver successfully solved a problem when (1) it returned with a success status and (2) its solution satisfies optimality conditions within tolerance. The second table below summarizes the frequency at which solvers return success (1) and the corresponding solution did indeed pass tolerance checks.
Percentage of problems where "solved" return codes are correct:
Computation time
We compare solver computation times over the whole test set using the shifted geometric mean. Intuitively, a solver with a shifted-geometric-mean runtime of Y is Y times slower than the best solver over the test set. See Metrics for details.
Shifted geometric mean of solver computation times (1.0 is the best):
Rows are solvers and columns are solver settings. The shift is$sh = 10$ . As in the OSQP and ProxQP benchmarks, we assume a solver's run time is at the time limit when it fails to solve a problem.
Optimality conditions
Primal residual
The primal residual measures the maximum (equality and inequality) constraint violation in the solution returned by a solver. We use the shifted geometric mean to compare solver primal residuals over the whole test set. Intuitively, a solver with a shifted-geometric-mean primal residual of Y is Y times less precise on constraints than the best solver over the test set. See Metrics for details.
Shifted geometric means of primal residuals (1.0 is the best):
Rows are solvers and columns are solver settings. The shift is$sh = 10$ . A solver that fails to find a solution receives a primal residual equal to the full primal tolerance.
Dual residual
The dual residual measures the maximum violation of the dual feasibility condition in the solution returned by a solver. We use the shifted geometric mean to compare solver dual residuals over the whole test set. Intuitively, a solver with a shifted-geometric-mean dual residual of Y is Y times less precise on the dual feasibility condition than the best solver over the test set. See Metrics for details.
Shifted geometric means of dual residuals (1.0 is the best):
Rows are solvers and columns are solver settings. The shift is$sh = 10$ . A solver that fails to find a solution receives a dual residual equal to the full dual tolerance.
Duality gap
The duality gap measures the consistency of the primal and dual solutions returned by a solver. A duality gap close to zero ensures that the complementarity slackness optimality condition is satisfied. We use the shifted geometric mean to compare solver duality gaps over the whole test set. Intuitively, a solver with a shifted-geometric-mean duality gap of Y is Y times less precise on the complementarity slackness condition than the best solver over the test set. See Metrics for details.
Shifted geometric means of duality gaps (1.0 is the best):
Rows are solvers and columns are solver settings. The shift is$sh = 10$ . A solver that fails to find a solution receives a duality gap equal to the full gap tolerance.
Beta Was this translation helpful? Give feedback.
All reactions