Add full KKT differentiation mode, split into new-style PyTorch Module/Function, automatically infer some batch sizes

[bamos]

No description provided.

[bamos]

For some more details, the full backwards pass does something like this (can more properly write out if we end up using this):

I've added this mode to the simple test we have, and there's also a hack in here to handle the bounds at \pm\infty for now.

Something that I needed to do to get the active set method working is use LSQR to find the least-squares solution for some near-indefinite cases that the full solve failed on, although perhaps regularizing the system and doing a direct solve's OK too.

Also I've added in a --qp-solver osqpth option to the OptNet sudoku experiment here that by default uses the full backward pass and converges as with qpth. In comparison to what I sent in #4, this is now converging:

Using the full KKT system

(Keeps converging as we would expect)

Using the active set KKT system

---

So from here, I'll compare the derivatives from the active and full backwards passes for this training run and pull out a specific QP that has mis-matching derivatives to further debug what's going on. Maybe we can review/merge in this PR in and further diagnose/discuss this in #6?

[bstellato]

Sure thanks! Apologies for the delay. It is still quite obscure to me why this happens.