Enable creating of NL files with binary data

[whart222]

This should provide a significant speed enhancement.

[ghackebeil]

I think this is a good spot to attach the scripts I used to generate lower bounds on file-based model output. The "fast MPS writer" is written in C and interfaced with Python using the cffi module. To build the interface run the file build_fast_mps.py. Then you can run test_fast_mps.py to see the timing comparison. Comments at the top of build_fast_mps.py explain what is in that file. Note that the fast MPS writer is called by passing raw NumPy buffers to the C function (there is no copying of data involved).

By tweaking the fast MPS implementation (explained in the comments in that file), I came up with the following timing results for a large linear system:

• The NL, LP, and MPS writers are within a factor of 1.3x of each other
• A fast MPS writer (written in C) can be a factor of 5-8x faster than our Python writers
• An additional order of magnitude speedup (on top of the previous speedup) can be gained if the floating point values (doubles) can be written as raw binary data.
• An additional order of magnitude on top of that is gained by not writing the floating point data at all (not feasible).

Thus, I predict that a binary NL writer (designed for a flattened LP Pyomo representation), could potentially be 50-80x faster than our current writers. It wouldn't be that different from the MPS writer, as linear constraints are expressed in a sparse matrix format in the "J" section of NL files.

These numbers were generated by testing a random sparse linear system with the following characteristics:

• (nrows, ncols): (500000, 500000)
• nnz: 25000000
• avg nnz per col: 50
• avg nnz per row: 50
• nnz in objective: 67917

Attached Python files have a .txt extension added to them so that GitHub won't complain.

build_fast_mps.py.txt
test_fast_mps.py.txt

[vvvol]

Dear developers, when we can expect to have feature of writing binary NL-file by Pyomo?
It is rather tediously to wait NL-file in for 5 minutes or more (for big models).
May be it has been implemented already? We can't find in documentation...

[jsiirola]

There is currently no estimated date for the availability of this feature. It is on the list of things that we would all like to have, but to my knowledge, no one is currently working on it.

PRs are always welcome.

[jsiirola]

Archived on the master Performance Proposals Issue (#1430). Closing this performance proposal until active development has begun.