8

We have a large-scale optimization problem (~10K vars and ~10K constraints) in the form of LP format file (generated using Cplex library).

We wanted to solve that problem file using Cvxpy (with Gurobi solver - Note: Cvxpy is unavoidable), which doesn't accepts LP format file directly (rather constraint matrices/list).

So, is it possible to somehow read (/transform/parse) that LP format file into regular Numpy matrices?

pqrz
  • 470
  • 2
  • 7
  • 1
    Careful. 1) cvxpy is a modelling tool (with tons of overhead due to it's powerful concepts) and if you already have a LP modelled, it's questionable what cvxpy would offer here. It's basically a slow wrapper then (potentially destroying your problem in some more complex QP/PSD cases -> internal eigenvalue-checks). 2) Numpy only knows dense arrays. Most real-world LP-instances would not fit into your memory when stored densely. scipy.sparse is what you are interested in (cvxpy supports both). 3) The task itself is 99% LP-format parsing, so focus on finding an accessible parser (in python). – sascha Feb 28 '22 at 20:15
  • Hi @sascha , Thanks! 1) We aren't looking for value addition, just wanted to profile the problems in Cvxpy. 2) Yeah both Numpy or scipy.sparse would suffice. 3) Exactly, its about LP-format parsing, I tried out https://github.com/aphi/Lp-Parser but it fails (not robust on large-scale problems) – pqrz Mar 01 '22 at 02:24
  • If you already have an .lp file (let's call it prob.lp), and you want to use gurobi, then why not just say m = read('probl.lp') and then m.optimize()? – EhsanK Mar 01 '22 at 03:10
  • 1
    Please Note: Cvxpy is unavoidable because Cvxpy does clever transformations which stabilizes and significantly reduces runtime for our large-scale problem – pqrz Mar 01 '22 at 03:38
  • In general - methods that avoids the real problem of back-transformation from lp format to python matrices would not work for us. – pqrz Mar 01 '22 at 03:55
  • 1
    That Cvxpy should do transformations a presolve in good a commercial optimizer does not do sound right. – ErlingMOSEK Mar 01 '22 at 09:19
  • 1
    I agree with @ErlingMOSEK. To answer the question though, I would write a parser using the original CPLEX model rather than the LP format, because that will be a lot of work. – Richard Mar 01 '22 at 09:53
  • @ErlingMOSEK - Please don't mind, I would slightly differ because - 1) From my experience, its not an either-or situation, rather both entities when ran successfully gives superior performance. 2) Commerical solver bills by runtime and we find Cvxpy's transformed problem decreases our runtime then raw problem – pqrz Mar 01 '22 at 10:15
  • 1
    The purpose of the Cvxpy transformations is to setup the data on the form that optimizer needs. Not to improve the formulation IMO. I think even the Cvxpy authors would be surprised by your statement/observation. – ErlingMOSEK Mar 01 '22 at 10:22
  • @Richard - Yes, writing a parser is a solution. But I found this to not a new problem, (given commerical solvers does LP parsing behind the scenes). Are you sure - we don't we a readymade solution for this? – pqrz Mar 01 '22 at 10:22
  • @ErlingMOSEK - I think you are right, there seems to be a gap b/w intent and results. I'll investigate further, if my results were fluke / too small to generalize. Otherwise would open a discussion thread on this with Cvxpy team. Thanks for bringing this up :) – pqrz Mar 01 '22 at 10:54
  • 1
    Others - I'll request to please come back to the original problem of back transformation of LP format parsing to numpy/scipy matrices. Especially, Do we have any ready made solution on this? – pqrz Mar 01 '22 at 10:58
  • 2
    There is this thing called performance variability in MIP (google it). For instance if CVXPY permutes the variables and or constraints you may see a big change in solution time. It is a random effect though. – ErlingMOSEK Mar 01 '22 at 10:58

1 Answers1

8

As has been discussed in the comments already, your suggested workflow is more complicated than it needs to be without providing any advantages. Gurobi is perfectly capable of handling LP files and if you want to solve the problem with Gurobi anyway there is no reasonable benefit of routing everything through cvxpy.

If you absolutely have to get the constraint matrix, then I still suggest using Gurobi:

import gurobipy as gp

m = gp.read("model.lp")
A = m.getA()

As was already mentioned, this matrix is stored as a sparse matrix (see Gurobi documentation on Model.getA() for more details) which is vastly superior to numpy's dense storage for practically any model.

mattmilten
  • 1,633
  • 8
  • 12