Speed

We establish a connection via an import script (written in the AMPL scripting syntax, later to be replaced by a Python script) to a collection of CSV and flat text data files.

We discuss why grouping sets of single entities into aggregate sets of tuples (aka ‘cartesian products’, ‘cross-joins’) dramatically increases the performance of our optimization app. Also, we discuss the accompanying increase in data memory footprint.

With constraints in place which ensure that selected LEGO® sets are complete, we now implement constraints limiting the search space to sets that the user desires. We then observe that, while functional, a faster-computing way to implement such ‘problem shrinking’ is via set manipulation rather than constraint imposition.

A close look at large-scale optimization modeling tools and languages, and linear and nonlinear solvers.

We explore the solving-time impact of our linear solver’s (HiGHS) various params/options, and use these to tune the solver for significantly faster performance on our optimization problem.

Now, let’s dream out loud about some fast hardware platforms to run our app. Physical memory allotment, CPU thread speed, hard disk speed, non-optimization process load, networking and remote database access bandwidth requirements … all impact our hardware specifications.

I reveal some undisclosed turns taken during my initial development of this app. Notably: an abandoned attempt to implement a recursive search for nested sets within sets that would potentially have enabled some cool additional features, and which I might return to later.

A recap of the main points of optimization craft covered in this series, including every major speed-enhancing manuever. I also ponder out loud what interactions with the Rebrickable team might hold for the apps real-world deployment and how to approach this interaction as good citizens.

A methodical walk across the optimization toolchain and down through the underlying stack/OS in search of performance bottlenecks and their remedies.