SERIES INTRO! # This is the first of eighteen videos over which we build an app for the awesome Rebrickable® community, which finds multiple LEGO® sets buildable concurrently from a user’s part inventory.

We sketch a plan for building our optimization application, starting with a bare-bones optimization model and then adding a data connection and simple text-based UI. Later we swap out the AMPL scripting language for Python and the text-based UI for Streamlit, and then containerize the app.

We walk through our LEGO data from Rebrickable.com and - using GraphXR for exploratory data analysis (EDA) - we translate its schema into an Entity Relationship Diagram, yielding our optimization app’s main entities! (We also meet Legolas Greenleaf, Frankie Poullain and The Darkness …)

Step-by-step we ideate model variables and constraints, jotting these down initially as natural language comments in our new AMPL model.mod file.

We establish a connection via an import script (written in the AMPL scripting syntax, later to be replaced by a Python script) to a collection of CSV and flat text data files.

We discuss why grouping sets of single entities into aggregate sets of tuples (aka ‘cartesian products’, ‘cross-joins’) dramatically increases the performance of our optimization app. Also, we discuss the accompanying increase in data memory footprint.

We add a simple but functional, attractive text-based user interface by way of a flat data file formatted for good readability and manual data input.

Optimization programming is an extremely declarative paradigm which is beyond fantastic and also exactly why many newcomers get into trouble.

One-by-one we convert our natural language comments representing model vars and constraints into actual model entities. In the process, we add display statements for health-checking the memory size and cardinality of our new model element.

An overview of the modern optimization toolchain typically deployed in commercial applications: data sources, glue languages, modeling langs/tools, solvers and optional user interfaces.

With constraints in place which ensure that selected LEGO® sets are complete, we now implement constraints limiting the search space to sets that the user desires. We then observe that, while functional, a faster-computing way to implement such ‘problem shrinking’ is via set manipulation rather than constraint imposition.

We add functionality to our script that drops unneeded constraints per user-specified options, effectively changing the form of our optimization model on-the-fly. We also discuss extreme approaches to such run-time model construction, in light of the modular nature of optimization model components.

A close look at large-scale optimization modeling tools and languages, and linear and nonlinear solvers.

Discussing common data sources for optimization models and their various means of connection to the model.

We explore the solving-time impact of our linear solver’s (HiGHS) various params/options, and use these to tune the solver for significantly faster performance on our optimization problem.

Having worked thus far in only the AMPL modeling and scripting languages respectively, we jettison the AMPL scripting language and replace it with Python. This transforms our model into a component callable from within Python via the AMPL-Python API.

A look at practical as well as fancy means by which humans can interact with optimization models during development and in production.

Discussing the use and interchangeability of general languages which ‘glue’ optimization application components together and implement out-of-model app logic.

Now that our optimization app’s non-model logic script (aka ’the main script’) is in Python, we add a lovely Streamlit UI to replace our text-based UI, created with help from our cheerful AI coding assistant.

Our new Streamlit interface makes it easy to imagine and implement an important new feature: the ability to ‘pin’ one or more suggested LEGO® sets, discarding the others, and resolving, so that the user can iteratively arrive at an attractive collection of buildable sets.

An unpacking of the five major components of an optimization model: SETS, parameters, Variables, Objective and Constraints.

Extolling the value of tools like Git, fast code/text editors, exploratory data analysis (EDA) tools like GraphXR, whiteboarding tools like Whimsical, containerization, model/data separation and friendly syntax and naming conventions.

We are almost done with our Multi-Build optimization app! Now is a good time to containerize our app so that it can easily be deployed to various environments and even scaled across multiple heterogeneous environments at once.

Now, let’s dream out loud about some fast hardware platforms to run our app. Physical memory allotment, CPU thread speed, hard disk speed, non-optimization process load, networking and remote database access bandwidth requirements … all impact our hardware specifications.

I reveal some undisclosed turns taken during my initial development of this app. Notably: an abandoned attempt to implement a recursive search for nested sets within sets that would potentially have enabled some cool additional features, and which I might return to later.

A recap of the main points of optimization craft covered in this series, including every major speed-enhancing manuever. I also ponder out loud what interactions with the Rebrickable team might hold for the apps real-world deployment and how to approach this interaction as good citizens.

My perspective on where and why mathematical optimization methods are commonly implemented in the commercial wild, sometimes as the Prescriptive component following earlier Predictive and/or Descriptive components in a modern Data Science pipeline.

A whirlwind tour through the main problem types in the field of Combinatorial Optimization with an emphasis on their various real-world commercial implementations.

Discussing the attributes of an optimization problem which make it harder to solve: combinatorial explosions, a lack of exploitable graph or other structure, etc. Also, a look at the usefulness of concepts like NP-hardness and Big O in addressing optimization problem hardness.

A look at various optimization methods such as Stochastic data perturbation, dynamic solving, multi-objective optimization, pattern generation, heuristic and Metaheuristic solvers, and more.

An invitation to consider the application of mathematical optimization in light of the view expressed by Kenneth O. Stanley and Joel Lehman in their book ‘Why Greatness Cannot Be Planned’ (2015). I posit that a richness of possible solutions can in many use cases be more valuable than a closer-to-optimum objective value.

A methodical walk across the optimization toolchain and down through the underlying stack/OS in search of performance bottlenecks and their remedies.

My perspective on the markets for optimization products, services and skills in 2024 and beyond. I address developers, students, tool buyers and vendors in light of the current AI explosion.

A show-and-tell discussion of generative AI’s current value in writing optimization models, data integrations, tests, UIs. Also some discussion about readability of natural vs AML language for humans.

Thanks to the folks without whom this would not happen: Seth Godin, Steve Walters, The Black Crowes, Martha Wells and Murderbot, Techno Tim, Christian Lempa, Veronica Explains, Lawrence Systems, Jeff Delaney and Fireship.io, Nuno Coração and Blowfish, the Rebrickable Team, Rebecca and you.

A call to the optimization community to build valuable resources together: models and integrations, open source solvers, a pro-bono project marketplace, tool-agnostic discussion forum(s).