Optimal but Stupid

If you pick up most water-related journals these days, it seems as if half the papers are devoted to applying some optimization techniques to solve some real problem. However, almost none of these methods actually find their way into water engineering and operations. Why is that?

When I was a young engineer, trying to justify my work in this area, I would read what sounded like a great paper in a journal. I’d call the author to hear about their success story when they applied their paper to the real world. The response was usually, “We didn’t actually apply it anywhere. We just developed the algorithm and made up some data to test it.”

I realized that while the goal of these studies is ostensibly to solve real-world problems, the real goal is to publish papers which will lead to funding of more research studies. I remember talking with one highly regarded professor who told me that his promotion and tenure are not tied to the usefulness of his work, but to how much research funding he could bring into the university. Why don’t these solutions work in practice?

There’s the story of the guy walking down the street one evening, and he comes upon a drunk on his hands and knees looking for something. He asks the drunk what he is doing and the drunk replies, “I lost a $20 bill and I’m looking for it.” The guy decides to help, but after a while, they are unsuccessful. The guy asks the drunk, “Where did you lose this money?” The drunk replies, “Over there in that vacant lot.” “Why aren’t we looking there?” The drunk replies, “It’s dark in the lot, there’s broken glass, and rusty nails, and some questionable people hang out there. The light is a lot better where we’re looking.” That’s pretty much what happens in a lot of research.

Optimization problems usually boil down to solving a mathematical problem:

• Maximize some object function(s)
• Subject to some constraints

The researcher starts with a real problem and needs to convert it into a set of objective functions and constraints. But that is difficult—a lot like looking for that money in the vacant lot. So, they make sufficient assumptions and simplifications to come up with equations that their mathematical approach can handle. Sometimes, they don’t even realize that they are making these assumptions since the papers they read have made these hidden assumptions and have been successfully published. These researchers are incredibly smart, and they then develop fascinating tools to optimize the simplified problem. Their paper gets published in a journal.

When a practicing engineer attempts to apply the method, the engineer quickly realizes that the results, while optimal, are really stupid. Assumptions and simplifications made along the way make the optimal solution unworkable. Researchers seldom get this kind of feedback because most journal reviewers are also researchers, who may have never worked on the problem they are attempting to optimize. As a result, our journals have their fair share of papers that fall into the category I call, “Optimal, but stupid.”

I’ve ranted about this problem for decades and am frequently dismissed as a Luddite who’s against research. On the contrary, I enjoy research. Sometimes researchers take my advice, and their work is all the better for it. Over the years, I may have moved the needle a little in making research more realistic.

I’m not opposed to the use of optimization. It can be a powerful tool. However, both the developer and the end-user need a deep understanding of the assumptions and limitations that go into any solution.

There have been successful applications for optimization, but they have been delivered by teamwork between software developers and model users, each of which has a deep understanding of the real-world problem.

Bentley’s Darwin optimization tools in WaterGEMS are not immune from these issues. Their ease of use can invite users to attempt to solve problems for which they were not intended. Users may end up with solutions that don’t make sense in the real world because the user did not understand the assumptions in Darwin. My recommendation is that before anyone use any optimization, the potential user spends a few days (or even weeks) developing solutions using simulation. This work may generate better general solutions than any optimization but at least will identify which portions of the solution space are most promising and what parameters are important.

My proposed solution requires that anyone wanting to develop an optimization method must work on that problem for a few years without optimization. Only then should they try to apply optimization. Of course, that’s another (optimal but stupid) idea that won’t work in the real world. A simpler version is for researchers to communicate with practicing engineers/operators of water projects. My solution can be summarized as “Take an operator to lunch”. I’ve found that simply listening to real-world decision-makers is a great way to help define the next research advancement.

We spend a lot of time at Bentley listening to our users so that we can address real-world users with real-world problems. This results in software that solves real problems and is not an effort in a solution in search of a problem.

If you want to look up past blogs, go to https://blog.bentley.com/category/hydraulics-and-hydrology/. 

If you want to contact me (Tom), you can email tom.walski@bentley.com.