The sticky ranch
We are running Rancher in combination with the in-built load balancer HAProxy. For each of our customers, our application is provided as a single container, many on the same physical server instance. Each of the customers' applications can be accessed via different URLs, so the usage of the HAProxy as the routing component part of the load balancer makes sense.
Accessing some of these containers by different URLs, we experienced some strange behaviour: In some given time range, no matter which URL was used, all requests were internally forwarded to the first of the containers for which an URL had been triggered.
After some research we found out that we did not set a stickiness flag, which seems to be required for the load balancer to differ incoming Url calls. So in Rancher + HAProxy this is settable in the load balancer configuration. In the lower section there are some tabs for SSL termination, stickiness, custom haproxy.cfg, labels and scheduling.
Just go to the "stickiness" tab, and apply "set a new cookie", give it a name like "sticky". We chose the mode "insert" and a wildcard domain "*.example.tld". Also we checked settings "indirect" and "send no-cache header". Submit by using the "Edit" button.
After running the load balancer upgrade, the issue was gone immediately.