Comments on: Elastic Load Balancer: An Elasticity Gotcha

By: High Availability Across Multiple Data Centers, Multihoming and EC2 | Development, Analysis And Research

Sat, 29 May 2010 11:26:53 +0000

[...] Elastic Load Balancer: An Elasticity Gotcha (Caching DNS and ELB releasing IPs) and Elastic Load Balancer is elastic – how the elasticity is achieved [...]

By: shlomo

shlomo — Tue, 27 Apr 2010 15:31:29 +0000

@Drew,

Please post back when you settle on a solution.

By: Drew

Drew — Tue, 27 Apr 2010 15:20:07 +0000

Yes. We’re using Apache because 1) legacy reasons, and 2) it performs server side includes in the response which are handled by Apache mod_include, mod_python, etc.

Varnish is certainly a much better proxy solution, we’re just not decoupled enough from our legacy (non-cloud) environment yet.

We’re testing out a few options right now.

By: shlomo

shlomo — Tue, 27 Apr 2010 14:30:53 +0000

@Drew,

I’m not familiar with any way to control the TTL of Apache’s DNS lookups.

Have you considered using squid or varnish as a reverse proxy instead of apache?

By: Drew

Drew — Tue, 27 Apr 2010 13:59:25 +0000

Thanks, @Shlomo. I’ve finally gotten to the bottom of this, and as you suspect, it is *not* an ELB issue (other than a desire to have fixed ELB IP addresses). Amazon’s 1 hour delay is working as expected.

We are proxying to the ELB from an Apache web server, and it turns out that Apache is secretly caching DNS lookups despite a short TTL on the underlying server and no obvious setting to enable, disable, or tweak this functionality in Apache. Any Apache process that happens to be running when an ELB IP is removes and continues to live longer than 1 hour may result in this situation.

While I now understand the issue, I’m currently looking for a solution… shortening Apache process lifespan, running Apache in worker mode, restarting Apache every hour… all would suffice. I’m still hoping to find a magical Apache setting that will resolve this for me.

Thanks for your input!

By: shlomo

shlomo — Tue, 27 Apr 2010 09:45:39 +0000

@Drew,

I do consider the issue with the ELB IP address being repurposed resolved, since the 1-hour delay before it can be reused by another ELB should be sufficient for all but the most egregiously broken DNS environments.

It seems to me that the issue you’re experiencing is not related to the ELB IP address, but rather to a different issue: the instance you originally registered in the ELB was not removed from the ELB when it shut down. The IP addresses of that instance remained in the load balancer pool, and when another user’s instance was assigned those IP addresses, your ELB routed traffic there. As mentioned in the thread, fixing this problem is on the AWS roadmap, but – as usual – no timeframe is given.

By: Drew

Drew — Mon, 26 Apr 2010 16:29:17 +0000

@Shlomo: Per SanD@AWS’s note, it seems unlikely that an ELB IP would be pointed to someone else’s site. Based on your understanding of this issue, do you consider it resolved?

I ask because I have experienced this unpleasurable situation as recently as last week (http://developer.amazonwebservices.com/connect/message.jspa?messageID=175874#175874). I’m wondering if I’m barking up the wrong tree.

By: Rick

Rick — Mon, 22 Mar 2010 19:33:10 +0000

@Drew,

I am experiencing this very issue with my ELB ever since March 5th. It always worked before then, but since then, we’ve been getting reports of stale caches every day.

I know you say this is supposed to be fixed, but is it really fixed? Is there anything that might cause this still?

By: The “Elastic” in “Elastic Load Balancing”: ELB Elasticity and How to Test it

Sun, 07 Feb 2010 21:25:12 +0000

[...] that does this) beyond the defined TTL, traffic may be misdirected during ramp-up or ramp-down. See my article for a detailed [...]

By: Drew

Drew — Mon, 25 Jan 2010 21:46:39 +0000

We've been experiencing this very issue recently and I'm glad to find an article that corroborates my theory. Thanks for posting, Shlomo.