This problem is not reproducible locally when I supply address to local memcached server:

docker run --name=play -e MEMCACHED_ADDR=host.docker.internal:11211 --rm -p 8080:8080 playground

Caching works fine. The root cause might be relevant to the deployment of Playground in the google cloud.

After jumping through a bunch of hoops, I was finally able to (web) ssh into the golang-org memcached VM's sole internal IP address.

And there's no memcached running on it.

docker ps -a says it Exited (255) 7 weeks ago.

It was a Container-Optimized OS VM (using konlet to pull the memcached container at start) with no public IP address, so the GCE web UI refused to let me ssh to it even from the web UI. And because its internal IP address was in a different network from Cloud Shell, I couldn't ssh from there either.

I couldn't get its logs easily (COS+konlet means it only goes to Stackdriver logging, and not locally, and I didn't want to load the web UI), but it dying at all (and not restarting!) is sketchy, so I just nuked it.

I've since re-created it with a hand-configured normal Debian Stretch VM, with an "external" IP address (that's firewalled off anyway).

Then I ran:

# apt-get install memcached
# nano /etc/memcached.conf ... remove the `-l 127.0.0.1` bit so it now listens on all interfaces, and adjusted the `-m` memory to 2048.
# systemctl restart memcached

And I added the "memcached" network tag to it and added a firewall rule that permits 11211 from 10.0.0.0/8.

I see that it's getting traffic now:

# (echo stats; echo quit) | nc localhost 11211 | grep get_
STAT get_hits 46
STAT get_misses 294
STAT get_expired 0
STAT get_flushed 0

Configuring a machine like this by hand is lame, but this is temporary and better than it not working.

We'll migrate to https://cloud.google.com/memorystore/ later.