How to do dns based pooling?

[WesselAtWork]

Currently trying to do a simple replication proxy by using a dns name that contains all the other memcached ip addresses

will look at more complicated hashing things later, for now, this is the simplest I can do

Using the dns name works kinda...

Setup

memcached contains 3 ips, order is randomized every time I invoke it
nslookup memcached

Name:   memcached.test.local
Address: 10.0.0.12
Name:   memcached.test.local
Address: 10.0.0.11
Name:   memcached.test.local
Address: 10.0.0.10

memcached -vv --port=22122 -o proxy_config=test.lua

test.lua

package.loaded["simple"] = nil
local s = require("simple")

verbose(true)

router{
  router_type = "flat",
  log = true,
}

pool{
  name = "default",
  backends = {"memcached:11211"},
}

simple.lua is in the same dir with a small modification (changed the o to a c)

            if c.r.log ~= nil then
                top = logreq_factory(c.pool)
            else
                top = function(r) return c.pool(r) end
            end

startup looks like:

router: overriding default for    router_type
making backend for... memcached:11211
setting up a zoneless flat pool
setting up a zoneless flat pool
setting up a zoneless flat pool
setting up a zoneless flat pool
<43 server listening (proxy)
<44 server listening (proxy)

Results

Setting a unique key in each instance separately, I can kind of get them from the proxy, but it's inconsistent.

10.0.0.11 is the most consistent, request the key that's located here, I (almost) always get a response
10.0.010 is less consitent, i get about 40% to 50% of the time
Weirdly, 10.0.012 is never queried! The proxy never hits this one!

Also of note the 4 "setting up a zoneless flat pool" lines. I would expect it to only output 3 becuase there is only three ips, but it called mcp_config_routes 4 times?
Maybe there is a bad address in the list?

There is obviously some kind of resolution going on inside of memcached but it's not transferring correctly to the proxy, or my env is messing with the setup.

---

I don't mind digging into the lua myself, but if possible can you send me on the right track?

Where do I need to focus?

As a start I should probably install luasocket and then hack around the dns module, but if I can avoid that, it would be ideal.

Additionally I wonder about freshness. How does the resolution function?
Is it a one-and-done or does it query each time?

[dormando]

Hey,

First off, you'll probably want to stick to routelib: https://github.com/memcached/memcached-proxylibs/tree/main/lib/routelib - it makes a lot more sense and simple should be deprecated at this point (did I forget to mark it?)

Next, well as far as I know none of the existing memcached proxies allowed for dns based pool definitions, so we definitely don't support that out of the box. If you give a hostname to a backend it'll pick the first dns response and stick to that forever.

That said it's doable but you'll have to set up a connector to load your server list. There are a few options:

• You have an external cron/script that checks your DNS, and if it updates, write out a little lua-style datafile to disk and send a HUP signal to memcached. Then, at the top of your routelib config, load this file and parse it into a table that can be passed to pools{} (I can help with some examples of this if needed). DNS SOA numbers can be useful here to avoid having to do a lot of comparison work.

• You can use the proxy's cron utility: https://github.com/memcached/memcached/blob/master/t/proxycron.lua#L7 to do something similar but internally to the proxy. Every N seconds, do a DNS lookup (you can try to do this via luasocket, or just have lua call to a script in a more familiar language, or etc). If DNS changed from the last check time, you put stash the variable in a global variable somewhere, then call mcp.schedule_config_reload(), which will re-run the main config which you can pick up from there.

It's pretty easy to mix a cron file with routelib, which you do via passing multiple start scripts (in order):
memcached -o proxy_config=routelib.lua:mycron.lua,proxy_arg=config.lua

This sounds a little complicated, but because the system is flexible about where the server list comes from, it's designed to plug into any kind of server discovery already. The user might just need to do a little work to get the server list into the thing :)

If you can give some hints as to which direction would work best for you I can help come up with a more complete framework example to save you some time.

[dormando]

oh also: the reason why zoneless flat pool got printed four times is because part of the config code is executed once per worker thread, and you're probably running the default of four threads.

[WesselAtWork]

Thanks for the quick reply!
Ok I'm going to try RouteLib and see where that puts me first then I'll come back.

So far the proxycron is the most appealing.

Question though, why do I see the behaviour of the proxy hitting some of the instnaces. If what you said was true

"If you give a hostname to a backend it'll pick the first dns response and stick to that forever."
then I would expect it to always hit one of the instances.

Is it because there is 4 threads? It seems the ip address order is randomized when it's queried, am I just seeing the result of some of the threads getting a different address?

[dormando]

Just in case it wasn't obvious from the above; you could dump your whole routelib focused config.lua from something else and issue a HUP if that's easiest. don't need to pull the backend list from within config.lua unless that's easier than the other way.

If it's a big enough thing we can look into adding some internal dns utilities

[dormando]

Is it because there is 4 threads? It seems the ip address order is randomized when it's queried, am I just seeing the result of some of the threads getting a different address?

This, yup. Each worker thread has its own connection to the backend servers (unless you flip an option to turn on a consolidating io thread). So you're seeing the result of each threads' DNS lookup.

[WesselAtWork]

Definitely easier from inside the conf.lua, as few dependencies as possible as far as I am willing to push it.

I could probably get away with calling os.execute("nslookup") but I probably shouldn't.

The dns utils inside memcached sound great!
But there are alternatives available.

I'm gonna try following the cron template and use luasocket to see what I can come up with.

[dormando]

Sounds good. Please don't suffer too long though; if something is frustrating that's probably a pain point for me to clarify, document, or patch if necessary. Though we probably wouldn't see direct DNS utilities for a while.

So if something is going sideways you can show me what you have so far and I'll adjust the framework or add an example config to the routelib repo or something.

The proxy's in pretty heavy use but not that many total users so far, so I've been focusing on polish this year to help more varied use cases.

Good luck! Thanks for checking it out.

[WesselAtWork]

Ok made some progress

package.loaded["socket.core"] = nil
local s = require("socket.core")

verbose(true)
debug(true)

-- to be set via env later
local port = "11211" --string

local function dns_to_ips(address, socket)
  local info = socket.dns.getaddrinfo(address)
  if info == nil then
    print("NO DNS IPs!")
    return {}
  end

  local ips = {}
  for index, value in ipairs(info) do
    ips[index] = value.addr..':'..port
  end

  return ips
end

settings{
  active_request_limit = 100,
  backend_connect_timeout = 3,
}

pools{
  myself = {
    backends = {
      "localhost:11211",
    }
  },
  other = {
    backends = dns_to_ips("memcached", s)
  },
}

-- using cmap for the future and to see what different routing policies do 
routes{
  cmap = {
    [mcp.CMD_GET] = route_allfastest{
      children = { "other" },
    },
    [mcp.CMD_SET] = route_allfastest{
      children = { "other" },
    },
  },
  default = route_direct{
    child = "myself",
  },
}

--variance setup so that multiple instances don't get synced and spike the DNS server.
local variance = math.random(3, 6)

math.randomseed(os.time())

mcp.register_cron("dns",
{ every = variance,
  func = function ()
    mcp.schedule_config_reload()
  end })

The hardest part was actually installing and figuring out luasocket!
I needed to set both LUA_PATH, LUA_CPATH correctly before it was able to find it.

Sadly luasocket's dns capabilities are very very basic.
I wanted to use srv records (so port config is also defined in DNS) but all I can do is the ips :(

I can also see it reloading so that's nice!

verbosity set to:    true
debug set to:    true
settings:
{ ["backend_connect_timeout"] = 3,["active_request_limit"] = 100,} 
pools config:
{ ["other"] = { ["backends"] = { [1] = 10.0.0.110:11211,[2] = 10.0.0.111:11211,[3] = 10.0.0.112:11211,} ,} ,["myself"] = { ["backends"] = { [1] = localhost:11211,} ,} ,} 
routes:
{ ["cmap"] = { [9] = { ["f"] = allfastest,["a"] = { ["children"] = { [1] = other,} ,} ,} ,[7] = { ["f"] = allfastest,["a"] = { ["children"] = { [1] = other,} ,} ,} ,} ,["default"] = { ["f"] = direct,["a"] = { ["child"] = myself,} ,} ,} 
loaded
changing global setting:    backend_connect_timeout    to:    3
changing global setting:    active_request_limit    to:    100
making pool:    myself    
    { ["backends"] = { [1] = localhost:11211,} ,} 
making backend for... localhost:11211
making pool:    other    
    { ["backends"] = { [1] = 10.0.0.110:11211,[2] = 10.0.0.111:11211,[3] = 10.0.0.112:11211,} ,} 
making backend for... 10.0.0.110:11211
making backend for... 10.0.0.111:11211
making backend for... 10.0.0.112:11211
Checking child for route:    children
Checking child for route:    children
Checking child for route:    child
mcp_config_pools: done

The ips get shuffled every reload.

---

A different problem. I am probably doing it wrong but none of the existing routers are giving valuable functionality.

• route_direct: looks like it always chooses one instance (seems to be the first one in the list)
• route_allsync: described as requesting between pools in parallel, did not request all instances in parallel.
• route_failover: thought it would failover inside the pool, but seems to want to failover between pools
• route_allfastest: sadly this one behaves the same as direct. because it does not query in parallel but serial and because memcached responds so fast, what ends up happening is that the first instance queried responds first, practically always. After reload the order is shuffled, and we are hoping the instance ends up in the first entry.

What is the conceptual reasoning behind pools? I probably missed it on the wiki.

Should I be creating a pool-per-backend? Or am I missing something crucial?

[dormando]

Congrats!

Well I was hoping saying "pool" everywhere would be pretty clear, but this is actually the second time this week it's confused someone :) I'll see about adding some extra examples/docs somewhere.

The primary use case of memcached is "hash keys against a list of servers, and store/fetch that key from exactly one server", thus adding servers to a "pool" increases available memory. In the proxy a pool is exactly this (there are configuration options to control the key hashing). The routes are for directing/copying keys between pools. So "allfastest" will route a key to all pools. It doesn't know/care if a pool is a single server or 100.

The concept of copying routes between pools was originally for things like "Datacenter A and B" or "availability zones 1/2", or racks/cages/etc.

So if you just have a small list of memcached servers and you want keys copied to all of them, yes you need to create one pool per backend.

You can create a "pool set" to make the configuration a bit easier. There's an example here: #7 (comment)

[dormando]

Fwiw you should see if you can move more of the logic into the cron so you're only reloading the configuration if the backends actually change. It's designed to be reloaded very frequently but it is a bit wasteful on CPU.

[WesselAtWork]

Agreed

-- imports
package.loaded["socket.core"] = nil
local s = require("socket.core")

-- env
local port = os.getenv("BACKEND_PORT") or "11211" --string
local dnsname = os.getenv("BACKEND_HOSTNAME") or "memcached" --string

-- setup
verbose(true)
debug(true)

function DNS2IPS(address, socket)
  local info = socket.dns.getaddrinfo(address)
  if info == nil then
    -- say("DNS returned no IPs!")
    return {}
  end

  local ips = {}
  for index, value in ipairs(info) do
    ips[index] = value.addr..':'..port
  end
  -- dsay("Got DNS: "..dump(ips))
  return ips
end

-- https://stackoverflow.com/a/54140176
function SIMPLE_TABLE_COMPARE(tA, tB)
  return table.concat(tA) == table.concat(tB)
end

----

-- settings
settings{
  active_request_limit = 100,
  backend_connect_timeout = 3,
}

-- dns
DNS_IPS = DNS2IPS(dnsname, s)
table.sort(DNS_IPS) -- performance is 1s per 1M entries

-- pools
pools{
  myself = {
    backends = {
      "localhost:11211",
    }
  },
  other = {
    backends = DNS_IPS
  },
}

-- routes
routes{
  cmap = {
    [mcp.CMD_GET] = route_allfastest{
      children = { "other" },
    },
    [mcp.CMD_SET] = route_direct{
      child = "other",
    },
  },
  default = route_direct{
    child = "myself",
  },
}

-- cron
local variance = math.random(3, 6)
math.randomseed(os.time())

mcp.register_cron("dns",
{ every = variance,
  func = function ()
    new_ips = DNS2IPS(dnsname, s)
    table.sort(new_ips)
    if not SIMPLE_TABLE_COMPARE(new_ips, DNS_IPS) then
      mcp.schedule_config_reload()
    end
  end })

Unsure why, but dsay and say don't work the way I expect them to.
They are global in the lib, but they don't do anything if I use them in this file.

[WesselAtWork]

I'm going to see what the zoned config can do.

[WesselAtWork]

Using zpools

package.loaded["socket.core"] = nil
local s = require("socket.core")

local port = os.getenv("BACKEND_PORT")                  or "11211" --string
local dnsname = os.getenv("BACKEND_HOSTNAME")           or "memcached" --string

verbose(true)
debug(true)

local_zone("zlocal") -- we NEED to define a local zone

function GET_DNS(address, socket)
  local info = socket.dns.getaddrinfo(address)
  if info == nil then
    -- say("NO DNS IPs!")
    return {}
  end
  return info
end

-- IPP = IP:PORT
function DNS2IPPS(address, socket)
  local info = GET_DNS(address,socket)

  local ipps = {}
  for index, value in ipairs(info) do
    ipps[index] = value.addr..':'..port
  end
  -- dsay("Got DNS: "..dump(ips))
  return ipps
end

function IPPS2POOLS(ipps)
  local pools = {}
  for index, ipp in ipairs(ipps) do
    pools[ipp] = { ["backends"] = { [1] = ipp } }
  end
  -- dsay("Generated pool: "..dump(pools))
  return pools
end

-- https://stackoverflow.com/a/54140176
function SIMPLE_TABLE_COMPARE(tA, tB)
  return table.concat(tA) == table.concat(tB)
end

settings{
  active_request_limit = 100,
  backend_connect_timeout = 3,
}

DNS_IPPS = DNS2IPPS(dnsname, s)
table.sort(DNS_IPPS) -- performance is 1s per 1M entries

local bmyself = {
  backends = {
    "localhost:11211",
  }
}
local szmain  = IPPS2POOLS(DNS_IPPS)
szmain["zlocal"] = bmyself

-- pools
pools{
  pmyself = bmyself,
  set_zmain = szmain
}

-- routes
routes{
  cmap = {
    [mcp.CMD_GET] = route_zfailover{
      children = "set_zmain",
      stats = true,
      miss = true,
    },
    [mcp.CMD_SET] = route_allsync{
      children = "set_zmain",
    },
  },
  default = route_direct{
    child = "pmyself",
  },
}

local variance = math.random(3, 6)
math.randomseed(os.time())

mcp.register_cron("dns",
{ every = variance,
  func = function ()
    candidate_ipps = DNS2IPPS(dnsname, s)
    table.sort(candidate_ipps)
    if not SIMPLE_TABLE_COMPARE(candidate_ipps, DNS_IPPS) then
      mcp.schedule_config_reload()
    end
  end })

The idea is to make the 'local instance the default pool, it is the "closest" so could be checked first before the rest.

Now that i am sorting the order of the dns responses, let me try the original settings again. Maybe it'll hash/route correctly now.

I would prefer it to function the way you described pools.
Basically: the more memcached instances there are, the more caching space you have.

Currently it's coping all the keys which is cool for HA, but less cool for scaling.

You mentioned there are distribution options available?
What is the difference between dist_jump_hash and dist_ring_hash?

---

I saw this in the Wiki

Since every proxy is also a normal memcached, it is possible to create an "L1/L2" layered cache. The details will depend highly on your needs, but doing so can remove the added latency and CPU overhead of having extra network requests added from having to first go through the proxy.

How do I set that up?
Is there a special flag or something I need to set?

[WesselAtWork]

Ok cool.

Doing this now works fine:

-- pools
pools{
  pmyself = {
    backends = {
      "localhost:11211",
    }
  },
  pother = {
    backends = DNS_IPPS
  },
}

-- routes
routes{
  cmap = {
    [mcp.CMD_GET] = route_direct{
      child = "pother",
    },
    [mcp.CMD_SET] = route_direct{
      child = "pother",
    },
  },
  default = route_direct{
    child = "pmyself",
  },
}

I can always get a key that was set through the proxy.

Sadly if the ip table changes, chances are high that the new table will hash differently.
Sometimes it would still hash to the instance with the key, but more often then not the setup would "lose track" of the key.

Which feels wrong to me. The key is in the pool, just not where you expect it to be.

Depending on your use case this could be a non issue: short lived keys (5s to 1m) would practically be uneffected. long lived keys and "permanent" keys would start showing issues.
Considering low instance churn (probably once every hour or so), it would effectively reset the cache every time.
Not super ideal.

I might do something like this:

local bmyself = {
  backends = {
    "localhost:11211",
  }
}
local szmain  = IPPS2POOLS(DNS_IPPS)
szmain["zlocal"] = bmyself

-- pools
pools{
  pmyself = bmyself,
  pother = {
    backends = DNS_IPPS
  },
  set_zmain = szmain
}

-- routes
routes{
  cmap = {
    [mcp.CMD_GET] = route_zfailover{
      children = "set_zmain",
      stats = true,
      miss = true,
    },
    [mcp.CMD_SET] = route_direct{
      child = "pother",
    },
  },
  default = route_direct{
    child = "pmyself",
  },
}

That guarantees I set a key somewhere once, and I can find it where-ever it is.

I could do

    [mcp.CMD_SET] = route_zfailover{
      children = "set_zmain",
      stats = true,
      shuffle = true,
    },

That would always set zlocal first.

[WesselAtWork]

Ok I figured out something a little goofy.

-- pool sets
local szmain  = IPPS2POOLS(DNS_IPPS)
szmain["zlocal"] = {
  backends = DNS_IPPS
}

-- pools
pools{
  pmyself = {
    backends = {
      "localhost:11211",
    }
  },
  set_zmain = szmain
}

-- routes
routes{
  cmap = {
    [mcp.CMD_GET] = route_zfailover{
      children = "set_zmain",
      stats = true,
      miss = true,
    },
    [mcp.CMD_SET] = route_zfailover{
      children = "set_zmain",
      stats = true,
      shuffle = true,
    },
  },
  default = route_direct{
    child = "pmyself",
  },
}

Basically I am setting zlocal equal to the list of backends:

set_zmain:
  "10.0.0.110:11211":
    backends: ["10.0.0.110:11211"]
  "10.0.0.111:11211":
    backends: ["10.0.0.111:11211"]
  "10.0.0.112:11211":
    backends: ["10.0.0.112:11211"]
  zlocal:
    backends:
    - "10.0.0.110:11211"
    - "10.0.0.111:11211"
    - "10.0.0.112:11211"

Now i get the best of both :^).

1. It tries zlocal, Hashing works and I get the benefits of it's deterministic set and get
2. An instance gets added/removed/changed, now the original hashing order is reset breaking most of the key locations.
3. It fails over, and eventually finds an instance that has the key.
4. The new hashing order will cause sets to go to a new location but will also get that new location.

This does mean that zombie keys could happen.

Consider a situation where an key is set for 5m TTL:

• 00:00 -> key SOMEKEY is set valueX to instance1
• 01:00 -> hashing gets reset, ( SOMEKEY now points to instance2)
• 01:30 -> we get SOMEKEY
• -> failover because it's not on instance2
• -> get valueX from instance1
• 02:00 -> we set SOMEKEY to valueY (it is set on instance2)
• 02:30 -> we get valueY from SOMEKEY on instance2
• 03:00 -> instance2 gets removed, hashing reset again (SOMEKEY now points to instance3)
• 04:00 -> we get SOMEKEY
• -> failover because it's not on instance3
• -> get valueX from instance1

So in this situation we got old data sadly, and a client expecting fresh data won't know. :(
The chances of this occurring are directly proportional to the size of the total pool, the TTL of the keys, average instance churn.

Regardless I should definitely do:

    [mcp.CMD_DELETE] = route_allsync{
      children = "set_zmain",
    },

Deletes should go everywhere.

Maybe I should just do the set routeall and hope the amplification isn't too bad?
I doubt there is an easy way to describe: if set occurs, propagate a global delete for the key then action the set.

Maybe a stop job? If an instance is going away, maybe it should issue a global delete for every key it contains?