How to make memcache accessible only from several ip's?

How to make memcache accessible only from several ip's?

I have 4 load balansed apache servers, that are using one memcache server.
I want to make memcache server accessible only from my this 4 servers. How
can I do it?
Can I write something like this in memcache's config?
-l 127.0.0.1, 127.0.0.2, 127.0.0.3, 127.0.0.4