Place potatoes in the bottom of a large pot and cover with water. Bring to a boil, then reduce heat and simmer until cooked through, about 1 hour. Remove from water and let cool.
Once cool enough to handle, pull the peel off each potato and discard. Roughly chop potatoes and transfer to a large mixing bowl. Add celery, onion, mayonnaise, mustard, relish, salt, cayenne and dried dill. Stir to completely combine.