In a large pot, add the potatoes and enough water to cover; bring to a boil over medium-high heat. Salt the water generously. Simmer until the potatoes are fork-tender, about 25 minutes.
Drain and peel the potatoes. While the potatoes are still warm, squeeze the flesh through a food mill or ricer into a large bowl, adding the butter halfway through to allow it to melt. Stir in the warmed milk. Fold in the creme fraiche. Season with 3/4 tsp. salt and freshly ground black pepper to taste. (Potatoes can be kept warm in a covered ovenproof bowl in a 325 degrees oven for up to 1 hour.)
