Rub the inside of a small saucepan all over with the garlic. Add the cream, the 2 whole rosemary sprigs and nutmeg; season with pepper and cook over medium-low heat until reduced by half, about 10 minutes. Discard the rosemary.
Bring a large pot of water to a boil, salt it, add the pasta and cook until al dente. Drain and transfer the pasta to a serving bowl. Add the rosemary cream, cauliflower and 1/2 cup cheese; toss to coat. Season with salt and pepper and top with the walnuts and chopped rosemary. Pass more cheese around the table.