Position a rack in the upper third of the oven; preheat to 475°.
In a medium bowl, toss the potatoes with a couple tablespoons olive oil, half the rosemary, half the garlic and the thinly sliced lemon; season with salt and pepper. Spread in a single layer on a rimmed baking sheet, arranging the lemon slices on top of the potatoes to prevent burning. Roast 12 to 15 minutes.
Meanwhile, season the lamb chops. In a small bowl, whisk the mustard, 2 tbsp. olive oil, the remaining rosemary, the remaining garlic and the lemon zest; spread the mixture all over the lamb chops.
Place the lamb chops on top of the potatoes; sprinkle with the capers. Roast until the potatoes are tender, 12 to 15 minutes. Douse the lamb and potatoes with the lemon juice. Serve with a handful of arugula alongside.