In a large pot, combine the potatoes and rosemary sprigs. Add enough water to cover by 1 inch. Season with salt. Bring to a boil, reduce heat and simmer until potatoes are tender, 15 to 20 minutes. Discard the rosemary. Drain the potatoes, reserving 3/4 cup of the cooking water. Reduce the heat to low. Return the potatoes to the pot and toss 1 to 2 minutes to thoroughly dry.
In a small skillet, heat 1 tsp. chopped rosemary and 1 tbsp. EVOO over medium, swirling the skillet occasionally, until the rosemary is crispy, about 5 minutes. Remove from the heat.
In a large bowl, mash the potatoes. Stir in the remaining 1/2 cup EVOO and enough reserved cooking water to make the mixture smooth. Mix in the lemon zest; season. Transfer to a serving bowl. Drizzle with the rosemary oil.