Fill a medium saucepan three-quarters full with cold water. Sprinkle about 2 tsp. salt into the water and bring to a boil.
Meanwhile, rest 1 end of a fine grater on the rim of a bowl. Finely zest the lemons into the bowl. Add the EVOO and sprinkle in 1/2 tsp. each salt and pepper; mix well with a rubber spatula.
Add the orzo to the boiling water and cook, stirring often, until tender. Drain the pasta in a colander and rinse well with cold water. Add the cooled orzo to the bowl with the lemon oil and toss.
Starting at the top of the celery bunch, thinly cut crosswise with a large knife until you have about 1 cup sliced celery and leaves; add to the pasta. Crumble in the feta. Pit the olives, tear them in half and add to the pasta. Gently toss the pasta and transfer to a platter.