Preheat a grill or grill pan to medium-high. Using a food processor, pulse the lemon peel, garlic and parsley until finely chopped. In a bowl, combine the olives, basil, capers and orange peel.
Remove the skin and dark flesh from the swordfish steaks and discard. Drizzle the fish with the EVOO, season with salt and pepper and rub with the lemon mixture. Place the fish between 2 layers of wax paper and gently pound out to 1/8 inch thick. Cut the fish into twelve 3-inch-wide strips no more than 5 inches long. Spread the orange-olive mixture on the strips. Roll up the fish strips and thread 3 rolls at a time onto skewers.
Grill the swordfish skewers, turning frequently, until firm to the touch and opaque, about 7 minutes. Grill the bread and serve alongside the skewers, with lemon and orange wedges for squirting on the fish.