In a medium bowl, whisk the olive oil and spices. Season with salt and pepper; toss with the shrimp. Preheat a large cast-iron skillet over medium-high.
Working in two batches in a blender or food processor, puree the gazpacho ingredients; season with salt.
Add the shrimp to the skillet and cook until opaque in the center and browned in spots, 3 to 6 minutes. Douse with vermouth and swirl to evaporate; remove from the heat. Squeeze the lime over the shrimp.
Serve the gazpacho in large mugs or chilled half-pint glasses with the shrimp alongside for dipping.