Heat the remaining 2 tablespoons of olive oil in a large skillet over medium-low heat. Pat the chicken dry, season with salt and pepper and place, skin side down, in the skillet. Cook until the skin is browned and crisp, about 5 minutes. Transfer the chicken, skin side up, to the prepared baking sheet. Bake until cooked through and the juices run clear, about 20 minutes.