Preheat the oven to 450. In a large bowl, sift 2 cups flour with the baking powder and 1/2 teaspoon salt. Using your fingertips, blend in the chilled pieces of butter until the mixture resembles coarse meal, then mix in the chicken. Stir in the buttermilk just until incorporated.
On a lightly floured surface, knead the dough until just blended, then pat into an 8-inch round. Using a 3-inch cookie cutter, cut out 6 biscuits; combine the scraps and cut out 2 more biscuits.
On a baking sheet, arrange the biscuits 2 inches apart and brush lightly with buttermilk. Bake until golden, about 15 minutes.
Meanwhile, in a medium skillet, fry the bacon over medium heat until crisp, about 4 minutes. Stir in the remaining 2 tablespoons butter until melted, then whisk in the remaining 3 tablespoons flour until incorporated. Whisk in the chicken broth, milk, 1/2 teaspoon salt and the cayenne and simmer over medium-low heat until thickened, 3 to 4 minutes. Serve the gravy alongside the chicken biscuits.