Preheat the oven to 200 degrees . Combine the flour, brown sugar, baking powder, salt and baking soda in a medium bowl. In a separate bowl, whisk together the buttermilk, peanut butter, egg and vanilla. Lightly mix the wet ingredients into the dry ingredients; the batter will be fairly thick.
Pour enough oil into a small, deep saucepan to reach a depth of 1-1/2 inches and heat over medium-high heat until the oil registers 310 degrees on a deep-fry or candy thermometer.
Working in 3 or 4 batches, drop the batter by rounded teaspoons into the hot oil and cook, turning once, until lightly browned, about 1-1/2 minutes per side.
Using a slotted spoon, transfer the fritters to paper towels to drain. Keep the fritters in the oven as you continue frying. Serve hot with the whipped cream and jam.
