Preheat oven to 350°F (175°C). Grease a 9x5-inch loaf pan and line with parchment paper leaving an overhang.
Whisk flour, granulated sugar, brown sugar, baking soda, baking powder, salt, cinnamon, nutmeg, and ginger in a large bowl.
In another bowl, whisk eggs, pumpkin puree, oil (or melted butter), milk, and vanilla. Pour into dry ingredients and fold gently until just combined.
Beat softened cream cheese until smooth, then add powdered sugar, egg, and vanilla and beat until lump-free.
Spoon half the pumpkin batter into the pan, dollop the cream cheese mixture over it, then add remaining batter. Use a knife to create a gentle swirl.
Bake 55–65 minutes, tenting with foil if the top browns too quickly. A toothpick inserted into the center (avoiding the cream cheese) should come out with a few moist crumbs.
Cool in the pan 10–15 minutes, then remove with parchment and finish cooling on a rack before slicing.