In a large bowl, whisk together the pumpkin purée, milk, vanilla pudding mix and pumpkin pie spice until it starts to thicken. Place the mixture into a piping bag or large plastic zip-top bag and refrigerate until you’re ready to build the trifle.
In a stand mixer fitted with a whisk attachment, whip the heavy cream to soft peaks. Add the sugar, ginger and salt, and whip to combine. Scrape the whipped cream mixture into another piping bag or large plastic zip-top bag.
Place a layer of graham crackers in the bottom of a large trifle dish or glass bowl—you may need to break them up a bit to make a flat layer. Pipe ¼ of the pumpkin mixture over the graham crackers. Place another layer of graham crackers on top and pipe ⅓ of the whipped cream over top. Continue to build, alternating between whipped cream and pumpkin filling layer, ending with pumpkin filling on top. Place the trifle into the fridge and chill for at least 8 hours or overnight.