In a medium bowl, add egg yolks and granulated sugar. Whisk together until the mixture is pale and slightly thickened.

In a medium saucepan over medium heat, add whole milk, ground cinnamon, and ground nutmeg. Heat until the mixture just begins to boil, stirring occasionally to prevent scorching.

Remove the saucepan from heat. While whisking the egg mixture constantly, slowly drizzle in the hot milk mixture, one spoonful at a time. This step is important to temper the eggs, adding the hot milk too quickly can cause them to scramble.

Once all the hot milk has been whisked into the egg mixture, pour the entire mixture back into the saucepan. Return the pan to medium heat and cook, stirring constantly with a wooden spoon or silicone spatula, until the mixture thickens slightly. It should coat the back of a spoon, this will take about 3 to 5 minutes. Do not let it boil.

Remove the pan from heat and allow the mixture to cool completely at room temperature.

Once cooled, whisk in the heavy whipping cream, vanilla extract, and brandy (if using).

Cover and refrigerate for at least 4 hours, or up to overnight, until well chilled.

When ready to serve, pour into glasses and sprinkle with a little ground cinnamon or ground nutmeg.