Peel, core, and cut 1.5 pounds Granny Smith apples into wedges.
Partially cooked apples: melt ¼ cup unsalted butter in a large skillet over medium-high heat. Add apples and sprinkle ½ cup sugar on top. Stir with a wooden spoon or spatula and cook for about 3 minutes, until they begin to caramelize here and there. Do not let them turn too dark or too soft. Remove from heat, add ½ teaspoon cinnamon and 1 tablespoon lemon juice and arrange them in the prepared dish. Scatter ⅓ cup raisins, dried cranberries, chopped walnuts or pecans over if using them.
Trim 1 pound brioche and cut it into big chunks or cubes.
Arrange the bread on top of the apples. It will be uneven.
Stir to combine 1 cup whole milk, 1 cup heavy cream, 2 large eggs and 1 egg yolk in a medium bowl or pitcher. Do not beat.
Pour mixture slowly over bread, gently pressing down to absorb the liquid.
Let it soak at room temperature for about 15 minutes, pressing a few times so that the bread absorbs as much of the milk mixture as possible before going into the oven.
Preheat oven to 350ºF (180°C).
Bake for about 40 minutes, until the liquid is absorbed and the crust is golden. Check the inside by lifting the bread with a fork. You don't want liquid, but it's fine if not completely set.
Let rest for a few minutes on a wire rack.
Serve warm or at room temperature with caramel sauce, homemade dulce de leche or crème anglaise (custard sauce). For a showstopper dessert, add a dollop of whipped cream or vanilla ice cream and pour the sauce over it.