This is because List.push returns a new list with n pushed to listOfNumbers. It does not change listOfNumbers itself. So you would want the following:
// `var` instead of `let`, so you can re-assign it.
stable var listOfNumbers = List.nil<Nat>();
// ...
// Update `listOfNumber` to the new list with `n` added to it.
listOfNumbers := List.push(n, listOfNumbers);