Yes, it is always recommended to start from there. Starting from the art, the 'end look' is an awful idea. Art is more or less easy to produce (or get using 3rd party assets) but also is time consuming.
It's recommended to start from all the 'systems' (character controller, enemies AI, triggers, etc. Specially if you work in engines like Unity, GODOT or Unreal, and similar.) because that way you can see if an idea you had is easy to do, makes sense, is fun, etc (and that way you don't spend weeks or months working in a enemy or a level that will have to be discarded).
It's not necessarily to use only primitives (cubes and triangles), if you need you can use temporary simple art (placeholders) to get an idea if the game feel and looks may be remotely similar to the what you have in your head.
Fun may be more or less subjective, but when starting the game, the most important part is make sure the ideas you have can be executed.