Using D4-D20 is totally something I'd like to explore over time. I've thankfully got the dice created in such a way that I can create any type of die mesh that I'd like and it "should" be fully functional with only minor configuration needed.
I've got some plans for expansion on the rogue-like mechanics, but hadn't thought about mixing some die for the abilities and other die for their effects. I'd like to try and keep the UX streamlined so that the player just needs to click a single button for each ability and the system can automatically figure out which die it's supposed to use to pull that ability off. That might still work with the mixing of ability die and effects die, but it'll definitely take some experimentation.
I haven't done too much brainstorming about enemy variety yet, but as you mentioned there are several ways that enemies could contribute to the combats. I'm personally a big fan of simplified enemy designs that only do 1-3 different things and then creating interesting combats based on the combination of the different enemies (with any kind of Elite or Boss special combats having more complexity to make them feel more special). I've seen a couple games where the enemies roll dice for their behaviors (thinking Dicey Dungeons and Aristea), but I kinda like the more deterministic method for enemy behaviors because it makes the player gambling choices more explicit in the moment and less determined by RNG of enemy die rolls.