You forgot about the Nautilus in "Dark Waters". Liam somehow managed to locate Nemo in Storybrooke and get here by submarine. My theory is that the the ship uses mermaid magic to traverse dimensions. It's the best explanation I can come up with, since it obviously didn't arrive on the dirigible.
I agree that it likely has the ability to cross realms. I like your theory that he somehow harnessed mermaid magic, I didn't really have an explanation for it.
If it can cross realms, that would also explain why Nemo doesn't seem to fit in with the Enchanted Forest, and possibly how Liam was able to age - they were in another realm where time was able to pass. Say, what if they ended up in the Dark Realm...? That could explain why Nemo is coming back.
Unfortunately, the show didn't make it clear. Like you said, it couldn't have come in on that dirigible, but Nemo may have, and then Liam and some of the rest of the crew were left behind but came to Storybrooke on their own. That's the likely, but non-canon, explanation.
The show has not made it clear...yet. Now, they may or may not make it clear at some point, but I hope that we have not seen the last of Nemo, Liam, or the Nautilus.
Now, as for the the realm-crossing ability, I agree that it does seem likely that the sub has somehow acquired this ability. Maybe is has harnessed mermaid magic, maybe it was build by merpeople. Maybe there is another reason we don't know yet.
Given that Liam is Hook's half-bro, it's very likely we'll see them again. Of course, since 20,000 Leagues Under The Sea is my favorite Disney flick, I'm really hoping for a spin-off series. Unlikely, but you never know.
For those who are unaware, Nemo and Liam will be returning In the episode "Page 23" on March 26 and the next episode "A Wondrous Place". I think I can guess their part in the story. Our heroes will most likely be traveling to Agraba by way of the Nautilus, most likely by the way I described it in my earlier comments. That or some similar method. In any case, March 26 can't get here fast enough.