As you’ve discovered, multiple addresses per client isn’t currently supported, but we do have an active feature request that you may wish to add your vote to (simply comment on the thread).
You’re not restricted to 1 tag per invoice, in fact you can have up to 5 each. So you could tag it to a tag for a job number, one for client reference, one for location - the choice is yours.
You could even set up default tags for suppliers and clients too to help make it a bit quicker.
Hopefully that will help. However, other users are welcome to add their comments below too.