Just doing end of 1/4 rec and have come across an interesting issue!
For a while I have bought domain names using a credit system, I am moving to buying on a per case basis.
When renewing those domains manually, the registrar charges my card individually. The result is I get 3 or 4 transactions from the same supplier, with the same description for the same amount on one day. This is fine in the bank account and used to be fine when manually uploading a CSV file.
However, it doesn’t work with Yodlee feed - the result is that only one transaction appears for that day, meaning the actual bank statement and the QF bank account are out of kilter and I have to manually add the missing entries.
This is something that’s unfortunately unavoidable as it’s very tricky to know what is a duplicate transaction when everything is the same. There are however 2 things that should be noted here.
Firstly, we offer 2 types of duplication checking, which are check description, date and amount, or just date and amount. While in your situation this may not be a resolution, it can help in some cases.
Secondly, the duplicate transactions aren’t discarded and can easily be added. If you go to More Options >> Bank Feed Details from the bank statement view:
Here you will be presented with 3 tabs:
If you select the ‘Raw Yodlee Data’ tab, this will show you exactly what data we receive from Yodlee, with any duplicate transactions highlighted in orange. From here, you can import them at the click of a button:
Hmm… not sure. I think there is a slight flaw in the logic - if I want to import the bank feed I want to import the bank feed in total, as that is the “master” document. AT tagging you might add a query for “is this duplicate” or mark somewhere that suspected duplicates were not imported do you want to check.
However, I think I stick with my view that when I import, I want to import. If I imported again then sure it’s a duplicate, but not at first time of asking…
This is the way we’ve always handled duplicates - with the original Barclay’s feed, CSV imports, chrome bank feed and now Yodlee too.
There is no possible way for us to detect a duplicate transaction other than to check the available data. This is unfortunately the current situation with banking APIs. If banks offered their own APIs, I’m sure there would be a better way to handle this, but they aren’t available at the moment.
Previously we discarded duplicate transactions, but now we do keep these on record for them to be imported if you need them.
I think that as a minimum, when logging in for the first time after a feed, there ought to be a warning that some transactions were not included as suspected duplicates…
But we have no way of telling if 2 or more transactions that have the same description, date and amount are duplicates or not.
As I mentioned above, we store these separately so they don’t cause any issues with your bank balance within QuickFile but they can be imported with ease if required.
I’ll certainly pass this on to our development team and perhaps we can add a warning of some sort.
“But we have no way of telling if 2 or more transactions that have the same description, date and amount are duplicates or not.”
I’m not sure that matters. The bank’s account data and the one in QF should surely be in sync? After all the transaction has occurred in the actual bank account and surely the last thing anyone wants is a “copy” of the real account being different?
You would like to think they’re in sync, but that’s not always the case. The issue we have is a scenario such as this one for example:
Feed auto updates at 3am, imports 12 new transactions (all different)
Client pays at 9am
User manually runs feed at 10am, we’re presented 13 transactions
When we look at the bank, as you’re aware from your own bank statements, there are no timestamps, just dates. So in both instances where the feed is run in the example above, we can only check entries since, for example 3rd May 2017. In the scenario above, at the 10am run it would then, the bank would show 13 transactions and Yodlee would present 13 transactions to us. We have to use the duplication checker then to prevent the previous 12 being imported.
Another case could be, as you mention, what if someone adds a transaction manually or imports a CSV? The duplication checker could help prevent duplicates being created (depending on which method you have set up).
There are also scenarios where banks, notability RBS banks (NatWest, Royal Bank of Scotland etc.) change their descriptions of transactions, so again we need to account for this.
I think you are missing a difference in logic here.
When the auto feed runs the transactions do not exist so should be copied. If the user does a manual update then the autofeed already has the first 12 transactions so yes they are duplicates.
What you seem to be doing is checking every transaction against previous ones within the same feed. Rather than import in bulk (which keeps the accounts in sync) and only checking against transactions that are already present…
For example, when the auto feed runs there are 12 new transactions (all of which are essential) as they are after the last transaction in QF. (If not due to manual refresh or entry or upload) then there will be a clash). At 10 am the user does a manual update then there is a clash as the feed has already run.
Now this may mean you need a “holding area” where you compare the feed batch" with the existing transactions" rather than feeding each transaction in at a time and checking to see if it exists already (as I suspect from this conversation you do)
The good news is, we’re both right.
The bad news is, you’re more right than me
I’ve tested the CSV import and even though duplicates are present, they’re imported the first time around, which is correct. It does appear that Yodlee does use the duplication checker I mentioned above, but it should also take into account when things are being imported (e.g. anything that is showing as a duplicate in the same import should not be treated as such).
I’ve referred it to our development team to take a look and will come back to you.
[Edit]
Anything that has been blocked until now, you will find in the Yodlee Raw Feed tab in the feed settings. I would just import them from there for now. Just click “Import”, and we’ll sort the rest!
Getting the filtering logic working correctly for Yodlee has proved to be a challenge. We have a situation with a few UK banks where they retrospectively change transaction descriptions and Yodlee present these to us as completely new and distinct entries, even though they are not. The other issue is where Yodlee go back and fill in gaps for 2-3 days back, we’d expect to poll for the day previous and for that to be a complete and accurate log of all entries, but this is not always the case.
Due to these quirks we’ve had to employ some overly aggressive duplicate checking controls. However if Yodlee pull in a bunch of entries like this in a single batch then it should not consider those items as duplicates. It looks like we need a slight change to the filtering logic to account for this condition.
We’ve escalated this to the development team and we should have that looked at shortly. In the mean time we do keep a separate staging area (@QFMathew mentioned this in the first reply) where you can manually import any missing entries for the time being.
- I think I said that first off lol! Glad we now agree it’s not quite as it should be!
As for the manual import - ended up manually creating as I went through comparing end of day balances to sort out the difference… but now I know…
The only logic I see for why this is happening is the first yodlee feed get added, then the next, which checks for a dupe… rather than checking the batch for dupes…
Just a quick message to let you know that we’ve now released a fix for this. Please can you give it another go when you can, and let us know if that’s working as expected?
- I think I said that first off lol! Glad we now agree it’s not quite as it should be!