When you add your Fastmail account to your phone or computer using CardDAV for contact syncing, this does not copy any contacts (often synced to iCloud) already on your device back to Fastmail.
For instance, if you already have an iCloud mail account and you add your Fastmail mail account: your phone will show emails in both but it doesn't copy the emails automatically between them.
If you want to have all your contacts synced between your device and Fastmail, you need to export them from your current provider and then import them into Fastmail. CardDAV will then make sure that your contacts stay up to date on your device and on Fastmail.
Moving my device contacts to Fastmail?
You'll first need to export your contacts from your device. Look at the help for your device to find out how to do this.
Then import the contacts file into Fastmail using our tool. You'll need to manually check for duplicates.
Your device will end up with two copies of these contacts: one from the CardDAV sync from Fastmail, and the original copy on your device. You can now safely remove the original copy and just use the synced contacts.