I ended up finally transferring everything from my old gmail.com based Google account over to my custom domain Google account. I have been running them in parallel for ages switching between them to access different services. I read a number of blog posts and forum discussions on how to do this, also took a look at Googles own documentation.
According to Google you can merge two Google accounts but only if both our in the same organisation. And since my "new" one was the only one actually in an organisation and the other one isn't this wasn't an option. I couldn't figure out, from the documentation alone, if merging two gmail.com based Google accounts are possible or if it has two be two proper organisation/custom domain accounts.
Started out with Google Drive by sharing all my files from one account to the next. Then I could just copy the files on the new account. The issue I ran into was that then all the filenames started with "copy of". I didn't want that so I started looking at other options.
By just selecting all the files and select download I received a zip file with all my files. Most of my files has been created on Google Drive with the built in apps and they got converted to excel and word respectively when downloaded. Then I downloaded the Google Drive app for windows, made sure I turned on the "convert uploaded files to Google..." option and then just dropped all the files in there.
This was a bit more tricky then Google Drive. I found a nice option of sharing my entire repository between the accounts and could set the new account the automatically save the shared photos to it's of repository. It did how ever take ages so I quit that and just used Google Takeout to download two huge zip files with all my photos. I then configured the Google Drive app to backup the folder with the unzipped content and add photos and videos directly to Google Photos.
Yes I lost all my albums but the improvements of the Google Photos Assistant gives my correct suggestions for creating albums matching my trips and events daily now.