Photo library management software (duplicate removal!)

[alsoknownasDean]

So I've got photos everywhere. Hard drives, USB sticks, various cloud services, old iPhoto libraries from when I used a Mac, etc. Duplicates upon duplicates upon duplicates. I've probably got over 150GB used up for maybe 30-40GB of photos.

Instead of replacing the almost-full SSD in my laptop with a larger one, I need to sort it out once and for all! I'd like to keep a local copy (including on an external HDD), but I'm happy enough to have it sync to a cloud service.

What does everyone suggest for photo management software for Windows 10? Preferably something that can sort them all into date-arranged folders.

I've tried Google Photos, but I'm guessing that'd be an utter nightmare to download and sort a local copy once I've uploaded them all to Google (although it'd do a good enough job with duplicates). I tried Shotwell when I was using Linux for a while, and it did a good job, but there's no Windows version. Windows 10's default Photos app doesn't seem to do anything with duplicate management. The Photos app on macOS is not as good as iPhoto was, and of course when I moved away from a Mac I had photo libraries from both apps.

For this purpose, I don't mind paying for good software. There's a bunch of programs on the Microsoft store, but much of it seems pretty low-grade.

Thanks :)

[Boll weevil]

I currently use the low tech method of grouping all photos associated with a single trip/event in their own folder. Sub folders are used to group by camera and to hold the processed output files (edited jpgs).

I would also recommend running a program that looks for duplicate files. I don’t remember which one I used, but it was pretty powerful... it would even identify files with different names that were otherwise identical. And it wasn’t limited to pictures. It would also go through other file types like PDFs.

[IsThisAGoodUsername]

Why do you want to keep a local copy of all these pictures?

For the few remaining batches of photos that I haven't yet uploaded to Google Photos, I use a naming convention of yyyy-mm-dd subject which ensures all folders are sorted chronologically.  So my "2020-09-02 Mustache Trimmer Collection" folder of photos would always appear before my "2020-09-08 FIRE Bending Vacation" photos.

[alsoknownasDean]

Why do you want to keep a local copy of all these pictures?

For the few remaining batches of photos that I haven't yet uploaded to Google Photos, I use a naming convention of yyyy-mm-dd subject which ensures all folders are sorted chronologically.  So my "2020-09-02 Mustache Trimmer Collection" folder of photos would always appear before my "2020-09-08 FIRE Bending Vacation" photos.

I'd like to retain the 'originals' and not entrust my entire photo library solely to cloud storage, even if it's convenient when accessing from my phone.

Has anyone used Phototheca?

I'm honestly amazed at how difficult it has been to find Windows software that doesn't seem kinda spammy and low-grade. I'm happy to pay good money for the right software, but I've no idea what the right software is :)

What I want to be able to do is dump all the photos into the application, have it sort them into folders, and give me the ability to remove duplicates from it. I don't want to have to manually put it all into folders because it's a hell of a job.

[Daley]

I wound up needing to do this a couple years ago after my own transition away from Linux after some fifteen years or so back to Windows, and encountered similar struggles. So I know the pain first-hand, Dean. What I eventually used was a two-fold approach using a couple older and more obscure open-source apps that look pretty low rent and oldskool, but worked great and was faster and more accurate than anything else I'd tried before it without charging me money.

First thing I did was migrated all the various photo directories together as sub-directories into a common directory outside of the default Windows Pictures folder in your profile.

Second, I ran Duplicate Files Finder on the directory to quickly identify and purge out the obvious duplicate stuff, skewing towards the older timestamped file versions for any duplicates found.

Third, I ran VisiPics on the directory, and found all the duplicate files that were resized, edited, and saved in alternate formats. Again, I skewed towards the oldest versions in tact, and the originals when possible.

Finally, I just let the Microsoft Photos app import everything into the Pictures directory and auto-generate albums for me. This step? Use what you want to organize what's left like DigiKam, or Adobe Bridge, but I just rolled with the defaults and Photos worked for me. YMMV. Once the import was finished and I verified all the photos got copied over into the Pictures directory, I went ahead and deleted the folder I worked from in the first step.

[alsoknownasDean]

I wound up needing to do this a couple years ago after my own transition away from Linux after some fifteen years or so back to Windows, and encountered similar struggles. So I know the pain first-hand, Dean. What I eventually used was a two-fold approach using a couple older and more obscure open-source apps that look pretty low rent and oldskool, but worked great and was faster and more accurate than anything else I'd tried before it without charging me money.

First thing I did was migrated all the various photo directories together as sub-directories into a common directory outside of the default Windows Pictures folder in your profile.

Second, I ran Duplicate Files Finder on the directory to quickly identify and purge out the obvious duplicate stuff, skewing towards the older timestamped file versions for any duplicates found.

Third, I ran VisiPics on the directory, and found all the duplicate files that were resized, edited, and saved in alternate formats. Again, I skewed towards the oldest versions in tact, and the originals when possible.

Finally, I just let the Microsoft Photos app import everything into the Pictures directory and auto-generate albums for me. This step? Use what you want to organize what's left like DigiKam, or Adobe Bridge, but I just rolled with the defaults and Photos worked for me. YMMV. Once the import was finished and I verified all the photos got copied over into the Pictures directory, I went ahead and deleted the folder I worked from in the first step.

Thanks for that Daley, that sounds really good.

I'll have a look into those apps. I used a really good duplicate remover in OSX (from the Mac App Store) for one of my libraries stored there, but of course it's probably not available for Windows. So much of the software on the Microsoft Store looks really low quality unfortunately. The open-source stuff is probably better there. Hopefully VisiPics will be able to identify thumbnails and remove those.

Would Microsoft Photos be able to sort the files into folders by year/month/day?

I've got about 150GB of photos on the SSD in this laptop (much of which is in folders like 'imported from Dropbox' or 'from old phone' and full of duplicates, along with a bunch of external hard drives and USB thumb drives, backups on iCloud, Amazon Photos, Dropbox, Google and OneDrive, and I'm still paying for three of those services (iCloud, Google One and an Office 365 sub). I really need to sort it out once and for all, and then get rid of most of my USB drives. It's a real mess.

Plan would be to have it on one cloud service, one external HDD or USB stick here and another one at my parents' place for off-site backup. Hopefully there's a more elegant way to sync the library to the external media than simply copying files over.

I've even been paying extra for an Internet plan with a higher upload speed because I've been meaning to do this and sync it all to the cloud (I was on an ADSL plan with a 1mbps upload until November, now on a plan with 40mbps upload). Sort it out, drop to a cheaper plan and cancel at least one of the cloud services (I don't really need iCloud in particular).

[Daley]

I'll have a look into those apps. I used a really good duplicate remover in OSX (from the Mac App Store) for one of my libraries stored there, but of course it's probably not available for Windows. So much of the software on the Microsoft Store looks really low quality unfortunately. The open-source stuff is probably better there. Hopefully VisiPics will be able to identify thumbnails and remove those.

Yeah, the Microsoft store, much like the Android store, is a bit rough... but there's some gems there. What I can't find there, instead of installing various apps to the system, I fill out the rest of my app needs as much as possible using PortableApps. This lets me lock the system down to near Windows 10S levels, but lets me run and update apps that don't need admin privileges to install or update in system locations.

This also lets me back up my non-MS Store apps so they're just there after a system restore. I'll have the PortableApps launcher load on login so it auto-updates, and once it's done, I just close it out so it isn't running in the background. I've also got these scripts that auto-adds, updates, removes and replaces shortcuts from PortableApps into the Windows menu so I don't need the PortableApps launcher... but I can't remember where I got it. If you'd like it, I can email you a copy. This is the method that I use to install Gimp, Inkscape, LibreOffice, Audacity, KeePass XC, and VLC. In fact, I also keep a copy of Firefox on the thing, and use a massive micro USB thumb drive partitioned into two separate Bitlocker encrypted drives with PortableApps installed on the first partition, and the second partition used for my system backup drive with Windows. This gives me an alternative sneakernet backup of my system where most of my important apps and files can just be plugged into any Windows machine and accessible once I enter the password.

Would Microsoft Photos be able to sort the files into folders by year/month/day?

Photos just kinda lumps everything together in a "Camera Roll" folder, and sorts everything in app by metadata and facial recognition. If you want a more fine-tuned directory structure, you might want to use DigiKam instead.

I've got about 150GB of photos on the SSD in this laptop (much of which is in folders like 'imported from Dropbox' or 'from old phone' and full of duplicates, along with a bunch of external hard drives and USB thumb drives, backups on iCloud, Amazon Photos, Dropbox, Google and OneDrive, and I'm still paying for three of those services (iCloud, Google One and an Office 365 sub). I really need to sort it out once and for all, and then get rid of most of my USB drives. It's a real mess.

Plan would be to have it on one cloud service, one external HDD or USB stick here and another one at my parents' place for off-site backup. Hopefully there's a more elegant way to sync the library to the external media than simply copying files over.

I've even been paying extra for an Internet plan with a higher upload speed because I've been meaning to do this and sync it all to the cloud (I was on an ADSL plan with a 1mbps upload until November, now on a plan with 40mbps upload). Sort it out, drop to a cheaper plan and cancel at least one of the cloud services (I don't really need iCloud in particular).

I know the pain and feeling with the speed upgrade myself. The wife's having to teach online this semester, and Zoom is bandwidth hungry. We finally had to switch over to a 60/60 fiber plan to get some decent upload speeds, but it's opened up some... possibilities.

Look into Nextcloud, there's builds for Raspberry Pi. You also might find a purpose and use for Cryptomator.

[alsoknownasDean]

Yeah, the Microsoft store, much like the Android store, is a bit rough... but there's some gems there. What I can't find there, instead of installing various apps to the system, I fill out the rest of my app needs as much as possible using PortableApps. This lets me lock the system down to near Windows 10S levels, but lets me run and update apps that don't need admin privileges to install or update in system locations.

This also lets me back up my non-MS Store apps so they're just there after a system restore. I'll have the PortableApps launcher load on login so it auto-updates, and once it's done, I just close it out so it isn't running in the background. I've also got these scripts that auto-adds, updates, removes and replaces shortcuts from PortableApps into the Windows menu so I don't need the PortableApps launcher... but I can't remember where I got it. If you'd like it, I can email you a copy. This is the method that I use to install Gimp, Inkscape, LibreOffice, Audacity, KeePass XC, and VLC. In fact, I also keep a copy of Firefox on the thing, and use a massive micro USB thumb drive partitioned into two separate Bitlocker encrypted drives with PortableApps installed on the first partition, and the second partition used for my system backup drive with Windows. This gives me an alternative sneakernet backup of my system where most of my important apps and files can just be plugged into any Windows machine and accessible once I enter the password.

That's a good idea, but at this stage I don't really need it. I just try and minimise what I have installed on the PC. It's easy enough to reinstall where needed.

I've found that a mixture of Duplicate Files Finder, Systweak Photo Organizer and Duplicate Photos Fixer Pro has done most of what I needed (along with VisiPics for some items but I found I couldn't mark them by date).

Duplicate Photos Fixer Pro was the app that I found very useful in OSX, buuut it's $2.99 upfront on the Mac App Store at the moment, and $60 per year on the Microsoft Store. Hmm. I ended up holding my nose and paying it because I'll turn off any auto-renew as soon as I'm done, and I don't mind paying for a tool that does exactly what I want it to do.

The Systweak Photo Organizer sorts the photos in the year-month-day order that I wanted (from the metadata I guess), however it doesn't sort video files or RAW files. I'll have to do those separately, or find some other software.

So far all the photos (excluding any video content or RAW) is about 25GB.

When I signed up to this plan, it was $85 a month (up from the $60 of my old ADSL plan). They've increased the price twice and I'm grandfathered for a little while longer, but in November apparently it'll go to $105 per month. Going down from 100/40 to 50/20 will bring that back to $75.

[alsoknownasDean]

I've revisited this project lately. This time I've been using [urlhttps://dupeguru.voltaicideas.net/]dupeGuru[/url] to identify and remove duplicate files. Now it's almost done aside from cleaning up my documents folder and any video files and RAW photos and Systweak Photo Organizer doesn't deal with.

Part of the issue was that I didn't want to be doing it all on external drives, so I bought a 1TB Crucial MX500 SSD and swapped it into my laptop so I could sort it all locally. I've recently gone down from 300GB used  to just over 100GB used on the SSD (of which photos are just over 40GB). Not sure if it's worth swapping the old 256GB SSD back in or not :)

Probably unnecessary given I should have used dupeGuru to start with on the external drives (all since formatted), but whatever. I guess the other software had advantages with being able to mark via the dates in the metadata rather than just going through exact matches.

I might try to see if I can re-import the files using Photos due to the formats not covered by Photo Organizer.

Now to work out what to do with the spare drives. One USB drive might hold a backup of the Photos folder stored remotely, a couple may have a Windows 10 ISO and drivers stored just in case, things like that. Maybe I'll put together a NAS with a Raspberry Pi 4 and some external drives for a local backup? :)