Ask HN: Personal photo library recommendations? Open source, browser-based

Shameless plug: I'm currently working on a solution called Photonix, though it's still very much pre-1.0 at the moment.

Installation is fairly simple with Docker, frontend is web-based (React), backend is Python with a sprinkling of Tensorflow. So far auto-tagging of photos by location, object detection and colour is fairly decent. UI is progressing and is useable on most devices, though quite minimal.

Please feel free to check out the demo site and the GitHub issues. I'd really appreciate feedback and help. Thanks.

zerkten · 7 years ago

Will you have support for handling duplicate images? People are either super organised, or the complete opposite. When you have multiple copies for backup and then want to consolidate to a library you want to hide your messiness.

I've been struggling to find a tool that handles that handles the duplicates problem within a web interface. I've been experimenting with some approaches including perceptual hashes and was wondering if that's something you'll include?

Is there any way to use metadata from Lightroom Catalogs, or enable people tagging in your current implementation?

damianmoore · 7 years ago

I definitely hear you regarding duplication. This whole project evolved from a script I wrote to do just that. There is a concept of multiple versions (files) of an image so different edits with the same metadata timestamp will show only once and you'll be able to select the preferred one. I prefer to handle things this way to start and then suggest deletions based on same hash rather than deleting up front.

I don't have any experience of Lightroom but I can have a go at reading the files if you think it's useful.

damianmoore · 7 years ago

Face detection and recognition is definitely planned and looks like it should be entirely doable.

ConfusedDog · 7 years ago

I really like it, though I'm not photographer, so I won't be able to speak for your main user base. One thing I would like is when I open a large image, it should give me a loading status. Instead, it just blacked out for a moment and image suddenly loaded.

damianmoore · 7 years ago

Thanks for the feedback. I agree, there should some progress display while downloading. I'll be sure to make sure that's there. There are other optimisations I want to make in this area like selecting a resized version to download depending on the current screen dimensions and pixel density. This will improve loading times.

zerkten · 7 years ago

As a developing photographer, having the biggest image isn't as important as having a large-ish image with additional metadata.

alfredo_burgess · 7 years ago

Hey mate,

I have checked this out, running the docker-compose method, and kudos for your work. Looks great.

One issue I ran into was regarding videos (tried with a couple of MOV and MP4) - it doesn't generate a thumbnail and in fact throws an error along the lines of: File "/srv/photonix/photos/models.py", line 84, in base_image_path AttributeError: 'NoneType' object has no attribute 'base_image_path'

Happy to open a gitHub issue, but thought I'd drop you a line here to see if it was your intention to support videos (which would be cool!)

Cheers,

jayniz · 7 years ago

I have ~1TB of photos from various sources, including backups of old iPhoto libraries. Therefore I often have one photo available in many different sizes, cropped thumbnails, etc. I'd pay money for working duplicate detection :)

damianmoore · 7 years ago

OK, noted ;)

bloopernova · 7 years ago

Does Photonix work with a read-only volume for /data/photos?

I'm looking at the docker-compose.yml and wanted to give it a go, but not allow it any way of deleting anything :)

damianmoore · 7 years ago

Yep, it should work fine with a read-only photo volume as it doesn't make any changes to photo files by default - it just needs to write to the DB which is configured as another container in the Docker Compose file.

barrystaes · 7 years ago

I like this architecture and wil spin up a Photonix docker on my Unraid server in a few weeks or so. I wonder why you used graphql for the api?

damianmoore · 7 years ago

Thank you. I'd like to hear how that goes.

The thinking behind GraphQL was to allow for advanced filtering, supporting all the attributes we store without a lot of extra API work. The GraphiQL web interface makes it quite nice to explore the data and is bundled in and accessible at /graphql . GraphQL also has "subscriptions" which allows for pushing data from the server. The Apollo JS library I used also provides built-in extras like caching.

Personal plug: I'm working on PhotoStructure, after trying many, many open source photo projects (and being a committer for years of one of the most popular, "gallery.")

PhotoStructure is browser-based (using Vue), and scales to hundreds of thousands of assets over millions of files. Your library can be created on a Mac, saved on your NAS, then later opened and managed by a Linux box, seamlessly. Raw images have highlight restoration before rendering previews. Videos are auto transcoded for mobile and desktop web use. Corrupt images are detected automatically and culled. Image source sets are used to minimize network data and maximize viewing quality. XMP sidecars are imported for metadata. Importing aggressively coalesces duplicate images and videos using direct and inferred metadata, so even your downsized Google photos takeout will be deduped with your originals.

Once you've got a huge library, though, it needs a novel UX. Scroll-reverse-chron and a search bar shouldn't cut it. PhotoStructure has a couple novel and unique approaches to navigation, which you can read about here: https://blog.photostructure.com/introducing-photostructure/

It scales down to odroids, and up to as many CPUs as you can throw at it, and self-throttles CPU during library sync so the machine is still useable. Installation takes under a minute, and updates are automatic.

It's closed-source because it's how I want to pay for my food and clothing, but it's a corporate mandate to open source in case of business closure, which is also explained in that blog post.

I'm sending out another wave of beta testers later today, and during the beta it's free. I'm giving heavy discounts to my beta testers that share feedback.

I'd love to hear what you think.

xtracto · 7 years ago

Personally I don't like this response. The post specifically asks for Open Source software, and the top response is a closed-source service?

I am not making any judgement on the service, just that it is not an appropriate reply for what is being asked here.

mceachen · 7 years ago

Yeah, I hear you.

Having written many open source libraries (my rubygems have been downloaded several million times, my node packages are close to that, and I've contributed to other libraries for over a decade), I personally will choose open source projects over closed-source because I don't want to be victimized by abandonware or corporate whims.

It seems like photo software (both closed and open source) is especially prone to dying on the vine. It's a common need, and it's easy (and fun) to write a simple script that makes thumbnails from a folder of images. I seen countless photo projects on github, but as complexity ramps up quickly, the installer script (if there is one) breaks, updates fail, there aren't any tests, and the author gets bored and moves on.

I guess I felt justified here because a) it is self-hosted, and b) I'd added my corporate mandate to open-source the codebase in case of business closure. (I don't remember a corporation doing this open-source-on-close before, do you?)

I've actually already open-sourced some of the trickier bits: https://exiftool-vendored.js.org/ and https://batch-cluster.js.org/. I expect that to continue.

icebraining · 7 years ago

While not open source, it doesn't seem to be a service, but a self-hosted application.

pheeney · 7 years ago

I have been looking for a photo meta data app for a long time, not sure if this is something you have/will offer.

I am essentially looking to leave the original photos on external drives and have an app that indexes them and stores a customizable thumbnail with the app to view them on my local machine. This way I can browse through all my photos and figure out the original file path if I want to retrieve them. The most important aspect is that I can take the drives offline while the thumbnails and index remains within the app and re-indexing when connected again.

My existing workflow is to import all photos on my mac to the Photos app. I pull them from different devices (phones, camera, etc). The photos app does an OK job at de-duping any matches. I also run PhotoSweeper to further de-dupe which analyzes the photos itself and I can leave the best ones remaining. After that I run some custom scripts to export the data as [year]-[month]/[year]-[month]-[day] [hour].[min].[sec].jpg. I then merge those onto my external drive and kick off my backup process to clone them to other drives and sync to cloud.

The closest I have come is Lightroom which indexed the drive a little bit but the "thumbnails" and catalog is huge. It allows some tagging and other features to discover old photos but some processes are a bit manual. This workflow seems to be common among digital asset management software which is expensive and way more then I am looking for.

mceachen · 7 years ago

Yes, you can absolutely use PhotoStructure for your use case.

After you install, the second question you should pick "No thanks, I like my photos and videos where they already are." See https://support.photostructure.com/automatic-library-organiz...

Email me at hello at photostructure.com or sign up for the beta. I'd love to hear what you think!

MrGilbert · 7 years ago

Also just signed up for the beta. :) If I can choose, I'll pay for a project that allows self-hosting rather than pumping my pictures into the cloud. I'm doing the same for Bitwarden, my go-to password manager.

jayniz · 7 years ago

Signed up for the beta! I've been looking for something like this for my ~1TB photo database ranging back to 1999.

meepl · 7 years ago

awesome, just signed up for the beta

sfifs · 7 years ago

I created NasPics specifically for this reason. I wrote it in pure Go with no dependencies for ease of cross compilation since my NAS runs a weird old Debian based distribution of Linux on ARM. Running it is just cross compiling a binary, copying over and executing. For restart across boots, I just added a simple SystemD service file.

https://github.com/srinathh/naspics

(edit: some data destination paths are probably hard coded since I wrote it for myself but can be easily broken out into command line options if needed)

hwj · 7 years ago

I appreciate the "no dependencies" but it seems the client still needs NodeJS:

> It is build on React using the Create React App tool, written in Javascript and requries a NodeJS development environment.

It's needed just to build the front end on dev machine. The front end HTML JS and resource files are embedded into the binary executable during compilation.. so i literally just copy over a binary

asark · 7 years ago

You need node to "build" a React app, but not (generally) to serve it. Can typically just stick it on S3 or in an Apache htdocs directory or whatever. Unless your backend is Node and you're doing some server-side-rendering stuff.

howerkraft · 7 years ago

Do you have a demo somewhere or some screenshots of the client?

Yeah I should probably improve the README file.

https://m.imgur.com/gallery/gNFw1PU

https://m.imgur.com/gallery/NzatdKZ

canada_dry · 7 years ago

I'd love to find one that incorporates the functionality that Picasa (windows desktop) circa 2000 i.e. face recognition/matching.

It was way ahead of it's time, and actually worked!

Best of all, it did everything locally... not cloud based and thus retained privacy of your personal photo collection.

Picasa was so useful, so of course Google had to kill it.

It took multiple disparate photo directories and presented everything in a timeline of folders. And because everything was local, that happened quickly, rather than waiting for your browser to get the next 100 photo results from a javascript call or whatever.

Are there any photo clients for windows that present multiple folders as a single coherent timeline? And can manage tens of thousands of pictures? I've got stuff going back to the late 1990s and would love to be able to find all those old cat pictures or whatever.

> Are there any photo clients for windows that present multiple folders as a single coherent timeline?

PhotoStructure does this (and I believe is the only software that does robust time zone inference, as well). (I've posted elsewhere here with more details).

I'm now trying Shotwell, installed in my WSL Ubuntu instance under Windows, displaying to VcXsrv. Thankfully it can import-in-place rather than trying to copy gigabytes of photos. We'll see if it copes OK with the funky permissions and file layout I've got set up.

jamesbr · 7 years ago

I use and like Mylio (https://mylio.com/). Does face recognition & syncs to other devices without being in the cloud. I migrated from Aperture/Picasa to Mylio.

hazz99 · 7 years ago

Google Photos does some really handy stuff with facial recognition, which was super helpful when searching & collating photos.

Doesn't help much for privacy, however.

alphabettsy · 7 years ago

Privacy is a major point of concern along with the fact that friends might use this service submitting photos of people who maybe don’t want this tech used on their likeness.

froindt · 7 years ago

My biggest complaint is how long it takes for facial recognition to occur. I like searching by name, and if I do a large upload, I might have to wait a couple weeks to get the groupings. I don't get a push notification, I have to go check and suddenly I have 100 faces to tag.

dashundchen · 7 years ago

Digikam has facial recognition and tagging using OpenCV. Can't speak to its accuracy though.

This looks promising! Thanks.

I started writing PhotoStructure because I got tired of being burned by failed open and closed source media apps. I wrote more details about it in a top level post, but this blog post describes why I quit my job to do this full time, where the project is currently, and where it's going. https://blog.photostructure.com/introducing-photostructure/

I'd love to have you try out the beta and have you share feedback!

bpye · 7 years ago

I've signed up for your beta.

One question though - if you end up feeling it's unsustainable to continue developing PhotoStructure down the line do you have a plan? Obviously we would rather not be burned by it either, you you consider at that point making it open source so existing users can continue and make improvements?

Certainly I hope it never comes to it but it's nice to have a little reassurance.

Signed up, though a quick browse I wasn't able to find any mention of face matching/recognition.

It's really the killer-feature that made Picassa so great - find all the photos over several years of a family member or friend.

Sadly, I've needed this feature for funeral photo albums lately and could really use the old Picassa!

jonahbenton · 7 years ago

Not specifically for photos, but I have been a user of Perkeep, a long running project from one of the golang maintainers that focuses on long term storage of one's data:

https://perkeep.org/doc/

zimpenfish · 7 years ago

Although it's no longer actively maintained (IIRC) and some of the importers are currently broken for me (Mastodon importer can't talk to Pleroma, Pinboard has a JSON error, Twitter gives random "account not found", etc.)

Which is a shame because I really like Perkeep/Camlistore as a concept.

vorpalhex · 7 years ago

Could you try to troubleshoot some of the issues and open PRs? It doesn't take an army to resuscitate a project...

-- Edit --

Actually on checking, it doesn't seem like Perkeep is at all dead. I'm seeing several updates in the last month...

preek · 7 years ago

I did exactly that a couple of months ago. After exporting my pictures, I ran fdupes to find duplicates and then imported all images into Shotwell.

I can still take pics on the phone which will be synced via Dropbox and Shotwell picks them up immediately. The sync is faster than I’m used to on iCloud and finally I just have files that I can tag, again.

Shotwell is also super fast, has a similar UI to photos.app (automatic events for example), but it also had hierarchical tags which it can even write to the files itself. So it’s very simple and yet portable without lock-in. Couldn’t be happier. Of course ymmv.

Good luck!

andyjohnson0 · 7 years ago

I've been using Lychee [1] on a vps with good results.

It looks pretty good, has multi-user capability, metadata editing, etc. It would be nice if it had some geotagging integration and ability to group albums into sets.

I filed a bug report and the developers/maintainers fixed it very quickly.

[1] https://github.com/LycheeOrg/Lychee

dsd · 7 years ago

I've been helping on the laravel v4 project. The devs are pretty friendly and I haven't found anything with a better UI than lychee. Three project doesn't do everything but what it does do, it does well.

Do any of these open-source solutions support Apple's Live Photos and videos alongside ordinary photos? I don't really want to have multiple applications for my family memories stuff. Just want to toss them in a directory and let on-photo/video-file metadata tags sort it out. Face auto-tagging and geo- and time-based grouping a must. Otherwise, can (ideally, would, in fact) just serve up dead-simple ugly HTML, if it's gonna be a web-tech thing, to minimize the developer workload if I have to take it over because the project dies.