7/22/2026 at 6:52:20 AM
by ingve
7/26/2026 at 6:14:55 PM
Why do you need a cryptographic hash function for detecting duplicates? xxHash, rapidHash or anything that’s fast will do the job. There’s no need to be secure against inverses.by divyekapoor
7/26/2026 at 9:33:42 AM
Wow. Wasn’t expecting to see this here… I am the author…by jdefr89
7/26/2026 at 1:52:09 PM
Quick question, did you use an LLM to help with this? (Not judging one way or another).The only reason I ask is that for the past couple of weeks I’ve been making a similar utility to test out the Sol model and it’s hilarious how similar the architecture and even the CLI switches are to what Codex came up with.
by ac2u
7/26/2026 at 2:09:43 PM
(Sorry if I missed anything obvious from the documentation)Does this work on Windows
by albert_e
7/26/2026 at 12:27:55 PM
I'm not sure exactly how it compares, but a similar tool you can use that also does phashes of images/videos to find potential duplicate media files is czkawka [0].by psYchotic
7/26/2026 at 2:27:49 PM
I have a thing like this - one thing I have found useful is also going up the directory tree to find entire duplicated directories. (So dir hash is hash of the concatenation of the subdirs and the files).I also used go, it excels at just this sort of thing. Although I saturate the disk to memory bandwidth (for the SHA) way sooner than CPU on my laptop, so it still takes a while to run.
Nice.
by lanstin
7/26/2026 at 4:03:49 PM
I wrote a tool a short bit ago to handle massive parallel hashing. Originally, it was a testbed for a project I built to make multiprocessing easier in Go.https://github.com/indrora/hyperhash
In it, I learned that there's some interesting quirks about the way the traditional shaXsum tools work, specifically that they focus on a set of inputs from the shell command line, which means that you can exhaust the shell command line length fairly easily (try hashing every file in the Linux source tree, for instance; zsh, bash, and fish will all eventually tell you "Fuck off, there's too many characters in argv")
I also discovered that Go's handling of file handles varies drastically based on OS. Windows, for instance, has no problem with you taking a file handle to every single file on a disk. Linux will get upset but eventually capitulate, macos will stomp its feet and inform you that you are a bad child and kill you off.
by indrora
7/26/2026 at 7:19:06 PM
I always have all these parallel things take a configurable number of worker go routines, then try different numbers and run btop or iostat to see how close I am to being hardware bound. Never had a machine where the limit was anywhere close to the number of open FDs allowed. I guess I haven't proven this but my idea is having the generic scheduler have to sort 60k FDs or 2M (I have a lot of dup files in my "backup everything, randomly and repeatedly, since 1998 file system) FDs is going to thrash stuff more than just having go's runtime and my own code.by lanstin
7/26/2026 at 11:33:57 AM
Here's a tip, if you are making regular compressed backups of the configuration of your network devices, if you use gzip, make sure to set the mtime to 0 when writing the file as this can throw off deduplication. This also applies to the name field.https://docs.python.org/3/library/gzip.html#gzip.GzipFile.mt...
by chaz6
7/26/2026 at 1:12:40 PM
Reflink conversion would be another nice feature to have. It allows userspace apps to create filesystem-native copy-on-write clones of files, so to the next app they work exactly like another copy of the same data. But the data is on disk only once.by Maakuth
7/26/2026 at 5:07:12 PM
> …finds duplicates across large disks instantly:D
by bronlund
7/26/2026 at 8:39:55 AM
Interested to know what algorithm you use for --fuzzy. Presumably not any of the worst-case-quadratic diff algorithms... MinHash on n-grams?by akoboldfrying
7/26/2026 at 9:34:58 AM
I implemented it quite quickly I use SimHash for now but I need to refactor and possibly replace…by jdefr89
7/26/2026 at 8:30:09 AM
imo fclones is the one to beat, https://github.com/pkolaczk/fclones#benchmarksnice that there are actually some ok interfaces here. with fclones, i tend to generate a file of candidates then do a little review to make super certain everything is square / as expected. it's very built for intermediary files a core pattern, which is very unix, very convenient. but it did take me a little while to settle on this, and it felt like i wasn't being offered a ton of options for management out of box. looks like some real attempts to be more user friendly here.
by jauntywundrkind
7/26/2026 at 2:31:45 PM
Do you know if fclones implements anything similar to https://github.com/hpc/libcircle ?I also wonder if nftw(3) would be faster than readdir(3).
by ranger_danger
7/26/2026 at 3:42:48 PM
[dead]by daflkfdslkfds