#Anti-Public Deduplicator

#Anti-Public Deduplicator

This tool cleans a list of strings (logins, passwords, URLs, etc.) by removing lines that already exist in a "base" folder and removing duplicates.

#Modes

Mode 1 — Anti-Public (base + clean)

Use this when you have a base folder full of .txt files with lines you don't want in the result (e.g. public leaks).

• Every line from the base folder is loaded into memory.
• Lines in your target files that appear in the base are removed.
• Remaining duplicates among the target files are also removed.

Mode 2 — Simple Unique (across files)

Use this when you only want to remove duplicates between your target files. No base folder is needed.

#How to use

• Pick a mode at the top of the window.
• Mode 1 only: select the base folder.
• Select the target .txt files you want to clean (you can also drag and drop them).
• Click Start and confirm.

#How it works

• The base folder is scanned recursively — all .txt files inside it are used.
• A Bloom Filter is built from the base so even millions of lines fit in RAM.
• Each target file is then read line by line. Lines that exist in the base or were already seen are skipped.
• Only unique, non-base lines are saved.

#Output

A folder named ResultBASE_<date>_<time> is created next to the program. Inside you'll find:

• unique_lines.txt — the cleaned result.

The folder opens automatically when processing finishes.

#Stats you'll see

• Files — number of target files processed.
• Unique lines — how many lines made it into the result.
• Base lines — total lines read from the base folder.
• From base — how many target lines were removed because they exist in the base.
• Dupes — how many target lines were removed as duplicates.
• Total time spent.

#Notes

• Files are read as UTF-8 with errors ignored — bad characters won't break processing.
• The tool can handle very large files thanks to the Bloom Filter.