#Duplicate removal

This tool removes duplicate lines from large files. You can choose one of two modes depending on your file size.

#Modes

1. Memory mode — fast, but needs enough RAM (about twice the file size).

• Loads the whole file into memory.
• Removes duplicates.
• Saves sorted unique lines to a new file.

2. Disk mode — slower, but works with huge files that don't fit in RAM.

• Splits the file into small sorted parts on disk.
• Merges the parts and removes duplicates with minimal memory use.

#Before starting

• Checks that you have enough free RAM (for memory mode).
• Checks that you have enough free disk space (for disk mode).
• Cleans the temporary folder automatically.

#How to use

• Choose the mode from the menu: memory, disk, or exit.
• Select the input file.
• Select the output folder.

#Output

After processing, you'll see:

• Total lines.
• Unique lines.
• Number of duplicates removed.
• Time elapsed.

#Notes

• If a file is locked by another program, the tool will retry opening it.
• The output filename is generated automatically.