|
| | | |
D
U
P
F
I
L
T
E
R
| | | | |
dupfilter-0.1
--------------
---
This is 'dupfilter', a proggy that can remove duplicated lines form text
files. Thanks to 64bit FNV1a hash algorithm it is both very effective and
very fast.
---
NOTES:
[*]
Requires 68020(no FPU)+, OS2.04(theoretically)+, 64+ kilo of free memory.
[*]
It should be possible to use this program on streams too, so that constant
filtering can be provided.
[*]
If you use my other such proggy called 'deduuupe' then it is time to switch
as this one does the job so much quicker.
[*]
Aside from real duplicates empty lines will also be removed!
---
HELP:
> dupfilter ?
IF=INPUTFILE/A,OF=OUTPUTFILE/A,TL=TURBOLEVEL/N,IC=IGNORECASE/S
IF=INPUTFILE/A - Input file(with duplicates).
OF=OUTPUTFILE/A - Output file(without duplicates).
TL=TURBOLEVEL/N - Hash indexing space(must be to the power of 2).
This can eat a lot of memory. Value of 128 needs
2 megs and its the default! Pass 0 if you want just
the linear search.
IC=IGNORECASE/S - Duplicate removal is case insensitive.
---
USAGE:
; Cleanup something
dupfilter dirty.txt clean.txt
---
megacz
| |
| | | | |
|
