Ykhjuy

I have a file with multiple lines. I want to know, for each word that appears in the total file, how many lines contain that word, for example:

0 hello world the man is world

1 this is the world

2 a different man is the possible one

The result I'm expecting is:

0:1

1:1

2:1

a:1

different:1

hello:1

is:3

man:2

one:1

possible:1

the:3

this:1

world:2

Note that the count for "world" is 2, not 3, since the word appears on 2 lines. Because of this, translating blanks to newline chars wouldn't be the exact solution.

Another Perl variant, using List::Util

$ perl -MList::Util=uniq -alne '

  map { $h{$_}++ } uniq @F }{ for $k (sort keys %h) {print "$k: $h{$k}"}

' file

0: 1

1: 1

2: 1

a: 1

different: 1

hello: 1

is: 3

man: 2

one: 1

possible: 1

the: 3

this: 1

world: 2

It's a pretty straight-forward perl script:

#!/usr/bin/perl -w

use strict;



my %words = ();

while (<>) {

  chomp;

  my %linewords = ();

  map { $linewords{$_}=1 } split / /;

  foreach my $word (keys %linewords) {

    $words{$word}++;

  }

}



foreach my $word (sort keys %words) {

  print "$word:$words{$word}n";

}

The basic idea is to loop over the input; for each line, split it into words, then save those words into a hash (associative array) in order to remove any duplicates, then loop over that array of words and add one to an overall counter for that word. At the end, report on the words and their counts.

Straightfoward-ish in bash:

declare -A wordcount

while read -ra words; do 

    # unique words on this line

    declare -A uniq

    for word in "${words[@]}"; do 

        uniq[$word]=1

    done

    # accumulate the words

    for word in "${!uniq[@]}"; do 

        ((wordcount[$word]++))

    done

    unset uniq

done < file

Looking at the data:

$ declare -p wordcount

declare -A wordcount='([possible]="1" [one]="1" [different]="1" [this]="1" [a]="1" [hello]="1" [world]="2" [man]="2" [0]="1" [1]="1" [2]="1" [is]="3" [the]="3" )'

and formatting as you want:

$ printf "%sn" "${!wordcount[@]}" | sort | while read key; do echo "$key:${wordcount[$key]}"; done

0:1

1:1

2:1

a:1

different:1

hello:1

is:3

man:2

one:1

possible:1

the:3

this:1

world:2

A solution that calls several programs from a shell:

fmt -1 words.txt | sort -u | xargs -Ipattern sh -c 'echo "pattern:$(grep -cw pattern words.txt)"'

Here, the string "pattern", given through the -I option, is a placeholder for xargs that it substitutes for each single line in its standard input.

The indirection with sh is needed because xargs only knows how to execute a single program, given it's name, passing everything else as arguments to it. xargs does not handle things like command substitution or piping.