UNIQ(1)                    POSIX Programmer's Manual                   UNIQ(1)



PPRROOLLOOGG
       This manual page is part of the POSIX Programmer's Manual.  The OS
       implementation of this interface may differ (consult the corresponding
       OS manual page for details of OS behavior), or the interface may not be
       implemented on OS.

NNAAMMEE
       uniq -- report or filter out repeated lines in a file

SSYYNNOOPPSSIISS
       uniq [[-c|-d|-u]] [[-f _f_i_e_l_d_s]] [[-s _c_h_a_r]] [[_i_n_p_u_t___f_i_l_e [[_o_u_t_p_u_t___f_i_l_e]]]]

DDEESSCCRRIIPPTTIIOONN
       The _u_n_i_q utility shall read an input file comparing adjacent lines, and
       write one copy of each input line on the output. The second and
       succeeding copies of repeated adjacent input lines shall not be
       written.  The trailing <newline> of each line in the input shall be
       ignored when doing comparisons.

       Repeated lines in the input shall not be detected if they are not
       adjacent.

OOPPTTIIOONNSS
       The _u_n_i_q utility shall conform to the Base Definitions volume of
       POSIX.1-2017, _S_e_c_t_i_o_n _1_2_._2, _U_t_i_l_i_t_y _S_y_n_t_a_x _G_u_i_d_e_l_i_n_e_s, except that ''++''
       may be recognized as an option delimiter as well as ''--''.

       The following options shall be supported:

       --cc        Precede each output line with a count of the number of times
                 the line occurred in the input.

       --dd        Suppress the writing of lines that are not repeated in the
                 input.

       --ff _f_i_e_l_d_s Ignore the first _f_i_e_l_d_s fields on each input line when doing
                 comparisons, where _f_i_e_l_d_s is a positive decimal integer. A
                 field is the maximal string matched by the basic regular
                 expression:


                     [[:blank:]]*[^[:blank:]]*

                 If the _f_i_e_l_d_s option-argument specifies more fields than
                 appear on an input line, a null string shall be used for
                 comparison.

       --ss _c_h_a_r_s  Ignore the first _c_h_a_r_s characters when doing comparisons,
                 where _c_h_a_r_s shall be a positive decimal integer. If specified
                 in conjunction with the --ff option, the first _c_h_a_r_s characters
                 after the first _f_i_e_l_d_s fields shall be ignored. If the _c_h_a_r_s
                 option-argument specifies more characters than remain on an
                 input line, a null string shall be used for comparison.

       --uu        Suppress the writing of lines that are repeated in the input.

OOPPEERRAANNDDSS
       The following operands shall be supported:

       _i_n_p_u_t___f_i_l_e
                 A pathname of the input file. If the _i_n_p_u_t___f_i_l_e operand is
                 not specified, or if the _i_n_p_u_t___f_i_l_e is ''--'', the standard
                 input shall be used.

       _o_u_t_p_u_t___f_i_l_e
                 A pathname of the output file. If the _o_u_t_p_u_t___f_i_l_e operand is
                 not specified, the standard output shall be used. The results
                 are unspecified if the file named by _o_u_t_p_u_t___f_i_l_e is the file
                 named by _i_n_p_u_t___f_i_l_e.

SSTTDDIINN
       The standard input shall be used only if no _i_n_p_u_t___f_i_l_e operand is
       specified or if _i_n_p_u_t___f_i_l_e is ''--''.  See the INPUT FILES section.

IINNPPUUTT FFIILLEESS
       The input file shall be a text file.

EENNVVIIRROONNMMEENNTT VVAARRIIAABBLLEESS
       The following environment variables shall affect the execution of _u_n_i_q:

       _L_A_N_G      Provide a default value for the internationalization
                 variables that are unset or null. (See the Base Definitions
                 volume of POSIX.1-2017, _S_e_c_t_i_o_n _8_._2, _I_n_t_e_r_n_a_t_i_o_n_a_l_i_z_a_t_i_o_n
                 _V_a_r_i_a_b_l_e_s for the precedence of internationalization
                 variables used to determine the values of locale categories.)

       _L_C___A_L_L    If set to a non-empty string value, override the values of
                 all the other internationalization variables.

       _L_C___C_T_Y_P_E  Determine the locale for the interpretation of sequences of
                 bytes of text data as characters (for example, single-byte as
                 opposed to multi-byte characters in arguments and input
                 files) and which characters constitute a <blank> in the
                 current locale.

       _L_C___M_E_S_S_A_G_E_S
                 Determine the locale that should be used to affect the format
                 and contents of diagnostic messages written to standard
                 error.

       _N_L_S_P_A_T_H   Determine the location of message catalogs for the processing
                 of _L_C___M_E_S_S_A_G_E_S.

AASSYYNNCCHHRROONNOOUUSS EEVVEENNTTSS
       Default.

SSTTDDOOUUTT
       The standard output shall be used if no _o_u_t_p_u_t___f_i_l_e operand is
       specified, and shall be used if the _o_u_t_p_u_t___f_i_l_e operand is ''--'' and the
       implementation treats the ''--'' as meaning standard output. Otherwise,
       the standard output shall not be used.  See the OUTPUT FILES section.

SSTTDDEERRRR
       The standard error shall be used only for diagnostic messages.

OOUUTTPPUUTT FFIILLEESS
       If the --cc option is specified, the output file shall be empty or each
       line shall be of the form:


           "%d %s", <_n_u_m_b_e_r _o_f _d_u_p_l_i_c_a_t_e_s>, <_l_i_n_e>

       otherwise, the output file shall be empty or each line shall be of the
       form:


           "%s", <_l_i_n_e>

EEXXTTEENNDDEEDD DDEESSCCRRIIPPTTIIOONN
       None.

EEXXIITT SSTTAATTUUSS
       The following exit values shall be returned:

        0    The utility executed successfully.

       >0    An error occurred.

CCOONNSSEEQQUUEENNCCEESS OOFF EERRRROORRSS
       Default.

       _T_h_e _f_o_l_l_o_w_i_n_g _s_e_c_t_i_o_n_s _a_r_e _i_n_f_o_r_m_a_t_i_v_e_.

AAPPPPLLIICCAATTIIOONN UUSSAAGGEE
       If the collating sequence of the current locale has a total ordering of
       all characters, the _s_o_r_t utility can be used to cause repeated lines to
       be adjacent in the input file. If the collating sequence does not have
       a total ordering of all characters, the _s_o_r_t utility should still do
       this but it might not. To ensure that all duplicate lines are
       eliminated, and have the output sorted according the collating sequence
       of the current locale, applications should use:


           LC_ALL=C sort -u | sort

       instead of:


           sort | uniq

       To remove duplicate lines based on whether they collate equally instead
       of whether they are identical, applications should use:


           sort -u

       instead of:


           sort | uniq

       When using _u_n_i_q to process pathnames, it is recommended that LC_ALL, or
       at least LC_CTYPE and LC_COLLATE, are set to POSIX or C in the
       environment, since pathnames can contain byte sequences that do not
       form valid characters in some locales, in which case the utility's
       behavior would be undefined. In the POSIX locale each byte is a valid
       single-byte character, and therefore this problem is avoided.

EEXXAAMMPPLLEESS
       The following input file data (but flushed left) was used for a test
       series on _u_n_i_q:


           #01 foo0 bar0 foo1 bar1
           #02 bar0 foo1 bar1 foo1
           #03 foo0 bar0 foo1 bar1
           #04
           #05 foo0 bar0 foo1 bar1
           #06 foo0 bar0 foo1 bar1
           #07 bar0 foo1 bar1 foo0

       What follows is a series of test invocations of the _u_n_i_q utility that
       use a mixture of _u_n_i_q options against the input file data. These tests
       verify the meaning of _a_d_j_a_c_e_n_t.  The _u_n_i_q utility views the input data
       as a sequence of strings delimited by ''\\nn''.  Accordingly, for the
       _f_i_e_l_d_sth member of the sequence, _u_n_i_q interprets unique or repeated
       adjacent lines strictly relative to the _f_i_e_l_d_s+1th member.

        1. This first example tests the line counting option, comparing each
           line of the input file data starting from the second field:


               uniq -c -f 1 uniq_0I.t
                   1 #01 foo0 bar0 foo1 bar1
                   1 #02 bar0 foo1 bar1 foo1
                   1 #03 foo0 bar0 foo1 bar1
                   1 #04
                   2 #05 foo0 bar0 foo1 bar1
                   1 #07 bar0 foo1 bar1 foo0

           The number ''22'', prefixing the fifth line of output, signifies that
           the _u_n_i_q utility detected a pair of repeated lines. Given the input
           data, this can only be true when _u_n_i_q is run using the --ff 11 option
           (which shall cause _u_n_i_q to ignore the first field on each input
           line).

        2. The second example tests the option to suppress unique lines,
           comparing each line of the input file data starting from the second
           field:


               uniq -d -f 1 uniq_0I.t
               #05 foo0 bar0 foo1 bar1

        3. This test suppresses repeated lines, comparing each line of the
           input file data starting from the second field:


               uniq -u -f 1 uniq_0I.t
               #01 foo0 bar0 foo1 bar1
               #02 bar0 foo1 bar1 foo1
               #03 foo0 bar0 foo1 bar1
               #04
               #07 bar0 foo1 bar1 foo0

        4. This suppresses unique lines, comparing each line of the input file
           data starting from the third character:


               uniq -d -s 2 uniq_0I.t

           In the last example, the _u_n_i_q utility found no input matching the
           above criteria.

RRAATTIIOONNAALLEE
       Some historical implementations have limited lines to be 1080 bytes in
       length, which does not meet the implied {LINE_MAX} limit.

       Earlier versions of this standard allowed the --_n_u_m_b_e_r and ++_n_u_m_b_e_r
       options. These options are no longer specified by POSIX.1-2008 but may
       be present in some implementations.

FFUUTTUURREE DDIIRREECCTTIIOONNSS
       None.

SSEEEE AALLSSOO
       _c_o_m_m, _s_o_r_t

       The Base Definitions volume of POSIX.1-2017, _C_h_a_p_t_e_r _8, _E_n_v_i_r_o_n_m_e_n_t
       _V_a_r_i_a_b_l_e_s, _S_e_c_t_i_o_n _1_2_._2, _U_t_i_l_i_t_y _S_y_n_t_a_x _G_u_i_d_e_l_i_n_e_s

CCOOPPYYRRIIGGHHTT
       Portions of this text are reprinted and reproduced in electronic form
       from IEEE Std 1003.1-2017, Standard for Information Technology --
       Portable Operating System Interface (POSIX), The Open Group Base
       Specifications Issue 7, 2018 Edition, Copyright (C) 2018 by the
       Institute of Electrical and Electronics Engineers, Inc and The Open
       Group.  In the event of any discrepancy between this version and the
       original IEEE and The Open Group Standard, the original IEEE and The
       Open Group Standard is the referee document. The original Standard can
       be obtained online at http://www.opengroup.org/unix/online.html .

       Any typographical or formatting errors that appear in this page are
       most likely to have been introduced during the conversion of the source
       files to man page format. To report such errors, see
       https://www.kernel.org/doc/man-pages/reporting_bugs.html .



IEEE/The Open Group                  2017                              UNIQ(1)
