\documentclass[nojss]{jss}

%\VignetteIndexEntry{evclust_vignette}
%\VignetteDepends{kernlab,MASS,mclust}

%% -- LaTeX packages and custom commands ---------------------------------------

%% recommended packages
\usepackage{thumbpdf,lmodern}

%% another package (only for this demo article)
\usepackage{framed}

\usepackage{amsmath,amssymb}

%% new custom commands
\newcommand{\class}[1]{`\code{#1}'}
\newcommand{\fct}[1]{\code{#1()}}

%% For Sweave-based articles about R packages:
%% need no \usepackage{Sweave}
\SweaveOpts{engine=R, eps=FALSE, keep.source = TRUE}
<<preliminaries, echo=FALSE, results=hide>>=
options(prompt = "R> ", continue = "+  ", width = 70, useFancyQuotes = FALSE)
library("MASS")
@
\setkeys{Gin}{width=0.6\textwidth}

\newcommand{\reels}{\mathbb{R}}

\newcommand{\bm}{\mathbf{m}}
\newcommand{\bJ}{\mathbf{J}}
\newcommand{\bv}{\mathbf{v}}
\newcommand{\bV}{\mathbf{V}}
\newcommand{\bg}{\mathbf{g}}
\newcommand{\bG}{\mathbf{G}}
\newcommand{\bgbar}{\overline{\mathbf{g}}}
\newcommand{\bx}{\mathbf{x}}
\newcommand{\bB}{\mathbf{B}}
\newcommand{\bH}{\mathbf{H}}
\newcommand{\bS}{\mathbf{S}}
\newcommand{\bSbar}{\overline{\mathbf{S}}}
\newcommand{\bD}{\mathbf{D}}

\newcommand{\calO}{{\cal O}}
\newcommand{\calM}{{\cal M}}
\newcommand{\calF}{{\cal F}}
\newcommand{\calL}{{\cal L}}
\newcommand{\calI}{{\cal I}}
\newcommand{\calT}{{\cal T}}
\newcommand{\calP}{{\cal P}}

\newcommand{\bi}{\begin{itemize}}
\newcommand{\ei}{\end{itemize}}
\newcommand{\be}{\begin{enumerate}}
\newcommand{\ee}{\end{enumerate}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}

%% -- Article metainformation (author, title, ...) -----------------------------

%% - \author{} with primary affiliation
%% - \Plainauthor{} without affiliations
%% - Separate authors by \And or \AND (in \author) or by comma (in \Plainauthor).
%% - \AND starts a new line, \And does not.
\author{Thierry Den{\oe}ux \\Universit\'e de technologie de Compi\`egne}
\Plainauthor{Thierry Den{\oe}ux}

%% - \title{} in title case
%% - \Plaintitle{} without LaTeX markup (if any)
%% - \Shorttitle{} with LaTeX markup (if any), used as running title
\title{\pkg{evclust}: An \proglang{R} Package for Evidential Clustering}
\Plaintitle{evclust: An R Package for Evidential Clustering}
\Shorttitle{\pkg{evclust}: An \proglang{R} Package for Evidential Clustering}

%% - \Abstract{} almost as usual
\Abstract{
One of the current trends in clustering is the development of algorithms able not only to discover groups in  data, but also to describe group-membership uncertainty. Evidential clustering is a recent approach in this direction based on the Dempster-Shafer theory of belief functions, a mathematical formalism for uncertainty representation. The output of an evidential clustering procedure is a credal partition, defined as a tuple of mass functions representing the uncertain assignment of objects to clusters. The  \proglang{R} package \pkg{evclust} described in this paper contains a collection of efficient evidential clustering algorithms, as well as functions to display, evaluate and exploit credal partitions. The application of this package is illustrated though the analysis of several datasets with different characteristics.
}

%% - \Keywords{} with LaTeX markup, at least one required
%% - \Plainkeywords{} without LaTeX markup (if necessary)
%% - Should be comma-separated and in sentence case.
\Keywords{clustering, belief functions, Dempster-Shafer theory, \proglang{R}}
\Plainkeywords{clustering, belief functions, Dempster-Shafer theory,  R}

%% - \Address{} of at least one author
%% - May contain multiple affiliations for each author
%%   (in extra lines, separated by \emph{and}\\).
%% - May contain multiple authors for the same affiliation
%%   (in the same first line, separated by comma).
\Address{
  Thierry Den{\oe}ux\\
  Universit\'e de technologie de Compi\`egne\\
  Compi\`egne, France
  \emph{and}\\
  Institut universitaire de France\\
  Paris, France\\
  E-mail: \email{tdenoeux@utc.fr}\\
  URL: \url{https://www.hds.utc.fr/~tdenoeux/}
}

\begin{document}
\SweaveOpts{concordance=FALSE}
%%\SweaveOpts{concordance=TRUE}


%% -- Introduction -------------------------------------------------------------

%% - In principle "as usual".
%% - But should typically have some discussion of both _software_ and _methods_.
%% - Use \proglang{}, \pkg{}, and \code{} markup throughout the manuscript.
%% - If such markup is in (sub)section titles, a plain text version has to be
%%   added as well.
%% - All software mentioned should be properly \cite-d.
%% - All abbreviations should be introduced.
%% - Unless the expansions of abbreviations are proper names (like "Journal
%%   of Statistical Software" above) they should be in sentence case (like
%%   "generalized linear models" below).

\section{Introduction} 
\label{sec:intro}

Since the introduction of the $k$-means clustering algorithm in the 1960's \citep{lloyd82}, cluster analysis has been a very active research area in computational statistics and machine learning (see, e.g. \cite{jain88,kaufman90,xu09}). There is now an abundance of software tools for data clustering: for instance, the  Comprehensive \proglang{R} Archive Network (CRAN) task view on ``Cluster Analysis \& Finite Mixture Models''\footnote{\url{https://cran.r-project.org/web/views/Cluster.html}} lists 107 \proglang{R} packages (as of February 23, 2021). Among them, we can mention the packages \pkg{RSKC} for robust and sparse $k$-means clustering \citep{JSSv072i05}, \pkg{pdfCluster} for nonparametric clustering based on density estimation \citep{JSSv057i11},  \pkg{NbClust} for determining the relevant number of clusters in a data set \citep{JSSv061i06}, etc.

Whereas \emph{hard clustering} algorithms such as the $k$-means procedure only generate a partition of the dataset, an important research direction has been to develop algorithms that compute a richer representation reflecting \emph{group-membership uncertainty}. For instance, fuzzy clustering algorithms such as the fuzzy $k$-means \citep{bezdek81} implemented in the \proglang{R} package \pkg{fclust} \citep{RJ-2019-017},  as well as model-based clustering methods based on the EM algorithm \citep{mclachlan88} implemented in \pkg{mclust} \citep{mclust16} compute  \emph{fuzzy partitions}. In a fuzzy partition, each object $i$ is assigned a degree of membership $u_{ik}$ in the range $[0,1]$ to each cluster $k$, in such as way that, for each object $i$;  the sum $\sum_{k} u_{ik}$ of  membership degrees (which can also be viewed as probabilities) is equal to  one. In \emph{possibilistic clustering} \citep{krishnapuram93,yang06}, this sum constraint is relaxed: each the computed quantity $u_{ik}$ can then be interpreted as the \emph{degree of possibility} that object $i$ belongs to cluster $k$. Degrees of possibility  reflect the \emph{typicality} of each observation, atypical patterns (or outliers) having a low degree of possibility of belonging to any of the clusters. Yet another approach is \emph{rough clustering} \citep{peters13,peters14}, in which each object is assigned a \emph{set} of possible clusters. For each cluster, we can then define a \emph{lower approximation}, composed of objects that \emph{certainly} belong to that cluster, and an \emph{upper approximation}, composed of objects that \emph{possibly} belong to it. In \proglang{R}, rough clustering algorithms are implemented in the package \pkg{SoftClustering} \citep{peters15}.

All the above approaches are subsumed by the \emph{evidential clustering} approach \citep{denoeux04b,masson08}, a relatively new approach based on the Dempster-Shafer (DS) theory of belief functions \citep{dempster67a,shafer76,denoeux20b}. Evidential clustering algorithms quantify clustering uncertainty using  \emph{mass functions} assigning masses to  \emph{sets} of clusters, called \emph{focal sets}, in such a way that the masses sum to one. The collection of mass functions  related to all objects in the dataset is called a \emph{credal} (or \emph{evidential}) \emph{partition}. A credal partition boils down to a fuzzy partition when the focal sets are singletons, and it can  be converted into any of the simpler classical representations such as a fuzzy, possibilistic or rough partition for display and  interpretation \citep{denoeux16b}. Evidential clustering algorithms include the Evidential $c$-Means (ECM) algorithm \citep{masson08}, a prototype-based procedure in the $k$-means family, and EVCLUS \citep{denoeux04b,denoeux16a}, an algorithm inspired by multidimensional scaling \citep{borg97} and applicable to non-metric proximity data. Until recently, there was no publicly available software tool for evidential clustering. The \proglang{R} package \pkg{evclust} described in this paper fills this gap by proposing an implementation of the main evidential clustering available so far, as well as functions for displaying, evaluating and exploiting credal partitions.

The rest of this paper is organized as follows. The needed notions related to DS theory and the main concepts underlying evidential clustering are first introduced in Section \ref{sec:evclus}. The main algorithms implemented in \pkg{evclust} are then described in Section \ref{sec:models}, and the application of these algorithms to various clustering problems is illustrated in Section \ref{sec:illustrations}. Finally, Section \ref{sec:summary} summarizes the paper.
  

\section{Evidential Clustering}
\label{sec:evclus}

Evidential clustering is based on the representation of cluster membership uncertainty using the theory of belief functions. We first recall elements of this theory in Section \ref{subsec:DS}. We then introduce the notion of credal partition and its representation within package \pkg{evclust} in Section \ref{subsec:credpart}.

\subsection{Theory of Belief Functions}
\label{subsec:DS}

We start with the consideration of some question $Q$, which has one and only answer among a finite set of possibilities $\Omega=\{\omega_1,\ldots,\omega_c\}$ (called the \emph{frame of discernment}). We assume that any evidence about $Q$ can be represented by a \emph{mass function} on $\Omega$, defined as a mapping from the power set $2^\Omega$ to the interval $[0,1]$, such that
$
\sum_{A\subseteq \Omega} m(A)=1.
$
Each subset $A$ of $\Omega$ such that $m(A)>0$ is called a \emph{focal set} of $m$. Each mass $m(A)$ represents  the degree to which the evidence supports $A$, without supporting any strict subset of $A$ \citep{shafer76}. A mass function $m$ is said to be \emph{logical} if it has only one focal set, \emph{consonant} of its focal sets are nested (i.e., for any two focal sets $A$ and $B$, we have either $A\subseteq B$ or $B\subseteq A$), and \emph{Bayesian} if its focal sets are singletons. A mass function that is both logical and Bayesian has only one singleton focal set: it is said to be \emph{certain}. If $\Omega$ is the only focal set, mass function $m$ is said to be \emph{vacuous} and it represents total ignorance.



\paragraph{Belief and plausibility functions.} Whereas  a probability mass function induces a probability measure, a DS mass function induces two dual nonadditive measures:  a \emph{belief function}, defined as
\begin{equation}
Bel(A)=\sum_{\emptyset \neq B \subseteq A} m(B)
\end{equation}
for all $A \subseteq \Omega$ and a \emph{plausibility function} defined as
\begin{equation}
Pl(A)=\sum_{B \cap A \neq \emptyset} m(B).
\end{equation}
These two functions are linked by the relation $Pl(A)=Bel(\Omega)-Bel(\overline{A})$, for all $A \subseteq \Omega$.  The quantity $Bel(A)$ is a measure of total support for  $A$ (taking into account the support given to its subsets), while $Bel(\Omega)-Pl(A)=Bel(\overline{A})$ is a measure of support for the complement $\overline{A}$ of $A$, so that $Pl(A)$ can be seen as a measure of lack of support for $\overline{A}$. %When $m$ is consonant, the following equality holds for all subsets $A$ and $B$ of $\Omega$:
% \[
% Pl(A\cap B)=\max(Pl(A),Pl(B)).
% \]
% Function $Pl$ is, thus, a possibility measure \citep{zadeh78}. 
The function $pl: \Omega \rightarrow [0,1]$ that maps each element $\omega$ of $\Omega$ to its plausibility $pl(\omega)=Pl(\{\omega\})$  is called the \emph{contour function} associated to $m$. When $m$ is consonant, the whole plausibility function can be recovered from $pl$ as $Pl(A)=\max_{\omega\in \Omega} pl(\omega)$. Function $pl$ is then called a \emph{possibility distribution} \citep{zadeh78}.

% \paragraph{Degree of conflict.} The \emph{degree of conflict} between two mass functions  $m_1$ and $m_2$ over  the same frame $\Omega$ \citep{shafer76} is defined as
% \begin{equation}
% \label{eq:conflict}
% \kappa(m_1,m_2) = \sum_{A \cap B=\emptyset} m_1(A)m_2(B) 
% \end{equation}
% It ranges in the interval [0,1]. When $m_1$ and $m_2$ represent two independent pieces of evidence pertaining to \emph{the same question}, $\kappa(m_1,m_2)$ can be interpreted as a \emph{measure of conflict} between these two pieces of evidence \citep{shafer76}. In contrast, when $m_1$ and $m_2$ represent independent pieces of evidence about \emph{two distinct questions} $Q_1$ and $Q_2$ with the same  frame of discernment $\Omega$,  $1-\kappa(m_1,m_2)$  can be given a different interpretation as  the plausibility that the true answers to $Q_1$ and $Q_2$ are identical, as shown by \cite{denoeux04b} (see Section \ref{subsec:credpart} below). 

\paragraph{Conjunctive sum and degree of conflict.} Let $m_1$ and $m_2$ be two mass functions defined on the same frame $\Omega$. Their \emph{conjunctive sum} \citep{smets94a} $m_1\cap m_2$ is the mass function defined as 
\begin{equation}
\label{eq:dempster}
(m_1\cap m_2)(A)=\sum_{B\cap C=A} m_1(B)m_2(C)
\end{equation}
for all subset $A\subseteq\Omega$. The quantity $(m_1\cap m_2)(\emptyset)$ is called   \emph{degree of conflict} \citep{shafer76} between  $m_1$ and $m_2$ and it is denoted as $\kappa(m_1,m_2)$.
% \begin{equation}
% \label{eq:conflict}
% \kappa(m_1,m_2) = \sum_{A \cap B=\emptyset} m_1(A)m_2(B) 
% \end{equation}
% It ranges in the interval [0,1]. 
When $m_1$ and $m_2$ represent two independent pieces of evidence pertaining to \emph{the same question}, $\kappa(m_1,m_2)$ can be interpreted as a \emph{measure of conflict} between these two pieces of evidence \citep{shafer76}. In contrast, when $m_1$ and $m_2$ represent independent pieces of evidence about \emph{two distinct questions} $Q_1$ and $Q_2$ with the same  frame of discernment $\Omega$,  $1-\kappa(m_1,m_2)$  can be given a different interpretation as  the plausibility that the true answers to $Q_1$ and $Q_2$ are identical, as shown by \cite{denoeux04b} (see Section \ref{subsec:credpart} below). 


\paragraph{Distance measures.} The degree of conflict measures the disagreement between two mass functions, but it does not measure their dissimilarity. In particular, the degree of conflict $\kappa(m,m)$ between a mass function and itself is usually strictly positive. Several distance measures for belief functions have been proposed \citep{jousselme12}. One of the most widely used is Jousselme's distance \citep{jousselme01}, which is defined as follows. Let $A_1,\ldots,A_N$ be the subsets of $\Omega$ arranged in some order, with $N=2^c$. A mass function $m$ can be represented by an $N$-vector $\bm=(m(A_1),\ldots,m(A_N))^T$. Let $\bJ$ be the positive definite and symmetric $N\times N$ matrix whose general term $[\bJ]_{ij}$ is the Jaccard index between sets $A_i$ and $A_j$:
 \[
 [\bJ]_{ij}=\begin{cases}
 1 & \text{if } A_i=A_j=\emptyset\\
 \frac{|A_i\cap A_j|}{|A_i \cup A_j|} & \text{otherwise.}
 \end{cases}
 \]
The Jousselme distance between two mass functions $m_1$ and $m_2$ on the same frame $\Omega$ is defined as
\begin{equation}
\label{eq:jousselme}
d_J(m_1,m_2)=\left[\frac{1}{2} (\bm_1-\bm_2)^T \bJ (\bm_1-\bm_2)\right]^{1/2}.
\end{equation}
It verifies $0\le d_J(m_1,m_2) \le 1$ for all $m_1$ and $m_2$. Another useful distance measure is the \emph{belief distance} \citep{denoeux01b}, defined as
\begin{equation}
\label{eq:belief_distance}
d_B(m_1,m_2)= \frac{1}{2} \sum_{A\subseteq \Omega} \mid Bel_1(A)-Bel_2(A)\mid,
\end{equation}
where $Bel_1$ and $Bel_2$ are, respectively, the belief functions corresponding to $m_1$ and $m_2$. We also have $0\le d_B(m_1,m_2) \le 1$ for all $m_1$ and $m_2$.

\paragraph{Summarization of a mass function.} It is often useful to summarize the information contained in a mass function for communication or display purposes. This can be done in a number of ways. Here, we mention only two approaches implemented in \pkg{evclust}. The first approach is to transform a mass function $m$ into a probability distribution. This can be done using the so-called \emph{plausibility transformation} method \citep{voorbraak89}, which consists in normalizing the contour function, resulting in the following probability distribution:
\begin{equation}
\label{eq:plaus_tranform}
p_m(\omega) = \frac{pl(\omega)}{\sum_{\omega'=1}^c pl(\omega')},
\end{equation}
for all $\omega\in\Omega$. Alternatively, the \emph{pignistic transformation} \citep{smets94a} distributes each normalized mass $m(A)/(1-m(\emptyset))$ uniformly to the elements of $A$:
\begin{equation}
\label{eq:pignistic}
betp_m(\omega) = \sum_{\{A\subset \Omega: \omega\in A\}} \frac{m(A)}{(1-m(\emptyset))|A|}.
\end{equation}
The second approach is to approximate a mass function $m$ as a set. A simple choice is to select the focal set $A^*(m)$ with the largest mass: 
\begin{equation}
\label{eq:mass_transform}
A^*(m)=\arg \max_{B\subseteq \Omega} m(B).
\end{equation}
As an alternative with a more decision-theoretic foundation \citep{denoeux18c}, we may consider the strict \emph{interval dominance} relation $\succ$ on $\Omega$ defined as follows: $\omega$ dominates $\omega'$ (denoted as $\omega \succ \omega'$) iff $Bel(\{\omega\})> Pl(\{\omega'\})$, i.e., the degree of belief in $\omega$ is larger than the degree of plausibility in $\omega'$ (meaning that $\omega$ is unambiguously more supported than $\omega'$). The set $A^\circ(m)$ of maximal (i.e., non-dominated) elements of this relation is then defined as 
\begin{equation}
\label{eq:interval_dominance}
A^\circ(m)=\{\omega\in \Omega : \forall \omega'\in \omega, Bel(\{\omega'\}) \le Pl(\{\omega\}\}.
\end{equation}

\paragraph{Nonspecificity.} Several measures have been proposed to quantify the degree of uncertainty of a mass function. A mass function basically represents two kinds of uncertainty: \emph{imprecision} and \emph{conflict} \citep{klir99}. For instance, the vacuous mass function is maximally imprecise but not conflicting, whereas the uniform Bayesian mass function (such that $m(\{\omega\})=1/c$ for all $\omega\in\Omega$) has maximal conflict, but no imprecision. Several uncertainty measures quantifying imprecision, conflict, or both have been proposed. As a measure of imprecision, \emph{nonspecificity} seems particularly well justified \citep{klir99}. It is defined as follows:
\begin{equation}
\label{eq:nonspec}
N(m)=\sum_{\emptyset \neq A\subseteq \Omega} m(A) \log_2 |A| + m(\emptyset) \log_2 c.
\end{equation}
Nonspecifity is maximal for the vacuous mass function verifying $m(\Omega)=1$, and also for the mass function $m$ such that $m(\emptyset)=1$. The interpretation of the mass assigned to the empty set in evidential clustering will be addressed in Section \ref{subsec:credpart}. \cite{masson08} proposed a method  to determine the number of clusters in an evidential partition based on the nonspecificity measure \eqref{eq:nonspec}, which has been implemented in \pkg{evclust} (see Section \ref{subsec:credpart} below).

\subsection{Credal Partition}
\label{subsec:credpart}

Let $\calO=\{o_1,\ldots,o_n\}$ be a set of $n$ objects. We assume that each object in $\calO$ belongs to \emph{at most} one cluster in a set $\Omega=\{\omega_1,\ldots,\omega_c\}$. Using the formalism recalled in Section \ref{subsec:DS}, evidence about the cluster membership of each object $o_i$ can be described by a mass function $m_i$ defined on the frame on $\Omega$. The $n$-tuple $\calM=(m_1,\ldots,m_n)$ is called an \emph{credal} (or \emph{evidential}) \emph{partition} of $\calO$.

\paragraph{Generality of credal partitions.} The notion of evidential partition encompasses most classical clustering structures \citep{denoeux16b}. In particular, when  all mass functions $m_i$ are certain,  $\calM$ becomes equivalent to a hard partition; this case corresponds to full certainty about the group of each object. When mass functions are Bayesian,  $\calM$ boils down to a fuzzy partition; the degree of membership $u_{ik}$ of object $i$ to group $k$ is  then 
$
u_{ik}=Bel_i(\{\omega_k\})=Pl_i(\{\omega_k\}) \in [0,1]
$
and we have $\sum_{k=1}^c u_{ik}=1$. When all mass functions $m_i$ are consonant, they are equivalently represented by their contour functions $pl_i$, and $pl_{ik}=pl_i(\omega_k)$ can be interpreted as the degree of possibility that object $o_i$ belongs to cluster $\omega_k$, as computed by possibilistic clustering algorithms. Finally, when each mass function $m_i$ is logical with focal set $A_i\subseteq \Omega$, $m_i$ is equivalent to a rough partition: the lower and upper  approximations of cluster $\omega_k$ are then defined, respectively, as follows:
\begin{equation}
\label{eq:lower_upper}
\omega^l_k:=\{i \in \calO \mid A_i=\{\omega_k\}\} \quad \textrm{and} \quad \omega^u_k:=\{i \in \calO\mid  \omega_k \in A_i\}.
\end{equation}
They are interpreted, respectively, as the set of objets \emph{surely} belong to cluster $\omega_k$, and the set of objects \emph{possibly} belong to  $\omega_k$. We then have $Bel_i(\{\omega_k\})=I(i \in \omega_k^l)$ and $Pl_i(\{\omega_k\})=I(i \in \omega_k^u)$, where $I(\cdot)$ denotes the indicator function. 

\paragraph{Summarization of a credal partition.} Being more general than classical clustering structures, a credal partition can be summarized into any of them. For instance, we obtain a fuzzy partition by transforming each mass function $m_i$ into a probability distribution using \eqref{eq:plaus_tranform} or \eqref{eq:pignistic}. A hard partition can then be obtained by selecting for each object the cluster with the highest probability. Alternatively, we may summarize the credal partition into a rough partition by approximating each mass function $m_i$ by a single set $A_i$ using \eqref{eq:mass_transform} or \eqref{eq:interval_dominance}; the lower and upper approximations of each cluster can then be computed using \eqref{eq:lower_upper}.

\paragraph{Representation in \pkg{evclust}.} Credal partitions are represented in \pkg{evclust} by \code{S3} objects of class \class{credpart}. Some of the most important attributes of a \class{credpart} object are the following:
\begin{description}
  \item[\code{method}:] The clustering algorithm used to generate the credal partition;
  \item[\code{F}:] The focal sets, encoded as a matrix of binary numbers with $c$ columns and $f$ rows, where $f$ is the number of focal sets; the first row must correspond to the empty set;
  \item[\code{mass}:] The mass functions, encoded as a matrix of size $n \times f$;
  \item[\code{pl}:] The contour functions, encoded as a matrix of size $n \times c$;
  \item[\code{p}:] The probability distributions computed by the plausibility transformation \eqref{eq:plaus_tranform}, encoded as a matrix of size $n \times c$;
  \item[\code{y.pl}:] The $n$-vector of maximum-plausibility class labels;
  \item[\code{Y}:] The maximum-mass sets $A^*(m_i)$, encoded as a matrix of 0's and 1's of size $n\times c$;
  \item[\code{upper.approx}:] The upper approximation of the credal partition, computed from the sets $A_i=A^*(m_i)$ using \eqref{eq:lower_upper}, provided as a list of length $c$. The $k$-th component of the list is the vector of indices of $\omega^u_k$.
  \item[\code{lower.approx}:] The lower approximation of the credal partition, computed from the sets $A_i=A^*(m_i)$ using \eqref{eq:lower_upper}, provided a list of length $c$. The $k$-th component of the list is the vector of indices of $\omega^l_k$.
  \item[\code{N}:] The average nonspecificity of the credal partition, normalized to range between 0 and 1.
\end{description}
A \class{credpart} object also contains algorithm-specific information such as the prototypes (for prototype-based algorithms such as ECM) and the value of the criterion for algorithms minimizing a cost function. \code{S3} methods \fct{summary} and \fct{plot} display basic information about objects of class \class{credpart}.

Consider, for instance, the \code{butterfly} data:
<<butterfly>>=
data(butterfly, package="evclust")
x<-butterfly
@
This is a toy dataset composed of 12 objects with two attributes (Figure \ref{fig:cred_butterfly}a). Objects 1 to 11 can be naturally partitioned in two clusters, and object 12 is an outlier.



Function \fct{ecm} implements the ECM algorithm (see Section \ref{subsec:ecm} below); it returns an object of class \class{credpart}:
<<butterfly_ecm>>=
library("evclust")
set.seed(210121)
clus<-ecm(x, c=2, delta=5, disp=FALSE)
summary(clus)
@

\setkeys{Gin}{width=\textwidth}
\begin{figure}[t!]
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_butterfly, echo=FALSE, fig=TRUE, height=4,width=5>>=
plot(x, type="n", xlab=expression(x[1]), ylab=expression(x[2]))
text(x[,1],x[,2],1:12,cex=0.8)
@
(a)
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_cred_butterfly, echo=FALSE, fig=TRUE, height=4,width=5>>=
plot(1:12, clus$mass[,1], type="l", ylim=c(0,1), xlab="objects", ylab="masses", lwd=2, col=2, cex.lab=1)
lines(1:12, clus$mass[,2], lty=2, lwd=2, col=3)
lines(1:12, clus$mass[,3], lty=3, lwd=2, col=4)
lines(1:12, clus$mass[,4], lty=4, lwd=2, col=5)
legend(0.9,0.68,c(expression(paste("m(",symbol("\306"),")")),expression(paste("m(",omega[1],")")),
                 expression(paste("m(",omega[2],")")),expression(paste("m(",Omega,")"))) ,
       lty=1:4,lwd=3,cex=1,col=2:5,bty="n")

@
(b)
\end{minipage}
\caption{\code{butterfly} dataset (a) and credal partition generated by the ECM algorithm (b). \label{fig:cred_butterfly}}
\end{figure}


The mass functions are encoded in a matrix \code{clus$mass} of 12 rows and 4 columns, corresponding to the four focal sets in \code{clus$F}. They are displayed in Figure \ref{fig:cred_butterfly}b. We can see that Object 6 has a large mass on $\Omega=\{\omega_1,\omega_2\}$ because it cannot be unambiguously assigned to any of the two classes, while Object 12 has a large mass on the empty set, which signals it as an outlier. %The lower and upper approximations of both clusters are contained, respectively, in lists \code{clus$lower.approx} and \code{clus$upper.approx}. 
%obtained as
%<<butterfly_approx>>=
%clus$lower.approx
%clus$upper.approx
%@
%Object 6 belongs to the upper approximations of both clusters, whereas Object 12 does not belong to the lower or upper approximation of any cluster.

 \paragraph{Relational representation.} A (hard) partition of a dataset can be represented in two ways: as a list of object subsets, or as the incidence matrix of the corresponding equivalence relation. Similarly, a credal partition admits a \emph{relational representation} \citep{denoeux17a}, defined as a tuple 
 $R=(m_{ij})_{1\le i< j \le n}$, in which $m_{ij}$ is a so-called \emph{pairwise mass function} on the frame $\Theta_{ij}=\{s_{ij},\neg s_{ij}\}$ , where $s_{ij}$ denotes the proposition ``Objects $i$ and $j$ belong to the same cluster'', and $\neg s_{ij}$ denotes the negation of $s_{ij}$. As proved by \cite{denoeux17a}, paiwise mass function $m_{ij}$ can be obtained from $m_i$ and $m_j$ as 
\begin{subequations}
\label{eq:mij}
\begin{align}
m_{ij}(\emptyset) & =m_i(\emptyset)+m_j(\emptyset)-m_i(\emptyset)m_j(\emptyset)\\
m_{ij}(\{s_{ij}\}) & =\sum_{k=1}^c m_i(\{\omega_k\}) m_j(\{\omega_k\}) \label{eq:mijsij}\\
m_{ij}(\{\neg s_{ij}\}) & =\sum_{A \cap B=\emptyset} m_i(A)m_j(B)-m_{ij}(\emptyset) \label{eq:mijnegsij}\\
m_{ij}(\Theta_{ij}) & =\sum_{A \cap B\neq\emptyset} m_i(A)m_j(B)-m_{ij}(\{s_{ij}\}).
\end{align}
\end{subequations}
The degree of  plausibility that objects $i$ and $j$ belong to the same cluster is, thus,
\begin{equation}
\label{eq:plsij}
pl_{ij}(s_{ij})=m_{ij}(\{s_{ij}\})+m_{ij}(\Theta_{ij})=\sum_{A \cap B\neq\emptyset} m_i(A)m_j(B)=1-\kappa_{ij},
\end{equation}
where $\kappa_{ij}$ is the degree of conflict between mass functions $m_i$ and $m_j$. 

 In \pkg{evclust}, function \fct{pairwise\_mass} takes as input a \class{credpart} object and returns its relational representation as a list of three \class{dist} objects \code{Me}, \code{M1} and \code{M0} corresponding, respectively, to $(m_{ij}(\emptyset))_{i<j}$, $(m_{ij}(\{s_{ij}\}))_{i<j}$ and $(m_{ij}(\{\neg s_{ij}\}))_{i<j}$.
 
The relational representation is useful, in particular, to compare two credal partitions. \cite{denoeux17a} propose two extensions of the Rand index, a classical measure to compare two hard partitions: a \emph{similarity index}, defined as
\begin{equation}
\label{eq:rhoS}
\rho_S(\calM,\calM')=1-\frac{2\sum_{i<j} d(m_{ij},m'_{ij})}{n(n-1)},
\end{equation}
where $\calM$ and $\calM'$ are two credal partitions and $d$ denotes either the Jousselme distance \eqref{eq:jousselme} or the belief distance \eqref{eq:belief_distance}, and a \emph{consistency index}, defined as
\begin{equation}
\label{eq:rhoC}
\rho_C(\calM,\calM')=1-\frac{2\sum_{i<j} \kappa(m_{ij},m'_{ij})}{n(n-1)},
\end{equation}
where $\kappa$ denotes the degree of conflict. In \pkg{evclust}, these two indices are computed by function \fct{credal\_RI}, which has three arguments: two relational representation objects \code{P1} and \code{P2}, and \code{type} with possible values `\code{c}' for `consistency', `\code{j}' for Jousselme's distance, and `\code{b}' for belief distance. 



%% -- Manuscript ---------------------------------------------------------------

%% - In principle "as usual" again.
%% - When using equations (e.g., {equation}, {eqnarray}, {align}, etc.
%%   avoid empty lines before and after the equation (which would signal a new
%%   paragraph.
%% - When describing longer chunks of code that are _not_ meant for execution
%%   (e.g., a function synopsis or list of arguments), the environment {Code}
%%   is recommended. Alternatively, a plain {verbatim} can also be used.
%%   (For executed code see the next section.)

\section{Evidential Clustering Algorithms} \label{sec:models}

Table \ref{tab:overview} presents an overview of the evidential clustering algorithms implemented in \pkg{evclust}. Some of these algorithms require attribute data, whereas others can handle proximity data, i.e., a matrix of dissimilarities, or distances between objects. Naturally, algorithms in the latter category can also be applied to attribute data after computing a distance matrix. Finally, some algorithms can also handle pairwise constraints, or can be trained in supervised learning mode (with class labels for some objects). These  algorithms are reviewed below.

\begin{table}[t!]
\centering
\begin{tabular}{llp{5cm}}
\hline
Function           & Name & Inputs    \\ \hline
\fct{ecm} & Evidential $c$-means  & attribute data \\
\fct{recm} & Relational evidential $c$-means  & proximity data \\
\fct{cecm} & Constrained evidential $c$-means  & attribute data + constraints  \\
\fct{bpec} & Belief peak evidential clustering & attribute data \\ 
\fct{kevclus} & EVCLUS & proximity data \\
\fct{kcevclus} & CEVCLUS & proximity data + constraints\\
\fct{nnevclus} & NN-EVCLUS & attribute data (+ constraints + labels)\\
\fct{bootclus} & Bootstrap evidential clustering  & attribute data\\
\fct{EkNNclus} & EK-NNclus & attribute data \\
\hline
\end{tabular}
\caption{Overview of clustering algorithms in \pkg{evclust}. \label{tab:overview} }
\end{table}


\subsection{Evidential $c$-Means and Variants}
\label{subsec:ecm}

Introduced by \cite{masson08}, the evidential $c$-means (ECM) algorithm is an alternating optimization procedure  in the same family as the hard and fuzzy $c$-means algorithms \citep{bezdek81}. As these algorithms, ECM represents each cluster $\omega_k\in \Omega$ by a prototype $\bg_k\in \reels^p$, where $p$ is the dimension of the data. However, a particularity of ECM is that it also represents each nonempty set of clusters (or \emph{meta-cluster}) $A_j\subseteq \Omega$ by a prototype $\bgbar_j$, defined as the barycenter of the prototypes $\bg_k$ for $\omega_k\in A_j$:
\[
\bgbar_j = \frac{1}{|A_j|}\sum_{\omega_k \in A_j} \bg_k.
\]
Having specified a set $\calF=\{A_1,\ldots,A_f\} \subset 2^\Omega$
 of $f$ nonempty subsets of $\Omega$, the cost function of ECM is defined as
\begin{equation}
\label{eq:JECM}
J_{\mbox{\tiny ECM}}(\calM,\bG) = \sum_{i=1}^n \sum_{j=1}^f |A_j|^\alpha m_{ij}^\beta d_{ij}^2 + \sum_{i=1}^n \delta^2  m_{i\emptyset}^\beta,
\end{equation}
where $\calM=(m_1,\ldots,m_n)$ is the credal partition, $\bG$ is the $c\times p$ matrix of prototypes, $d_{ij}=\|\bx_i-\bgbar_j\|$ is the Euclidean distance between attribute vector $\bx_i$ and prototype $\bgbar_j$,  $m_{ij}=m_i(A_j)$ and $m_{i\emptyset}=m_i(\emptyset)$. Parameters $\alpha$, $\beta$ and $\delta$ control, respectively, the nonspecificity of the credal partition (larger values of $\alpha$ favor the allocation of mass to smaller focal sets), the ``fuzziness'' of the partition (higher values of $\beta$ favor more uniform allocation of masses to focal sets), and the proportion of outliers (smaller values of $\delta$ result in more outliers). ECM minimizes cost function $J_{\mbox{\tiny ECM}}(\calM,\bG)$ by alternating two steps:
\be
\item Update of $\calM$ for fixed $\bG$, as
\begin{equation}
\label{eq:ecm}
m_{ij} = \frac{|A_j|^{-\alpha/(\beta-1)} d_{ij}^{-2/(\beta-1)}}{\sum_{k=1}^f |A_k|^{-\alpha/(\beta-1)} d_{ik}^{-2/(\beta-1)}+\delta^{-2/(\beta-1)}} , 
\end{equation}
for  $i=1,\ldots,n$ and $j=1,\ldots,f$, and $m_{i\emptyset} = 1 - \sum_{j=1}^f m_{ij}$ for $i=1,\ldots,n$.  
\item Update of $\bG$ for fixed $\calM$ by solving a linear system of the form $\bH \bG = \bB$, where  $\bB$ is a matrix of size $c \times p$ and $\bH$ a matrix of size $c \times c$.
\ee

In \pkg{evclust}, ECM is implemented in function \fct{ecm}:
\begin{Code}
ecm(x, c, g0 = NULL, type = "full", pairs = NULL, Omega = TRUE,
  ntrials = 1, alpha = 1, beta = 2, delta = 10, epsi = 0.001, disp = TRUE)
\end{Code}
where the arguments without a default value are the input data matrix \code{x} (of size $n \times p$, where $p$ is the number of attributes) and the number \code{c} of clusters.
Argument \code{g0} is an optional matrix of initial prototypes. Argument \code{type} specifies the focal sets: its values are \code{"full"} for all the subsets of $\Omega$, \code{"simple"} for the empty set, the singletons and $\Omega$, and \code{"pairs"} for the empty set, the singletons, $\Omega$ and either all pairs, or only the pairs specified by \code{pairs}. Setting \code{Omega = FALSE} removes $\Omega$ from the list of focal sets. The meaning of the other arguments is either obvious or can be found using \code{?ecm}. The output is a \class{credpart} object encoding the computed evidential partition.

\paragraph{RECM.} The relational evidential $c$-means (RECM) algorithm is a relational version of ECM, which takes as input a distance matrix instead of a matrix of attribute values \citep{masson09a}. The convergence of the algorithm is guaranteed if the distances are Euclidean, but the algorithm will often converge even if they are not. Function \fct{recm} has the same arguments as \fct{ecm}, except that the first argument is a distance matrix \code{D} instead of an object-attributes matrix \code{x}. Also, the algorithm can be initialized with a matrix \code{m0} of masses, instead of the matrix \code{g0} of initial prototypes.

\paragraph{CECM.} The constrained Evidential $c$-Means (CECM) algorithm \citep{antoine12} is another variant of ECM that takes as inputs not only  attribute data, but also pairwise constraints specifying that some pairs of object belong to the same cluster (\emph{must-link} constraint) or belong to different clusters (\emph{cannot-link} constraint). The CECM algorithm minimizes the following penalized cost function:
\begin{multline}
\label{eq:CECM}
 J_{\mbox{\tiny CECM}}(\calM,\bG)   =  (1-\xi) J_{\mbox{\tiny ECM}}(\calM,\bG) + \\
 \frac{\xi}{|\textsf{ML}|+|\textsf{CL}|} \left[\displaystyle\sum_{(\bx_i,\bx_j) \in \textsf{ML}} pl_{ij}(\neg s_{ij})+ \displaystyle\sum_{(\bx_i,\bx_j) \in \textsf{CL}} pl_{ij}(s_{ij})\right], 
\end{multline}
where $\textsf{ML}$ and $\textsf{CL}$ are, respectively, the sets of must-link and cannot-link constraints, $pl_{ij}$ is the contour function corresponding to pairwise mass function $m_{ij}$ defined by \eqref{eq:mij}, and $\xi$ is a penalization coefficient. As ECM, CECM uses an alternating optimization scheme, minimizing the cost function with respect to $\calM$ and $\bG$ in turn. Here, the former optimization problem does not admit a closed-form solution and needs to be solved by a quadratic programming procedure. 

A version of CECM uses an adaptive metric to account for elliposidal clusters, in a similar way as the Gustafson-Kessel fuzzy clustering algorithm \citep{gustafson79}. A symmetric and positive definite matrix $\bS_k$ is defined for each cluster $\omega_k$, and the  matrix $\bSbar_j$ of each meta-cluster $A_j\subseteq \Omega$ is defined as the average of matrices $\bS_k$ for $\omega_k \in A_j$. The squared distance $d_{ij}$ between $\bx_i$ and the center $\bgbar_j$ of meta-cluster $A_j$ is then defined as $d_{ij}^2=(\bx-\bgbar_j)^T \bSbar_j(\bx-\bgbar_j)$. The cost function \eqref{eq:CECM} is then minimized with respect to $\calM$, $\bV$ and the matrices $\bS_k$. 

The CECM algorithm is implemented in function \fct{cecm}, which has the same arguments as \fct{ecm}, and four additional arguments: \code{ML}, \code{CL}, \code{xi} (with obvious meanings), and \code{distance} with possible values 0 for  Euclidean distance and 1 for adaptive metric.

\paragraph{BPEC.} The belief peak evidential clustering (BPEC) method \citep{su19} is similar to ECM, except that the cluster centers are selected as the ``belief peaks'' in a two dimensional ``$\delta$-$Bel$'' graph, defined  as points of high density (estimated by a $K$ nearest neighbor approach) and located far from  points with a higher density. The ECM algorithm is then run, keeping the cluster centers constant. In \pkg{evclust}, function \fct{delta\_Bel} takes as input the data matrix \code{x}, the number \code{K} or neighbors and a scale parameter \code{q}. It draws the $\delta$-$Bel$ graph, and the user is invited to manually select the lower right corner of a rectangle containing the belief peaks, which are then returned as output. Function \fct{bpec}, whose syntax is similar to that of \fct{ecm} can then be run with the belief peaks as fixed cluster centers.

\subsection{EVCLUS and Variants}
\label{subsec:evclus}

The EVCLUS algorithm, introduced by \cite{denoeux04b} and improved by \cite{denoeux16a}, is another evidential clustering algorithm that takes inspiration from multidimensional scaling \citep{borg97}. It takes as input  a symmetric  $n\times n$ dissimilarity matrix $\bD=(\delta_{ij})$, where $\delta_{ij}$ denotes the dissimilarity between objects $o_i$ and $o_j$. Dissimilarities may be  computed from attribute data, or they may be directly available. They need not satisfy the  axioms of a distance such as the triangular inequality. 

The fundamental assumption underlying EVCLUS is that the more similar are two objects, the more plausible it is that they belong to the same cluster. From \eqref{eq:plsij}, the plausibility $pl_{ij}(s_{ij})$ that two objects $o_i$ and $o_j$ belong to the same cluster is equal to $1-\kappa_{ij}$, where $\kappa_{ij}$ is the degree of conflict between $m_i$ and $m_j$. The credal partition $\calM$ should thus be determined in such a way that similar objects have mass functions $m_i$ and $m_j$ with low degree of conflict, whereas highly dissimilar objects are assigned highly conflicting mass functions. This can be achieved by minimizing the following loss function:
\begin{equation}
\label{eq:lossEVCLUS}
\calL(\calM)=\frac{2}{n(n-1)} \sum_{i<j} \left(\kappa_{ij}-\varphi(\delta_{ij})\right)^2,
\end{equation}
where $\varphi$ is a fixed nondecreasing mapping from $[0,+\infty)$ to $[0,1]$, such as
$\varphi(\delta)=1-\exp(-\eta \delta^2/d_0^2)$. Setting $\eta$ to $-\log \alpha$, $d_0$ is the threshold such that two objects $o_i$ and $o_j$ with dissimilarity larger than  $d_0$ have a plausibility at least $\alpha$ of belonging to different clusters. Typically, $\alpha=0.05$ and $d_0$ is set to some quantile of the dissimilarities $\delta_{ij}$.

Computing the loss function \eqref{eq:lossEVCLUS} requires to store the whole dissimilarity matrix, which may not be feasible for large datasets.  \cite{denoeux16a} showed that it is  sufficient to minimize the sum of squared errors for a subset of $k$ object pairs. This is achieved by changing the loss function  \eqref{eq:lossEVCLUS} to
\begin{equation}
\label{eq:lossEVCLUS1}
\calL(\calM;J)=\frac{1}{nk} \sum_{i=1}^n\sum_{j\in J(i)} \left(\kappa_{ij}-\varphi(\delta_{ij})\right)^2,
\end{equation}
where $J(i)$ is a randomly selected subset of $\{1,\ldots,n\}\setminus \{i\}$ with cardinality $k\le n-1$. %Experiments reported in \cite{denoeux16a} show that, for a number $n$ of objects between 1000 and 10,000, optimal results are obtained with $p$ in the range 100-500.

In \pkg{evclust}, EVCLUS is implemented in function \fct{kevclus}:
\begin{Code}
kevclus(x, k=n-1, D, J, c, type='simple', pairs=NULL, m0=NULL, ntrials=1, 
  disp=TRUE, maxit=1000, epsi=1e-5, d0=quantile(D,0.9), tr=FALSE, 
  change.order=FALSE, norm=1)
\end{Code}
where most arguments have  obvious meanings. If an object-attribute data matrix \code{x} of size $n \times p$ is supplied, then the dissimilarities are computed as the Euclidean distances between the rows of \code{x}. Otherwise, a dissimilarity matrix \code{D} must be supplied. If \code{D} is not square and has size $n \times k$, then a matrix \code{J} of the same size containing the indices to be used for computing the loss function \eqref{eq:lossEVCLUS1} must be provided.  Arguments \code{type} and \code{pairs} have the same meaning as in \fct{ecm}. The meaning of the other arguments  can be found using \code{?kevclus}. The output is a \class{credpart} object encoding the computed evidential partition. It contains, in addition to the usual attributes of a \class{credpart} object, a matrix \code{Kmat} of degrees of conflict (of size $n \times k$) and a matrix \code{D} of the same size containing the transformed dissimilarities $\varphi(\delta_{ij})$.

\paragraph{CEVCLUS.} CEVCLUS is a constrained version of EVCLUS that uses must-link and cannot-link constraints \citep{antoine14,li18}. It minimizes the following loss function
\begin{multline}
\label{stressCk}
  \calL_{C}(\calM;J)= \calL(\calM;J) + \frac{\xi}{2(|\mathsf{ML}|+|\mathsf{CL}|)}\left(\sum_{(i,j)\in\mathsf{ML}}pl_{ij}(\neg s_{ij})+1-pl_{ij}(s_{ij})]\right. +\\
  \left. \sum_{(i,j)\in\mathsf{CL}} [pl_{ij}(s_{ij})+1-pl_{ij}(\neg s_{ij})]\right)
\end{multline}
where, as before, $\xi$ is a penalization coefficient, and $\mathsf{ML}$ and $\mathsf{CL}$ denote, respectively, the sets of must-link and cannot-link constraints. In \pkg{evclust}, the CEVCLUS algorithm is implemented in function \fct{kcevclus}, which has the same arguments as \fct{kevclus}, and three additional arguments: \code{ML}, \code{CL}, and \code{xi}.

\paragraph{NN-EVCLUS.} As opposed to ECM reviewed in Section \ref{subsec:ecm},  the EVCLUS algorithm does not build a compact representation of clusters as a collection of prototypes, but it learns an evidential partition of the $n$ objects directly. If each mass function is constrained to have $f$ focal sets, the number of free parameters is, thus, $n(f-1)$, i.e., it grows linearly with the number of objects. This characteristic makes EVCLUS impractical for clustering very large datasets. Also, the algorithm learns an evidential partition of a given dataset, but it does not allow us to extrapolate beyond the learning set and make predictions for new objects. The NN-EVCLUS algorithm addresses these issues by learning a mapping from attribute vectors to mass functions using a multilayer feedforward neural network \citep{denoeux2020nnevclus}. The network is trained in an unsupervised way by minimizing a loss function similar to \eqref{eq:lossEVCLUS} or \eqref{eq:lossEVCLUS1} with respect to the network weights. To enhance its robustness to outliers, the learning algorithm can also use the output of a one-class support vector machine such as implemented in the \proglang{R} package \pkg{kernlab} \citep{karatzoglou04}; the mass functions computed by the neural network are then transformed so that outliers receive a large mass on the empty set. The network can also be trained in semi-supervised mode using either must-link and cannot-link constraints (in which case a loss function similar to \eqref{stressCk} is minimized), or using class labels for some learning instances. In the latter case, we assume that we have $n_s$ labeled attribute vectors $\{(\bx_i,y_i), i\in \calI_s\}$, where $\calI_s\subseteq \{1,\ldots,n\}$ and $y_i\in\Omega$ is the class label of object $i$, and the following term is added to the loss function:
\begin{equation}
\label{eq:reg_ss}
\frac{\nu}{n_s}\sum_{i\in \calI_s} \sum_{l=1}^c (pl_{il}-y_{il})^2.
\end{equation}
where  $y_{il}=I(y_i=\omega_l)$, $pl_{il}=pl_i(\omega_l)$,  $pl_i$ is the contour function corresponding to $m_i$, and $\nu$ is a penalization coefficient.

The function \fct{nnevclus} implementing the NN-EVCLUS algorithm has the following syntax:
\begin{Code}
nnevclus(x, k=n-1, D=NULL, J=NULL, c, type='simple', n_H, ntrials=1, 
  d0=quantile(D,0.9), fhat=NULL, lambda=0,y=NULL, Is=NULL, nu=0, 
  ML=NULL, CL=NULL, xi=0, tr=FALSE, options=c(1, 1000, 1e-4, 10),
  param0=list(U0=NULL, V0=NULL, W0=NULL, beta0=NULL))
\end{Code}
Function \fct{nnevclus} shares most arguments with \fct{kevclus} and \fct{kcevclus}. Specific arguments include: \code{n_H}, which specifies the architecture of the neural network: it is either a scalar equal to the number of units in a single hidden layer, or a two-dimensional vector containing the sizes of two hidden layers;  \code{fhat} is an optional vector of one-class support vector machine (SVM) outputs; \code{lambda} is a weight decay ($L_2$ regularization) coefficient; \code{y} is a vector of class labels for some instances, \code{Is} is a vector of corresponding indices, and \code{nu} is the penalization coefficient of the supervised part of the loss function; \code{param0} is a list of initial network parameters. More details can be found using the instruction \code{?nnevclus}.

Function \fct{nnevclus} uses a batch learning algorithm, which compute the gradient of the average loss at each iteration. For large datasets, a stochastic gradient descent (SGD) algorithm is more suitable. 
Function \fct{nnevclus\_mb} implements RMSprop, an accelerated SGD algorithm \citep{goodfellow16}. 
Function \fct{nnevclus\_mb} is called as follows:
\begin{Code}
nnevclus_mb(x, foncD=function(x) as.matrix(dist(x)), c, type='simple', n_H, 
  nbatch=10, alpha0=0.9, fhat=NULL, lambda=0, y=NULL, Is=NULL, nu=0, 
  disp=TRUE, options=list(Niter=1000, epsi=0.001, rho=0.9, delta=1e-8, 
  Dtmax=100, print=5), param0=list(V0=NULL, W0=NULL, beta0=NULL))
\end{Code}
where \code{foncD} is a function that computes distances, \code{nbatch} is the number of mini-batches used by the SGD algorithm, \code{alpha0} is the order of the quantile used to compute parameter $d_0$, and \code{epsi}, \code{rho} and \code{delta} are the usual parameters of RMSprop. Currently, the network architecture is limited to one hidden layer, and semi-supervised learning with pairwise constraints has not been implemented.

\subsection{Bootclus}
\label{subsec:bootclus}

The Bootclus algorithm uses the bootstrap and a model-based clustering approach to construct a credal partition that reflects second-order cluster-membership uncertainty \citep{denoeux20d}. The method can be summarized as follows:
\be
\item A finite mixture model, typically a Gaussian mixture model (GMM) is postulated;
\item $B$ bootstrap samples are generated;
\item Parameter estimates are computed from each bootstrap sample using the expectation-maximization (EM) algorithm; 
\item Let  $P_{ij}$ denote the probability that objects $i$ and $j$ belong to the same class. A bootstrap percentile confidence interval $[P_{ij}^l,P_{ij}^u]$ at level $1-\alpha$ on $P_{ij}$  is computed for each object pair $(i,j)$;
\item A normalized credal partition $\calM=(m_1,\ldots,m_n)$ (such that $m_i(\emptyset)=0$ for all $i$) is obtained as the solution of the following minimization problem:
\[
\min_\calM J(\calM)=\sum_{i<j} \left(m_{ij}(\{s_{ij}\})-P_{ij}^l\right)^2 + \left(m_{ij}(\{\neg s_{ij}\})-(1-P_{ij}^u)\right)^2,
\]
where $m_{ij}(\{s_{ij}\})$ and $m_{ij}(\{\neg s_{ij}\})$ are computed using, respectively, Eqs. \eqref{eq:mijsij} and \eqref{eq:mijnegsij}. 
\ee
The resulting credal partition is such that, for any pair $(i,j)$ of objects, $Bel_{ij}(\{s_{ij}\})\approx P_{ij}^l$ and $Pl_{ij}(\{s_{ij}\})\approx P_{ij}^u$, where $Bel_{ij}$ and $Pl_{ij}$ are, respectively, the belief and plausibility functions corresponding to $m_{ij}$. As a result, the intervals $[Bel_{ij}(\{s_{ij}\}),Pl_{ij}(\{s_{ij}\})]$ are approximate $1-\alpha$ confidence intervals on the pairwise probabilities: the credal partition is  said to be \emph{calibrated}.

In \pkg{evclus}, the Bootclus algorithm is implemented in function \fct{bootclus}:
\begin{Code}
bootclus(x, conf=0.90, B=500, param=list(G=NULL), type="pairs", Omega=FALSE)
\end{Code}
where \code{x} is the object-attribute matrix of size $n\times p$, \code{conf} is the confidence degree $1-\alpha$ and \code{B} is the number of bootstrap samples. Function \fct{bootclus} calls functions \fct{Mclust} and \fct{MclustBootstrap} of package \pkg{mclust} for model-based clustering with GMMs \citep{mclust16}. The argument \code{param} contains a list of arguments passed to  \fct{Mclust}. Arguments \code{type} and \code{Omega} have the same meanings as the corresponding arguments of function \fct{ecm} (see Section \ref{subsec:ecm}).

The output of function \fct{bootclus} is a list containing the credal partition (a \class{credpart} object), the mixture model estimation results provided as an \class{Mclust} object by function \fct{Mclust}, as well as the confidence intervals $[P_{ij}^l,P_{ij}^u]$ and the approximating pairwise belief and plausibility degrees.

\subsection{EK-NNclus}

Many clustering algorithms are based on the so-called ``decision-directed'' approach, in which  an initial classifier  is used to label the learning instances; the classifier is then updated, and the process is repeated until no changes occur in the labels. For instance, the  $c$-means algorithm is based on this principle: in that case, the nearest-prototype classifier is used to label the samples, and it is updated by recomputing the prototypes as the cluster centers. 

EK-NNclus \citep{denoeux15} is a decision-directed clustering algorithm in which the base classifier is the evidential $K$ nearest neighbors (E-KNN) rule \citep{denoeux95a}. Given a labeled training set $\calT=\{(x_1,y_1),\ldots,(x_n,y_n)\}$, where $y_i\in\Omega$ is the class label of instance $i$, the E-KNN rule classifies a new instance $x$ using the subset $N_K(x)\subset \calT$ of its $K$ nearest neighbors in $\calT$. For each neighbor $x_i \in N_K(x)$ a mass function $m_i$ on $\Omega$ is computed as
\begin{align*}
m_i(\{\omega_k\})&= \varphi(\|x-x_i\|) I(y_i=\omega_k), \quad k=1,\ldots,c\\
m_i(\Omega)&= 1-\varphi(\|x-x_i\|),
\end{align*}
where $I(\cdot)$ is the indicator function and $\varphi(\cdot)$ is a decreasing mapping from $[0,+\infty)$ to $[0,1]$. Typically, $\varphi(d)=\exp(-\gamma d^b)$, where $\gamma$ and $b$ are positive parameters. The $K$ mass functions $m_i$ corresponding to the $K$ nearest neighbors are then combined using the conjunctive sum operation \eqref{eq:dempster} and renormalized, an operation  called \emph{Dempster's rule} \citep{shafer76}.
% \begin{equation}
% \label{eq:eknnclus}
% m=\bigoplus_{\{i: x_i\in N_K(x)\}} m_i,
% \end{equation}
% and the class with the highest plausibility is selected as the label of $x$.

The EK-NNclus algorithm uses the above EK-NN rule for clustering. The algorithm is initialized by assigning  each object a random label. If the dataset is not too large, we can initially assume that there are as many clusters as objects and each cluster contains exactly one object. Objects are then considered in random order and classified from their $K$ nearest neighbors using the EK-NN rule. The algorithm stops when the class labels have not changed during the last iteration through the whole training set. By noticing the similarity with Hopfield neural networks \citep{hopfield82}, \cite{denoeux15} showed that this algorithm converges in a finite number of iterations. After convergence, we can compute a combined mass function for each object, which gives us a normalized credal partition. As  mass functions in this credal partition are obtained by combining $K$ mass functions using Dempster's rule, they are often very specific, with most of the mass concentrated on singletons. However, outlier are characterized by mass functions with a large mass on  $\Omega$. We also note that this method does not require the user to specify the number of clusters: starting with an arbitrarily large number of clusters, the algorithm usually converges to a meaningful partition.

In \pkg{evclust}, the EK-NNclus algorithm is implemented in function \fct{EkNNclus}
\begin{Code}
EkNNclus(x, D, K, y0, ntrials=1, q=0.5, b=1, disp=TRUE, tr=FALSE)
\end{Code}
where \code{x} is the $n\times p$ object-attributes data matrix, \code{D} is distance matrix (required only if \code{x} is not provided), \code{K} is the number of neighbors, \code{y0} is the $n$-vector of initial labels, \code{q} is a number in $(0,1)$ such that parameter $\gamma$ is set to the inverse of the $q$-quantile of distances between any attribute vector and its $K$ nearest neighbors.  Arguments \code{disp} and \code{tr} control, respectively, the display and storage of intermediate results. The output is a \class{credpart} object encoding the final credal partition.

%% -- Illustrations ------------------------------------------------------------

%% - Virtually all JSS manuscripts list source code along with the generated
%%   output. The style files provide dedicated environments for this.
%% - In R, the environments {Sinput} and {Soutput} - as produced by Sweave() or
%%   or knitr using the render_sweave() hook - are used (without the need to
%%   load Sweave.sty).
%% - Equivalently, {CodeInput} and {CodeOutput} can be used.
%% - The code input should use "the usual" command prompt in the respective
%%   software system.
%% - For R code, the prompt "R> " should be used with "+  " as the
%%   continuation prompt.
%% - Comments within the code chunks should be avoided - these should be made
%%   within the regular LaTeX text.

\section{Illustrations} \label{sec:illustrations}

In this section, we demonstrate the use of the main functions in the \pkg{evclus} package through the analysis of some datasets.

\subsection[fourclass dataset]{\code{fourclass} dataset}

The \code{fourclass} dataset is a synthetic dataset with two attributes and four classes of 100 points each, generated from a multivariate t distribution with five degrees of freedom:
<<fourclass>>=
data(fourclass, package="evclust")
x<-fourclass[,1:2]
y<-fourclass[,3]
@

\paragraph{EVCLUS.} Calling function \code{kevclus} with the argument \code{c=4}  creates an object \code{clus.evclus} of class \class{credpart}:
<<fourclass_evclus>>=
clus.evclus<-kevclus(x, c=4, disp=FALSE)
@
Figure \ref{fig:fourclass_evclus} contains the graphs generated by  \code{S3} method \fct{plot} applied to the credal partition \code{clus.evclus}:
\begin{Code}
plot(clus.evclus, x, plot_Shepard=TRUE)
\end{Code}
Figure \ref{fig:fourclass_evclus}a is a scatterplot of the data together with a view of the credal partition. The solid and broken lines are the convex hulls of, respectively, the lower and upper approximations of the four clusters. The symbol and color of each point indicate the maximum-plausibility cluster, and the size of the symbols is proportional to the plausibility of that cluster. Outliers are indicated by circles. Figure \ref{fig:fourclass_evclus}b is a ``Shepard diagram'' showing the degrees of conflict $\kappa_{ij}$ (vertical axis) vs. the transformed dissimilarities $\varphi(\delta_{ij})$ (horizontal axis). This diagram reflects, in some way, the quality of the credal partition.  

\begin{figure}[t!]
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_evclus, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.evclus, x)
@
(a)
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_evclus_Shepard, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(x=clus.evclus,plot_Shepard=TRUE)
@
(b)
\end{minipage}
\caption{Plot of the \code{fourclass} dataset with the credal partition generated by \fct{kevclus} (a) and plot of the degrees of conflict $\kappa_{ij}$ vs. the transformed dissimilarities $\varphi(\delta_{ij})$  (b). \label{fig:fourclass_evclus}}
\end{figure}

\paragraph{ECM} The ECM algorithm can be applied to the same data by calling function \fct{ecm} as:
<<fourclass_ecm>>=
clus.ecm<-ecm(x, c=4, type = "pairs", delta=3.5, disp=FALSE, Omega=FALSE)
@
We note that, in this call to function \fct{ecm}, we have restricted the focal sets to the empty set,  singletons, and  pairs  (thus excluding triplets and $\Omega$). The \code{delta} argument determines the proportion of outliers. The resulting credal partition is shown in Figure \ref{fig:fourclass_ecm}a. We notice that the obtained credal partition is more imprecise than that generated by EVCLUS, in the sense that more points are located in the boundary areas of the four clusters (defined as the set difference between the upper and lower approximations). However, we can obtain a more precise credal partition by increasing the value of parameter $\alpha$ in cost function \eqref{eq:JECM} from the default value $\alpha=1$ to $\alpha=3$, as
<<fourclass_ecm1>>=
clus.ecm1<-ecm(x, c=4, type = "pairs", delta=3.5, a=3, disp=FALSE, 
  Omega=FALSE)
@
By comparing Figures \ref{fig:fourclass_ecm}b and \ref{fig:fourclass_evclus}a, we can see that the credal partition generated by ECM is then very similar to that computed by EVCLUS.

\begin{figure}[t!]
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_ecm, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.ecm, x)
@
(a)
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_ecm1, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.ecm1, x)
@
(b)
\end{minipage}
\caption{Plots of the \code{fourclass} dataset with the credal partitions generated by \fct{ecm} with $\alpha=1$ (a) and  $\alpha=3$ (b). The prototypes are displayed as filled circles. \label{fig:fourclass_ecm}}
\end{figure}



\paragraph{NN-EVCLUS.} Let us now consider the application of NN-EVCLUS to the same data. We may first train a one-class SVM using function \fct{ksvm} of package \pkg{kernlab}:
<<fourclass_svm>>=
library(kernlab)
svmfit<-ksvm(~., data=data.frame(x), type="one-svc", kernel="rbfdot",
  nu=0.2, kpar=list(sigma=0.2))
fhat<-predict(svmfit, newdata=x, type="decision")
@
Vector \code{fhat} containing the one-class SVM output is then provided as input to function \fct{nnevclus} to train a neural network with one hidden layer of 20 units:
<<fourclass_nn>>=
clus.nn<-nnevclus(x, k=100, c=4, n_H=20, type='pairs', fhat=fhat,
  options=c(0,1000,1e-4,10), tr=TRUE)
@
By setting the argument \code{k} to 100, we only use the distances from each input vector to 100 randomly selected input vectors, which speeds up the calculations. The obtained credal partition displayed in Figure \ref{fig:fourclass_nn}a is similar to those generated by EVCLUS and ECM. Figure \ref{fig:fourclass_nn}b shows the decrease of the loss   \eqref{eq:lossEVCLUS1} as a function of the number of iterations.

\begin{figure}[t!]
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_nn, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.nn, x)
@
(a)
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_nn_loss, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.nn$trace$fct[clus.nn$trace$fct>0],type="l",xlab="iteration",ylab="loss")
@
(b)
\end{minipage}
\caption{Plots of the \code{fourclass} dataset with the credal partitions generated by \fct{nnevclus} (a) and evolution of the loss as function of the number of iterations (b).  \label{fig:fourclass_nn}}
\end{figure}


An advantage of NN-EVCLUS is the possibility to predict the cluster-membership of new data. The \code{S3} method \fct{predict} for \class{credpart} objects takes as input a credal partition of class \class{credpart} generated by NN-EVCLUS or ECM as well as a new data matrix, and outputs a credal partition for the new data.  The following code computes an array \code{PL} containing the plausibilities of the four clusters at the nodes of a $50\times 50$ grid in the two-dimensional attribute space:
<<contour_NN-evclus>>=
nx <- 50
xmin <- apply(x,2,min) - 2
xmax <- apply(x,2,max) + 2
xx <- seq(xmin[1], xmax[1], (xmax[1] - xmin[1]) / (nx-1))
yy <- seq(xmin[2], xmax[2], (xmax[2] - xmin[2]) / (nx-1))
PL <- array(0, c(nx, nx, 4))
for(i in 1:nx){
  X1 <- matrix(c(rep(xx[i],nx),yy),nx,2)
  x1 <- data.frame(X1)
  names(x1) <- c("x1","x2")
  fhat <- predict(svmfit, newdata=x1, type="decision")
  clus.t <- predict(clus.nn, X1, fhat)
  PL[i,,] <- clus.t$pl
}
@
The corresponding contour plots are shown in  Figure \ref{fig:fourclass_nn_pl}.

\begin{figure}[t!]
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_nn_pl1, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(x[,1],x[,2],col=y,pch=y,xlim=c(xmin[1],xmax[1]),ylim=c(xmin[2],xmax[2]),
     main=expression(paste("pl(",omega[1],")")),
     cex.main=1.5,xlab="",ylab="")
contour(xx,yy,PL[,,1],col=1,lty=1,lwd=1.5,add=TRUE,labcex=1.1)
@
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_nn_pl2, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(x[,1],x[,2],col=y,pch=y,xlim=c(xmin[1],xmax[1]),ylim=c(xmin[2],xmax[2]),
     main=expression(paste("pl(",omega[2],")")),
     cex.main=1.5,xlab="",ylab="")
contour(xx,yy,PL[,,2],col=1,lty=1,lwd=1.5,add=TRUE,labcex=1.1)
@
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_nn_pl3, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(x[,1],x[,2],col=y,pch=y,xlim=c(xmin[1],xmax[1]),ylim=c(xmin[2],xmax[2]),
     main=expression(paste("pl(",omega[3],")")),
     cex.main=1.5,xlab="",ylab="")
contour(xx,yy,PL[,,3],col=1,lty=1,lwd=1.5,add=TRUE,labcex=1.1)
@
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_fourclass_nn_pl4, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(x[,1],x[,2],col=y,pch=y,xlim=c(xmin[1],xmax[1]),ylim=c(xmin[2],xmax[2]),
     main=expression(paste("pl(",omega[4],")")),
     cex.main=1.5,xlab="",ylab="")
contour(xx,yy,PL[,,4],col=1,lty=1,lwd=1.5,add=TRUE,labcex=1.1)
@
\end{minipage}
\caption{Contour plots of the plausibilities of the four clusters computed from  the \code{fourclass} dataset using the NN-EVCLUS algorithm.  \label{fig:fourclass_nn_pl}}
\end{figure}

\paragraph{Comparison of credal partitions.} The quality of a credal partition can be assessed by its consistency with the true partition (when it is known) and by its nonspecificity. Function \fct{create\_hard\_credpart} transforms a vector of class labels encoding a hard partition into a \class{credpart} object, and function \fct{pairwise\_mass} computes the relational representation of a credal partition (defined in Section \ref{subsec:credpart}). Function \fct{nonspecificity} computes the average nonspecifity of pairwise mass functions $m_{ij}$ in the relational representation of a credal partition. The following piece of code computes the consistency degrees and nonspecificities of the credal partitions generated by EVCLUS, ECM and NN-EVCLUS:
<<RI_NS>>=
Ptrue <- pairwise_mass(create_hard_credpart(y))
P_evclus <- pairwise_mass(clus.evclus)
RI_evclus <- credal_RI(Ptrue,P_evclus,type="c")
NS_evclus <- nonspecificity(P_evclus)
P_ecm <- pairwise_mass(clus.ecm)
RI_ecm <- credal_RI(Ptrue,P_ecm,type="c")
NS_ecm <- nonspecificity(P_ecm)
P_nn <- pairwise_mass(clus.nn)
RI_nn <- credal_RI(Ptrue,P_nn,type="c")
NS_nn <- nonspecificity(P_nn)
print(c(RI_evclus, RI_ecm, RI_nn))
print(c(NS_evclus, NS_ecm, NS_nn))
@
We can see that the three credal partitions have similar consistency degrees. The credal partition generated by EVCLUS is more specific because, by default, masses are assigned to the empty set, singletons and $\Omega$ (and to no strict subset of $\Omega$ of cardinality greater than 1).

\subsection[S2 dataset]{\code{S2} dataset}

With $c$ clusters, the number of focal sets of the mass functions in a credal partition can be as high as $2^c$, which is not tractable for large $c$. If we allow masses to be assigned to pairs of clusters, as suggested by \cite{denoeux04b} and \cite{masson08}, the number of focal sets becomes proportional to $c^2$, which is manageable for moderate values of $c$ (say, until 10), but is still impractical when $c$ is very large.  It is clear, however, that only the  pairs of overlapping clusters will be assigned some mass during the learning process. 

To determine which pairs of clusters can potentially become focal sets,  a two-step approach was proposed by \cite{denoeux16a}:
\be
\item In the first step, a credal clustering algorithm is run with focal sets of cardinalities 0, 1 and $c$. A credal partition $\calM_0$ is obtained. The similarity between each pair of clusters $(\omega_j,\omega_\ell)$ is measured by
\begin{equation}
S(j,\ell)=\sum_{i=1}^n pl_{ij}pl_{i\ell},
\end{equation}
where $pl_{ij}$ and $pl_{i\ell}$ are the normalized plausibilities that object $i$ belongs, respectively, to clusters $j$ and $\ell$.  We then determine the set $\calP_k$ of pairs $\{\omega_j,\omega_\ell\}$ that are mutual $k$ nearest neighbors, according to  similarity measure $S$.
\item In the second step, the  credal clustering algorithm is initialized with the previous credal partition $\calM_0$, but  adding as focal sets the pairs in $\calP_k$; it is run again until convergence.
\ee

To illustrate this methodology, we consider the \code{S2} dataset \citep{franti18}  composed of $n=5000$ two-dimensional vectors grouped in 15 Gaussian clusters:
<<load_S2>>=
data(s2, package="evclust")
n <- nrow(s2)
@
We first use the EK-NNclus algorithm with $K=200$ neighbors, starting from five different initial random hard partitions in 500 clusters:
<<EK-NNclus_S2>>=
clus.eknnclus <- EkNNclus(s2, K=200, y0=sample(500,n,replace=TRUE), 
  ntrials=5, q=0.9, disp=FALSE)
print(clus.eknnclus$N)
@
The resulting partition is close to a hard partition, with a mean nonspecificity close to zero. It is plotted in Figure \ref{fig:S2_pairs}a. We can see that the algorithm has correctly identified the 15 clusters. This credal partition can be used to initialize the EVCLUS algorithm. We first compute the Euclidean distances between each input vector and \code{k=100} random input vectors:
<<s2_distance>>=
Dist <- createD(s2,k=100)
@
and use these distances as inputs to function \fct{kevclus}, with focal sets restricted to the empty set, singletons and $\Omega$:
<<EVCLUS_S2_1>>=
clus.evclus1 <- kevclus(D=Dist$D, c=15, J=Dist$J, type='simple', 
  d0=quantile(Dist$D,0.25), m0=clus.eknnclus$mass, maxit=100, epsi=1e-4, 
  disp=FALSE)
@

Function \fct{createPairs} then finds the mutual \code{k} nearest neighbor pairs of clusters; here, setting \code{k=2}, we get 11 cluster pairs: 
<<pairs_S2>>=
P <- createPairs(clus.evclus1, k=2)
print(t(P$pairs))
@
Finally, we run the EVCLUS algorithm a second time, starting with the previous credal partition \code{P\$m0} and adding the pairs of clusters \code{P\$pairs} found in the previous step as focal sets:
<<EVCLUS_S2_2>>=
clus.evclus2 <- kevclus(D=Dist$D, c=15, J=Dist$J, type='pairs', 
  pairs=P$pairs, d0=quantile(Dist$D,0.25), m0=P$m0, maxit=100, epsi=1e-4, 
  disp=FALSE)
@
The final credal partition is displayed in Figure \ref{fig:S2_pairs}b. It is clearly more informative than the initial partition found by EK-NNclus, as overlapping regions between clusters are now clearly identified.

\begin{figure}[t!]
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_s2_eknnclus, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.eknnclus, s2)
@
(a)
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_s2_evclus, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.evclus2, s2)
@
(b)
\end{minipage}
\caption{Plots of the \code{S2} dataset with the credal partitions generated by \fct{EkNNclus} (a) and \fct{kevclus} with selected pairs of clusters as focal sets (b).  \label{fig:S2_pairs}}
\end{figure}

\subsection[Iris dataset]{\code{Iris} dataset}

To illustrate the Bootclus algorithm, we consider the famous \code{Iris} dataset, which contains the measurements in centimeters of  sepal and petal length and width for 50 flowers from each of three species of iris: \emph{Iris setosa}, \emph{versicolor}, and \emph{virginica}:
<<load_iris>>=
data("iris", package="datasets")
x<-iris[,1:4]
Y<-as.numeric(iris[,5])
n<-nrow(x)
@
The Bootclus algorithm can be applied to these data by running function \fct{bootclus}:
<<iris_bootclus>>=
fit <- bootclus(x, param=list(G = 3))
@
where the argument \code{G = 3} is passed to function \fct{Mclust} from package \pkg{mclust} called internally by  \fct{bootclus}. This function searches for the best GMM out of 14 models with different contraints on the cluster volumes, shapes and orientations. The selected model can be displayed as
<<show_model_iris_bootclus>>=
print(fit$clus$modelName)
@
It is model ``VEV'' corresponding to ellipsoidal clusters with equal shape. The default confidence level of the bootstrap percentile confidence intervals is $1-\alpha=0.9$ (See Section \ref{subsec:bootclus}). By default, the focal sets are the singletons and the pairs; the computed mass functions thus have $f=6$ focal sets in this case. Figure \ref{fig:iris_approx} displays the pairwise belief and plausibility degrees vs. the lower and upper bounds of the 90\% bootstrap percentile confidence intervals. We can see that  the confidence bounds are quite well approximated by the belief-plausibility intervals. Some belief values are smaller than the lower bounds of the confidence intervals (Figure \ref{fig:iris_approx}a), which suggests that the coverage probability of the belief-plausibility intervals is actually larger than the 90\% specified level.



\begin{figure}[t!]
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_iris1, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(as.dist(fit$CI[1,,]),as.dist(fit$BelPl[1,,]),pch=20,xlab="",ylab="")
title(xlab="Lower bound of 90% CI",ylab="Belief",line=2.2,cex.lab=1.7)
abline(0,1)
@
(a)
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_iris2, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(as.dist(fit$CI[2,,]),as.dist(fit$BelPl[2,,]),pch=20,xlab="",ylab="")
title(xlab="Upper bound of 90% CI",ylab="Plausibility",line=2.2,cex.lab=1.7)
abline(0,1)
@
(b)
\end{minipage}
\caption{Approximation of confidence intervals by belief-plausibility intervals for the \textsf{Iris} data. (a) Lower bound $P_{ij}^l$ of the 90\% confidence interval on pairwise probabilties $P_{ij}$ ($x$-axis) vs. belief degree $Bel_{ij}(\{s_{ij}\})$ ($y$-axis); (b) Upper bound $P_{ij}^u$ of the 90\% confidence interval on $P_{ij}$ ($x$-axis) vs. plausibility degree $Pl_{ij}(\{s_{ij}\})$ ($y$-axis).\label{fig:iris_approx}}
\end{figure}


Figure \ref{fig:iris_credalpart} shows the obtained credal partition encoded in \code{fit$Clus}. We can see that the \emph{setosa} group, which is well separated from the other two, has a precise representation (for that cluster, the lower and upper approximations are equal). In contrast, the other two groups are overlapping, and assigning some instances to only one group would be highly uncertain. Some instances are thus assigned to the  meta-cluster formed by the union of the \emph{versicolor} and \emph{virginica} groups, and belong to  the upper approximations of these two clusters.  

\begin{figure}[t!]
\centering
<<plot_iris_partition, echo=FALSE, fig=TRUE, height=5,width=5>>=
col<-1:3
lower.approx <- fit$Clus$lower.approx
upper.approx <- fit$Clus$upper.approx
plot.credal <- function(x,y){
  points(x,y,col=col[fit$Clus$y.pl], pch=Y, cex=apply(fit$Clus$pl,1,max))
  for(i in (1:3)){
    icol <- col[i] 
    xx <- cbind(x[lower.approx[[i]]], y[lower.approx[[i]]])
    polygon(xx[chull(xx),], lty = 1, lwd=1.5, border=icol)
    xx <- cbind(x[upper.approx[[i]]], y[upper.approx[[i]]])
    polygon(xx[chull(xx),], lty = 2, lwd=1.5, border=icol)
  }
}
pairs(x, panel = plot.credal)
@
\caption{Plots of the \code{Iris} dataset with the credal partitions generated by \fct{bootclus}. The true groups are represented by different symbols (o: setosa; triangle: versicolor; +: virginica), and the maximum-plausibility  groups are represented by different colors. The solid and broken lines represent, respectively, the convex hulls of the lower and upper approximation of each cluster. \label{fig:iris_credalpart}}
\end{figure}

The following confusion matrix shows that five objects from the \emph{versicolor} group are misclassified into the \emph{virginica} group in the initial partition created by \fct{mclust}:
<<confusion_iris>>=
table(iris[,5],fit$clus$classification)
@
In contrast, after summarizing the  credal partition into a rough partition by approximating each mass function by a set using \eqref{eq:mass_transform}, nine instances from the \emph{versicolor} class and one instance from the \emph{virginica} class are assigned to the meta-cluster formed by the union of the \emph{versicolor} and \emph{virginica} groups, and ony one instance from the \emph{versicolor} is incorrectly assigned to the \emph{virginica} group: 
<<confusion_rough_iris>>=
rough_partition<-apply(fit$Clus$Y,1,paste,collapse="")
table(iris[,5],rough_partition)
@
(In the above table, sets are represented in binary notation, i.e., `100' represents $\{\omega_1\}$, `101' represents $\{\omega_1, \omega_3\}$, etc.). 

\subsection[Bananas dataset]{\code{Bananas} dataset}

Discovering clusters with complex shapes is a challenging task for clustering algorithms. For instance, function \fct{bananas} generates two banana-shaped clusters:
<<generate_bananas>>=
data<-bananas(300)
x<-data$x
y<-data$y
@
Clustering algorithms such as ECM or EVCLUS do not perform well on these data, as they are implicitly based on the definition of a cluster as a set of objects similar to each other. To detect the two clusters, we need additional information, such as must-link and cannot link constraints. Given the true class labels, function \fct{create\_MLCL} generates such constraints by randomly picking pairs of objects from the same class and from different classes with equal probabilities. Here, we use it to generate 400 constraints:
<<generate_constraints>>=
const<-create_MLCL(y,400)
@
The output \code{const} is a list with two components: a matrix \code{ML} of must-link constraints, and a matrix \code{CL} of cannot-link constraints, both with two columns and as many rows as constraints. Note that these 400 pairwise constraints (some of which represented in Figure \ref{fig:constraints}a) represent less than 1\% of the 44850 object pairs.

There are two main complementary approaches to exploit such additional information. The first one is to account for the constraints in the cost function; this is the approach used in the CECM and CEVCLUS algorithms. For instance, we can run the \fct{cecm} function as
<<bananas_cecm1>>=
clus.cecm <- cecm(x, c=2, ML=const$ML, CL=const$CL, ntrials=5, xi=0.9, 
  distance=1, disp=FALSE)
@
where \code{xi} is the hyperparameter in \eqref{eq:CECM}, and the algorithm is run \code{ntrials=5} times. The resulting credal partition is plotted in Figure \ref{fig:constraints}b. The adjusted Rand index (ARI) between the maximum-plausibility hard partition and the true partition can be computed using function \fct{adjustedRandIndex} of package \pkg{mclust} as
<<ARI_CECM>>=
library("mclust")
print(adjustedRandIndex(clus.cecm$y.pl, y))
@

\begin{figure}[t!]
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_constraints, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(x,pch=y,col=y,xlab=expression(x[1]),ylab=expression(x[2]))
for(k in sample(nrow(const$ML),20)) lines(x[const$ML[k,],1],x[const$ML[k,],2])
for(k in sample(nrow(const$CL),20)) lines(x[const$CL[k,],1],x[const$CL[k,],2],lty=2)
@
(a)
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_bananas_cecm, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.cecm,x,ytrue=y,plot_approx=FALSE,plot_protos=TRUE)
@
(b)
\end{minipage}
\caption{(a): \code{Bananas} dataset with 20 must-link constraints (solid lines) and 20 cannot-link constraints (broken lines) out of the 400 constraints generated by function \fct{create\_MLCL} for the \code{bananas} dataset; (b): credal partition generated by \fct{cecm} using the 400 pairwise constraints.\label{fig:constraints}}
\end{figure}


The other approach is to map the data to a new feature space in which pairs of objects known to belong to the same cluster are close to each other, while pairs of objects known to belong to different clusters are far apart. One such method is the kernel pairwise constrained component analysis (KPCCA) algorithm proposed by \cite{mignon12} and implemented in \pkg{evclust} as function \fct{kpca}; this function takes as input a kernel matrix, which can be generated by function \fct{kernelMatrix} of package \pkg{kernlab}:
<<kpca>>=
library(kernlab)
rbf <- rbfdot(sigma = 0.2)
K <- kernelMatrix(rbf, x)
res.kpcca <- kpcca(K, d1=2, ML=const$ML, CL=const$CL, epsi=1e-3, 
  disp=FALSE)
@
where \code{K} is a kernel matrix, \code{d1} is the number of extracted features, and the algorithm stops when the rate of change of the cost function is less than \code{epsi}. Function \fct{kpca} returns a list with three components: the new feature matrix \code{z} of size $n \times d_1$, the projection matrix \code{A} of size $d_1 \times n$, and the  Euclidean distance matrix \code{D} in the new feature space. 

Having extracted \code{d_1=2} features, we can run function \fct{ecm} in the new feature space:
<<bananas_ecm>>=
clus.ecm<-ecm(res.kpcca$z,c=2,disp=FALSE)
@
The generated partition is diplayed in the transformed and original feature spaces, respectively, in Figures \ref{fig:bananas_ecm}a and \ref{fig:bananas_ecm}b. The result is similar to that obtained with the previous method according to the ARI of the  maximum-plausibility partition:
<<ARI_ECM>>=
print(adjustedRandIndex(clus.ecm$y.pl, y))
@
Combining the two methods further improves result in terms of ARI:
<<bananas_cecm2>>=
clus.cecm1 <- cecm(res.kpcca$z, c=2, ML=const$ML, CL=const$CL, ntrials=5, 
  xi=0.9, distance=1, disp=FALSE)
print(adjustedRandIndex(clus.cecm1$y.pl, y))
@


\begin{figure}[t!]
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_bananas_ecm1, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.ecm,res.kpcca$z,plot_approx=TRUE)
@
(a)
\end{minipage}
\begin{minipage}[b]{.5\linewidth}
\centering
<<plot_bananas_ecm2, echo=FALSE, fig=TRUE, height=5,width=5>>=
plot(clus.ecm,x,ytrue=y,plot_approx=FALSE,plot_protos=FALSE)
@
(b)
\end{minipage}
\caption{Credal partition generated by \fct{ecm} for the \code{bananas} dataset in the transformed feature space, with 400 randomly generated pairwise contraints. The partition is shown in the transformed (a) and original (b) feature spaces. \label{fig:bananas_ecm}}
\end{figure}

%% -- Summary/conclusions/discussion -------------------------------------------

\section{Conclusions} \label{sec:summary}

Evidential clustering is a new approach to clustering in which uncertainty about  cluster membership is described by Dempster-Shafer mass functions. Evidential clustering algorithms compute credal partitions defined as tuples of mass functions over the set $\Omega$ of clusters. For each object $i$, the mass  $m_i(A)$  assigned to  nonempty set $A$ of clusters reflects ambiguity in the assignment of the object to clusters in $A$, while the  mass $m_i(\emptyset)$ assigned to the empty set reflects the possibility that the object might not belong to any of the clusters and allows us to detect outliers. This representation is very general and encompasses other approaches such a fuzzy, possibilistic and rough clustering as special cases.     

The \pkg{evclust} package described in this paper implements a set of efficient evidential  clustering algorithms, as well as functions to display, evaluate and exploit credal partitions. These algorithms are based on different principles and make it possible to address a variety of clustering problems, including difficult ones such as clustering nonmetric dissimilarity data, discovering complex-shaped clusters, and constrained clustering. For lack of space, we did not address the important problem of determining the number of clusters. Whereas the EK-NNclus algorithm does not require to specify that number, other algorithms do. In the case of the Bootclus algorithm relying on model-based clustering, analytical criteria such as AIC can be used. For the other algorithms, the nonspecifity of the credal partition can be used to select a good credal partition \citep{masson08}. Another approach is to use generic indices such as implemented in the \pkg{NbClust} package \citep{JSSv061i06}. 

Whereas \pkg{evclust} is dedicated to clustering, a companion package \pkg{evclass}  focusses on evidential supervised classification \citep{denoeux17}. The author hopes that the availability of these packages will contribute to a more widespread dissemination of tools based on belief functions in data analysis and machine learning, and draw the attention of an increasing number of data scientists  to this new research direction.


% \section*{Acknowledgments}
% 
% \begin{leftbar}
% All acknowledgments (note the AE spelling) should be collected in this
% unnumbered section before the references. It may contain the usual information
% about funding and feedback from colleagues/reviewers/etc. Furthermore,
% information such as relative contributions of the authors may be added here
% (if any).
% \end{leftbar}


%% -- Bibliography -------------------------------------------------------------
%% - References need to be provided in a .bib BibTeX database.
%% - All references should be made with \cite, \citet, \citep, \citealp etc.
%%   (and never hard-coded). See the FAQ for details.
%% - JSS-specific markup (\proglang, \pkg, \code) should be used in the .bib.
%% - Titles in the .bib should be in title case.
%% - DOIs should be included where available.

\bibliography{refs}


%% -- Appendix (if any) --------------------------------------------------------
%% - After the bibliography with page break.
%% - With proper section titles and _not_ just "Appendix".



\end{document}
