The electrocaloric effect, that is, the temperature change experienced by an insulator upon application of an electric field, offers promising ecofriendly alternatives to refrigeration. However, the theoretical treatments of this response are mostly case specific and lack a unified picture revealing the similarities and differences among the various known effects. Here, we show that the electrocaloric effect lends itself to a straightforward interpretation when expressed as a Taylor series in the external field. Our formalism explains in a unified and simple way the most notable small-field effects reported in the literature, namely the so-called normal and inverse electrocaloric responses, corresponding to an increase or decrease of temperature under applied field, as usually found in ferroelectrics or antiferroelectrics, respectively. This helps us to clarify their physical interpretation. We then discuss in detail atomistic simulations for the prototype ferroelectric PbTiO3, explicitly evaluating subtle predictions of the theory, such as the occurrence of competing contributions to the electrocaloric response.