Informal caregivers are overlooked, healthcare actors. They are at particular risk of distress and suffer from poor mental health. This study aimed to investigate the perceived stress and modulating factors during the first COVID-19 lockdown in Europe, regardless of the illness that care recipients suffer from. Sociodemographic data, coping resources, and perceived stress level using the Perceived Stress Scale (PSS-10) questionnaire were assessed using a web-based survey in Switzerland, France, and Belgium with 232 informal caregivers. Mediation analyses were used to identify the factors that modulate stress. Higher perceived stress among informal caregivers was associated with a younger age for the care recipient, family relationship with the care recipient, cohabitation, and female sex of the informal caregiver. These associations were partially mediated by the fear of getting ill (age, cohabitation), the conviction that lockdowns had a negative impact on health (age, kinship), and the perceived deterioration of the care recipient’s health (gender). The fear of losing the ability to cope with caregiving tasks due to an illness (COVID-19 and/or other) and the negative impact of the lockdown on care recipients’ health, particularly on the mental health of young care recipients, increased the stress of informal caregivers. Our results emphasize the importance of informal caregiving support to prevent heightened stress in lockdown conditions, regardless of care recipient illness or kinship.