Background Fear of recurrence (FOR) is a primary concern for both cancer survivors and their caregivers, yet little is known about what care-related factors exacerbate this worry. Objectives This study aimed to establish the role of care-related stressors—as distinct from survivor characteristics—in predicting FOR in head and neck cancer caregivers. Methods HNC survivor–caregiver dyads took part in a mailed survey. Survivors provided information on health and quality of life (using the Functional Assessment of Cancer Therapy Questionnaire). Caregivers provided sociodemographic information, impact of caring on their time and finances, as well as their level of social support (Oslo Support Scale), loneliness (3-point loneliness scale), and completed the Worry of Cancer Scale (to measure FOR). Results Data from 180 dyads were available for analysis. Multiple regression analysis was used to examine the role of caregiver stressors, social support, and loneliness while controlling for caregiver and survivor characteristics. The model explained 28% of the variance in Worry of Cancer scores (FOR). Caregivers who reported more loneliness, spent more time caring, and had greater financial stress from caring had higher scores on Worry of Cancer (FOR). Female caregivers, those caring for younger survivors, and those with survivors who had undergone less extensive forms of surgery also reported higher FOR. Discussion A combination of factors place caregivers at greater risk of cancer-related worry, paving the way for designing interventions aimed at reducing FOR in caregivers of patients with head and neck cancers.