Joanna Hiffernan was a Roman Catholic, born around 1843 in Ireland. Her mother, Katherine Hiffernan, died in 1862, aged 44. Her father, Patrick Hiffernan, was described by Pennell as 'a sort of Captain Costigan', a drunken Irishman in Thackeray's Pendennis. Pennell also described him as 'a teacher of polite chirography [calligraphy]' who used to speak of Whistler as 'me son-in-law'. Joanna Hiffernan had a sister called Bridget Agnes Hiffernan (later Singleton). Walter Greaves claimed that Joanna Hiffernan had a son called Harry but there is no record of him. She married at some point after 1881, perhaps abroad to a man named Abbot.
Whistler first met Hiffernan in 1860 while she was at a studio in Rathbone Place, according to Ionides, and she went on to have a six year liaison with him. She modelled for some of Whistler's most famous paintings during this period. She was in France with Whistler during the summer of 1861, and in Paris during the winter of 1861-62 sitting for Symphony in White, No. I: The White Girl (YMSM 38) at a studio in Boulevard des Batignolles. It is possible that this is where she met Courbet for whom she later modelled. However, as well as being a model, she painted and drew a little herself.
Hiffernan was not well accepted within Whistler's family. They did not consider her to be respectable. When Whistler's mother visited from America in 1864, alternative accommodation had to be found for her away from 7 Lindsey Row. She also seems to have been the cause of Whistler's quarrel with Legros in 1863.
Hiffernan attended séances with Whistler at D. G. Rossetti's house in Chelsea in 1863. She spent the summer and autumn of 1865 in Trouville with Whistler, and posed for Courbet in Portrait de Jo, la Belle Irlandaise. There are four versions (New York: Metropolitan Museum of Art, Zurich: private collection, Kansas: Nelson-Atkins Museum of Art, New York: private collection). Hiffernan also posed for the oil sketch Portrait de Jo, 1865 (New York: private collection).
In 1866, Whistler gave Jo power of attorney over his affairs whilst he was in Valparaiso for seven months. He made provision for household expenses and gave authority to Hiffernan to act as an agent in the sale of his works. She called herself Mrs Abbott, especially when selling Whistler's works around dealers. During this time Hiffernan travelled to Paris and posed for Courbet in The Sleepers or Le Sommeil (1866, Paris: Musée du Petit Palais). It is likely that they had an affair. Whistler and herself parted shortly after his return. However, according to the Pennells, she adopted Whistler and Louisa Fanny Hanson's son Charles who was born in 1870. He came to live with her at 5 Thistle Grove. Hiffernan continued to look after him as late as 1880 when Whistler was away in Venice with Maud Franklin, his current mistress.
Du Maurier, Daphne, ed., The Young George du Maurier: A Selection of his Letters, 1860-67, London, 1951; Ionides, Luke, Memories, Paris, 1925; Pennell, Elizabeth Robins, and Joseph Pennell, The Life of James McNeill Whistler, 2 vols, London and Philadelphia, 1908; Pennell, Elizabeth Robins, and Joseph Pennell, The Whistler Journal, Philadelphia, 1921; MacDonald, Margaret F. et al, Whistler, Women and Fashion, New Haven and London, 2003.