The Memory of All That: George Gershwin, Kay Swift, and My Family's Legacy of Infidelities

The Memory of All That is Katharine Weber’s memoir of her extraordinary family. 

Her maternal grandmother, Kay Swift, was known both for her own music (she was the first woman to compose the score to a hit Broadway show, Fine and Dandy) and for her ten-year romance with George Gershwin. Their love affair began during Swift’s marriage to James Paul Warburg, the multitalented banker and economist who advised (and feuded with) FDR. Weber creates an intriguing and intimate group portrait of the renowned Warburg family, from her great-great-uncle, the eccentric art historian Aby Warburg, whose madness inspired modern theories of iconography, to her great-grandfather Paul M. Warburg, the architect of the Federal Reserve System whose unheeded warnings about the stock-market crash of 1929 made him “the Cassandra of Wall Street.” 

As she throws new light on her beloved grandmother’s life and many amours, Weber also considers the role the psychoanalyst Gregory Zilboorg played in her family history, along with the ways the Warburg family has been as celebrated for its accomplishments as it has been vilified over the years by countless conspiracy theorists (from Henry Ford to Louis Farrakhan), who labeled Paul Warburg the ringleader of the so-called international Jewish banking conspiracy. 

Her mother, Andrea Swift Warburg, married Sidney Kaufman, but their unlikely union, Weber believes, was a direct consequence of George Gershwin’s looming presence in the Warburg family. A notorious womanizer, Weber’s father was a peripatetic filmmaker who made propaganda and training films for the OSS during World War II before producing the first movie with smells, the regrettable flop that was AromaRama. He was as much an enigma to his daughter as he was to the FBI, which had him under surveillance for more than forty years, and even noted Katharine’s birth in a memo to J. Edgar Hoover.

Colorful, evocative, insightful, and very funny, The Memory of All That is an enthralling look at a tremendously influential—and highly eccentric—family, as well as a consideration of how their stories, with their myriad layers of truth and fiction, have both provoked and influenced one of our most prodigiously gifted writers.