James (Jimmy) Henry Jones was born in Memphis, Tennessee in 1918. As a child he started on both guitar and piano. Jones's father was an English professor at LeMoyne-Owen College in Memphis and also played violin and conducted choirs at churches in Memphis and then in Chicago when the family moved there in the early 1920s. Jones's mother played piano at home. As a teenager, Jones took private piano lessons and sang in his church. By 1936 Jones was working in Chicago with various bands. He joined the Stuff Smith Trio in 1943 working alongside friend and bassist John Levy for two years. Throughout the 1940s he was an in-demand session pianist recording with many of the biggest names in New York including Dizzy Gillespie, Don Byas, Buck Clayton, Mildred Bailey, Coleman Hawkins, Ben Webster, Johnny Hodges, Etta Jones, and Rex Stewart. In 1947 Jones began a four year stint with Sarah Vaughan appearing on many of her recording sessions and live dates.

In the early 1950s Jones contracted tuberculosis and took three years off from performing. After his leave of absence he picked up where he left off with Sarah Vaughan as well as resuming his duties in the studio being the pianist on the noteworthy Helen Merrill/Clifford Brown recording session. In the late 1950s Jones began to perform and record often with current and past members of the Duke Ellington Orchestra including Cat Anderson, Harold 'Shorty' Baker, and Clark Terry. In 1959 he sat in with the Ellington band and shortly thereafter recorded with Mercer Ellington for the Colors in Rhythm album. Duke listened to the album and was impressed with Jones's writing. He would remain a trusted alternate pianist (and arranger) for the Ellington Orchestra for years to come.

By 1960 Jones had built a reputation as a solid vocal accompanist and in the decade the followed recorded with an impressive list of singers including Chris Connor, Big Joe Turner, Monica Zetterlund, Joe Williams, Gloria Lynne, and also Shirley Horn, Nancy Wilson, Mel Torme, Carmen McRae, and Ella Fitzgerald for whom he served as an arranger as well. In Jones's arranging he was able to employ the same sensitivity that he had shown in his accompanying dates. Other than the singers mentioned above, Jones also arranged for Wes Montgomery, Cannonball Adderley, and Johnny Hodges. In 1967 he wrote the arrangement of I Like the Sunrise for the Frank Sinatra/Duke Ellington album. Although un-credited, he also played piano on that one track.

Jimmy Jones passed away in 1982 in Los Angeles, California.