WARNING: This page is about a different car, the 2009 Cadillac STS. However, it is still accessible from the selected car via links, so may be relevant.
Fig 1: View Of Center Console End Panel
Courtesy of GENERAL MOTORS CORP.
Release the retaining clips using a blunt trim tool.