Michele Flournoy, 53, was widely considered the frontrunner for the post of defense secretary and if confirmed would have been the first woman to hold the position.
But Flournoy, the former number-three-ranking official at the Pentagon, wrote to the board of the think tank she leads and founded, the Center for a New American Security (CNAS), saying she would not be taking up the position.
"I can confirm she has written to the CNAS board informing them she has taken herself out of the running for the defense secretary's job," said the source, who spoke on condition of anonymity.
The reasons she cited for the decision were "family-related," the source said.
The Foreign Policy website first broke the news.
Flournoy's withdrawal presents a potential headache for President Barack Obama as he searches for a candidate that can gain acceptance from lawmakers from both parties.
Speculation will shift to other possible candidates for the post, including Ash Carter, an academic by training who rose to the number-two job at the Pentagon, and known as an expert on budgets and hi-tech weaponry.
"Last night I spoke with President Obama and removed myself from consideration due to family concerns," Flournoy wrote to the CNAS board, according to Foreign Policy.
"After much agonizing, we decided that now was not the right time for me to re-enter government. The good news is that you all are stuck with me for the indefinite future!"
Flournoy told board members that family health considerations had influenced her decision to withdraw, as well as the prospect of her two children leaving for college in the next two years, the source said, confirming Foreign Policy's report.
Obama announced Hagel's resignation on Monday, with officials indicating the former senator had stepped down under pressure amid reports he had lost the confidence of the president and some of his top aides.
The next defense secretary will face the daunting task of managing the air war against the Islamic State group in Iraq and Syria, at a moment when the president's strategy has come under mounting criticism.