I don't think Caleb wanted power at all. I think Caleb wanted knowledge. He was painfully curious because he was Erudite. I don't think there is a worse faction for an Erudite to be born into than Abnegation for the fact that it is the only faction you are not allowed to thirst for knowledge. You're not allowed to be curious or ask questions. Caleb wanted out because those were the only things he did want. Caleb wanted out beacuse he could no longer go along with what he was "supposed" to do. He wanted to feel like he belonged somewhere. He didn't want to feel like he was suppressing the only thing he was.
He joined Jeanine because she was the first person to seemingly provide him with the information he thought he craved. He thought she was giving him information that was kept from him all this time. Both about his birth faction and about the society he was raised into. He breathed for knowledge, he craved it, and he was deprived of it for his entire life up until the point when Jeanine started handing it to him.
She used that. She knew what sort of power she had over him. She used him, and deceived him. And that is why we should feel sorry for him.