The answer is: Captian Anthony who was probably his father. Anthony is the clerk for Colonel Lloyd, managing Lloyd's surrounding plantations and the overseers of those plantations. Anthony is a cruel man who takes pleasure in whipping his slaves, especially Douglass's Aunt Hester.
