I'm dieting: Can you lose breasts size even though it's part of genetics?

Majority of women on my mom and dad's side are busty women. I'm tried to lose some of it. I lost quiet a few lbs, but still busty. My grandma told me I can't get rid of it, it's the female inheritance of our family.