Sister Wives’ Kody and Robyn ‘Made an Agreement’ to ‘Free Each Other’ If They Fell Out of Love

During Sunday’s ‘Sister Wives: One on One’ special, Kody Brown said his three now-ex-wives were “frustrated” upon learning of his pact with Robyn Brown

(L-R) Kody and Robyn Brown are pictured. Photo: TLC

Kody Brown made a pact in his marriage with Robyn Brown many years back that wasn’t extended to all of his wives.

“Robyn and I made an agreement that if we weren’t in love, that we wouldn’t drag each other out,” Kody, 54, said during Sunday’s Sister Wives: One on One special, hosted by Sukanya Krishnan. “That we would free each other.”

When asked by Sukanya whether he made that “same covenant” with his first wife Meri Brown, the Brown family patriarch said, “No, no.”

“I remember after Robyn and I had that agreement, then I went to the other wives and said I have this agreement with Robyn,” he recalled. “They even got frustrated by, like anybody who wants to leave, you’re free to go. And that even made Meri mad.”

Sister Wives’ Kody Brown on His ‘Deep State of Mourning’ and ‘Strange’ Transition to Monogamy with Robyn (Exclusive)

Meri, 52, spoke to Sukanya about the “inaction” from Kody within their marriage. While she upheld the commitment to maintain their marriage, Meri noted how Kody only “holds it dear with Robyn.”

Describing the difference between her covenant with Kody compared to his with Robyn, she said her now-ex-husband “fell out of love” with her. “I only base that off of that scene where he said to her, ‘If I ever fall out of love with you, don’t string me along,’ or whatever his words were,” she explained.

Kody Brown is pictured on ‘Good Morning America’ on September 21, 2011.

Ida Mae Astute/Disney General Entertainment Content via Getty

“He’s sitting here telling Robyn, ‘That’s the agreement that you and I made, that if I ever felt like that, then I wouldn’t hold you in this place.’ Their agreement of, you know, their sacred covenant,” she said. “I’m like, ‘Then why did you not respect me enough as a human being?'”

Kody’s marriages to Meri, Christine Brown and Janelle Brown fell apart within 14 months. Though he’s now only married to Robyn, the pair — who wed in 2014 — have experienced many challenges within their relationship following his three divorces.

Robyn, 45, told Sukanya during Sunday’s special that Kody “tries to” sabotage their relationship and she has to “stop him all the time.”

(L-R) Christine Brown, Janelle Brown, Kody Brown, Robyn Brown and Meri Brown are pictured in a promotional still for TLC’ ‘Sister Wives’. TLC

Sister Wives’ Robyn Brown Says Life with Kody Has Been ‘Hell’ at Times amid Splits but ‘He’s Doing His Best’ (Exclusive)

“He picks fights with me [over] dumb things,” she said through tears. “We’re in different places. We’re in different places about the bomb that went off on our family. And like, I’m in a major place of mourning and he’s angry. He just wants to just go find whatever, you know? And I’m like, ‘No, I can’t feel like that.'”

Admitting to “sabotaging myself a little bit … out of the frustration and anger,” Kody also acknowledged he was purposely hurting his marriage with Robyn because he wanted to “punish” himself. He even considered the possibility of leaving Robyn as a way to further punish himself.

“I thought of myself leaving Robyn and having another lover and looking at this lover and going, ‘I don’t love you. I’m in love with another woman. I’m in love with a woman that I left because I was too much a piece of s— to manage the relationship,'” he explained. “What it was is just anger and I’m embarrassed. So, you get embarrassed and then you get self-deprecating. It’s like, just bad behavior.”

Kody added, “A lot of dark winds were going through me. A lot of devil, a lot of temptation, which would be destructive of my relationship with Robyn. So I dealt with a lot of anger and she would get frustrated. My anger was a turn-off. It was scary.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.