I think you're missing the point of that scene and Luke's character in general. First, he states to Vader that he feels the good in him, the conflict. He mentions this at least four times in the movie, twice to Vader, once to Obi-Wan, once to Leia. In TLJ, in that scene with Ben, he EXPLICITLY states that ALL he felt was darkness, that Snoke had already poisoned his mind, that there was no going back for Ben. If you listen closely you can even hear people screaming out and dying. Luke didn't want this to be his greatest failure. And yet even after that, he felt shame for even thinking of killing Ben. He says the thought "passed as quickly as a shadow" and he talks of feeling shame for this. By the time Ben woke up it was too late to fix it, and we know what happens. The temple burns, Luke is at a breaking point with the Jedi, and he exiles himself (Yoda did the same thing). (Also, I point out, it was JJ who stuck Luke on that island, not Rian). That, coupled with the fact that for some reason people are forgetting just how impulsive Luke was and is. In Empire, he defies Yoda and goes to Cloud City and loses, putting his friends in greater peril. In Jedi, he nearly kills his own father even AFTER sensing the good in him. Only after seeing his machine hand does he pull back. And in Jedi again, he waltzes into Jabba's palace and thinks he can wreck **** up and yet he ends up with the Rancor and miraculously escapes. This is Luke's character. This is why he exiled himself, because he failed...failure is a huge part of this movie and Yoda even says to him that failure is our greatest teacher. I feel strongly that people didn't like this movie because of preconceived notions as to what we should have seen, such as a lightsaber battle, or Rey being related for some reason to a Kenobi or Skywalker, or Snoke being someone other than just an **** who was Kylo Ren's master, or Luke going all super Jedi and killing everyone. Maybe Rian Johnson went too filmy with this one, too much subtext, not enough familiarity, but that's exactly what people complained about with TFA. To me, they handled Luke's character perfectly and gave him the best possible ending. TLJ is as much about his redemption and newfound status as a true legend (see the last scene) as anything else.
