The longer I go, I think trading Duren and shifting Stew to starting 5 and making Ausar the primary rim runner solves a lot of issues for now.
Stew’s undersized but let’s be real he still defends a lot better than Duren and is more switchable. His lack of off-the-dribble skill is nowhere near as bad at the 5 as it is at the 4, and that 38% from three is actually a real plus there and would make a huge difference with the spacing. 
If by some miracle we could land OG in free agency and trade Duren and #5 for Ingram (or another wing) and #21 as mocked on the Athletic and draft whoever falls of the stretch 4 or 5’s (Tyler Smith, Daron Holmes, Filipowski, Da Silva, etc.). Even better if it were Ivey, Duren, and 5 for Mikal Bridges (but probably unrealistic). Mikal, Ausar, OG, and Stew as starting 2-3-4-5 would nasty even if it’s just a dream haha
Cade / Ivey / Sasser
Ausar / Grimes
Ingram or other trade target/ Fontecchio
OG / #21
Stewart / FA?
Basically everyone in that starting lineup is 6’6” to 6’8” and switchable. Ausar, OG, and Stew are legit plus defenders. Only 1 non-shooter. (If only Cade, OG, and Ingram could play more than 50% of games)
Just need some size off the bench and/or in the draft. Probably couldn’t afford much after these moves. Maybe a Drummond reunion as backup C 

In the long run you hope to land a true starting 5 that can protect the rim, shoot, and rim run and bring stew off the bench.