- What to do with waste products, if not feed them to animals?
Organic (of or relating to an organism) waste products all compost and if you want everyone in the world to live on grains and vegetables then you want all the compost you can get. In fact, you want to start using some of the waste products that the west mostly does waste already (nightsoil).
Wouldn't it make sense to have at least some animals around to eat these waste products and upcycle them?
No, it doesn't necessarily make sense. There is a cost and a benefit to having animals around. Their feed is part of the cost but it is not all of it. They have a footprint on the land. They may have emissions of their own. There are other handling costs. And so on. Their value as food is one benefit. If their waste stream can be captured and made use of then that is a value. They may fill in for certain ecosystem services.
Feeding some products from the waste stream itself has a cost. Those products are no longer available to be used in another re-cycling process. It has a benefit. The feed cost of the animals is reduced.
The cost/benefit analysis on having the animals around will come to some conclusion. Feeding materials from the waste stream to those animals will change the cost/benefit analysis but it doesn't necessarily change it all the way from "costs outweigh benfits" to "benefits outweigh costs". It doesn't even necessarily move it in that direction. It is likely that a particular waste stream, particular animals, and a particular system needs to be analysed to understand what the real effects are.
Biological systems can be quite complex. As an example of how the benefit of diverting some of the waste stream could be outweighed by the cost of keeping livestock around, consider that methane might be produced as a side-effect of keeping livestock around. The total amount of carbon in the system obviously has not changed but in methane the warming effects of that carbon have been greatly magnified. The carbon would have been much less impactful if it had ended up in a carrot or a soil microorganism. This specific issue is well-studied from multiple angles (from measurement - https://advances.sciencemag.org/content/7/14/eabe9765 - to extrapolation from prior data - https://extension.psu.edu/livestock-methane-emissions-in-the-united-states - to give just two examples).
- Most grass-lands are not arable
This part of the question implicitly mixes two concerns. The area available for non-forage crop production doesn't obviously or directly impact the sustainability of the use of that area (though in the extremes certainly this matters - you can get away with a lot on 1 square meter inside 1 billion square meters of healthy surrounding ecosystem; likewise, convert too much area to crop production and you risk consequences like desertification which may then destroy the capability of that land to produce crops).
Instead, what the area available for non-forage crop production directly impacts is total production capacity - and therefore the number of people you can feed.
If we consider these two factors - production capacity and sustainability - explicitly then we end up with the idea of carrying capacity.
A population can temporarily exceed the carrying capacity by increasing production at the expense of sustainability.
So, what is the carrying capacity of earth if everyone is vegan? How does this compare to the carrying capacity if some people are eating some animal products?
Here there is significant disagreement. Some argue that livestock cannot exist in any meaningful numbers without doing more harm than good. Some argue that livestock are part of the solution, not part of the problem, and we should keep as many of them as we can (using systems that result in carbon draw-down). If there is any legitimate consensus, it is not clear to me what it is.
Additionally, the world is changing around us. As we get closer to certain tipping points (the exact position of which we don't even really know) the considerations change. What's better - N CO2-equivalent units of methane in the air and M CO2-equivalent units of carbon sequestered in the soil (call this the livestock scenario), or X CO2-equivalent units of methane in the air and Y CO2-equivalent units of carbon in the air (call this the vegan scenario)? The answer twenty years ago is probably not the same as the answer today and may not be the same as the answer in five years.