I'm going to throw an answer out here, but surely there people with more knowledge out there. Please contribute as you can, and suggest corrections to any errors I make here. I have tried to detail my reasoning, but this answer is more hand-waving. For example, I can find no reliable sources to evaluate different types of pollution.
I am going to assume that all materials are going to make it into the recycling stream, since that seems to be the OP's intent. I'm also going to assume that the recycling stream is not being corrupted by "aspirational recycling" where the stream is corrupted by unrecyclable or poorly sorted materials. It affects all of the materials anyway, and again seems to be inline with the OP's intent. I'm also going to ignore the pollution from energy since it is variable depending on the source, but instead just talk about energy as is: more is worse, less is better. It's a good reason why it is necessary to electrify everything, and produce that electricity from renewable resources. (I realise this is a bit of a fudge, especially since paper mills are often located well away from energy sources and are instead located where the wood is, and therefore often burn fuels for energy since it's tough to locate solar panels in forests. It's a good reason to continue to research biofuels.)
I agree with the OP's evaluation of packaging per volume and weights as they relate to transportation costs. Square packages are more efficient in packaging by volume, which also is a plus for boxed wine's transportation costs due to volumes.
The boxed wine consists of a cardboard box and an inside "tetra-pak" plastic. In Ontario, Canada these tetra-paks are recyclable (and there are deposit refunds at The Beer Store). The bottle consists of glass; we'll ignore the metal cap with plastic insert (more fudging :)
Paper/cardboard comes from a renewable resource. These trees do not need to be old forest trees, but are usually replanted forests harvested specifically for paper/cardboard production. (I'll take renewable forest harvesting over drilling or mining every time.) Paper requires a lot of water and energy to produce, and produces toxic pollution (chlorine other chemicals - the paper pulp sludge is toxic). There's a reason pulp/paper mills exist next to lakes and rivers. Cardboard boxes can include a relatively high amount of recycled materials, especially when used as frame cardboard (i.e a wine box or a shipping box, vs. a box that holds food directly), and have good recyclablility (that's a word, right?). Recycling paper takes less energy than recycling glass, which requires a lot of heat.
Glass is made from sand, but only special sand that is typically harvested from lake and river beds. We're currently using more than is being regenerated, so we should count this as a non-renewable resource. Glass is indefinitely recyclable, unlike plastics which can only be recycled a few times. Glass can include (near?) 100% recycled materials; glass is made food safe (there's separation of contaminants in the production process). However, glass can only be recycled with like colours: clear for clear products, green for green products, etc. Glass cannot be recycled when broken (due to worker safety), but I'll assume they make it whole. Wine bottles are shipped in boxes with dividers. I think we can hand-wave ourselves to evaluating the cardboard use in bottle shipping is roughly equal to that of wine boxes and their shipping. I might even suggest that wine bottles use slightly more. Glass production produces less toxic pollution than paper or plastic (not counting energy sources). The sand harvesting, however, does significantly disrupt the environments from which it's sourced, which is a kind of pollution.
Plastic uses fossil fuels to make, which is obviously non-renewable, is damaging to the environment to extract, and takes energy to refine. Plastic takes significantly less energy to make than cardboard or glass. Plastics can only be recycled a few times (e.g. 3-4 cycles) before they break down too much to be viable for new materials. Unfortunately at that point they become waste. Worse they leak chemicals and break down into smaller micro-particles that can get into water sources and the food chain. Food safe plastics contain 0% recycled materials (probably because of concerns about the potential toxicity of the sources). There are numerous articles questioning the recyclability of tetra-paks as they consist of different thin layers of paper and plastic and metal. It's difficult for recyclers to separate these components, so the recyclability of these materials should be a big question mark. I think they should be considered as "less recyclable" than paper or glass, and even other plastics.
There is some hope of bio-plastics coming soon, but that's not what's generally being used today. These bio-plastics can be food-safe and are bio-degradable, so that would be a boon.
Cardboard and plastic produce toxic pollution in their construction. All take energy to be produced, although plastics take the least. All can be produced from recycled materials, but in reality food-safe plastics are made only from new plastics. Plastics have limited recyclability, due to poor separation between types of plastic and limited number of times it can be recycled (about 3-4 cycles). Tetra-paks are considered worse for recyclability than plastics.
Plastics tend to win in terms of production and shipping, but plastic is less reyclable than the other materials, and tetra-paks even worse. Plastics become unrecyclable waste even when they are recycled since they break down in the process. Worse, their waste pollutes the food chain. Plastics/tetra-paks lose in my evaluation if eco-friendliness for these reasons, and hence so does the wine box. If in the future the tetra-pak were made from bio-plastics that can decompose without toxic pollution, I think then the wine box would win over the glass bottle.
One option not mentioned here: wine can come in kegs now. :) You can get it served as draught in some places (wineries, bars). These kegs are reusable, which is always better than recycling.
References:
(Some of these references might have conflicts of interest, so add salt.)