The shuttle was supposed to make LEO (Low Earth Orbit) more affordable. In that regard it was a complete failure. The engines were not really reusable since they must be taken apart and rebuilt after every launch. Since almost any major failure will kill the crew, safety and reliability matter more than economics. (Of course, most of this was not realized until after the design was completed.)
The X-33 project was doomed from the start. "Most technologically challenging" is simply another way to say "least likely to work." In any case, a single stage to orbit is neither economically nor technologically effective. It should be noted that the Ares I and V represent a step back with only the solid fuel boosters being reusable.
The essential problem is that since the fuel makes up over 90% of the total mass of the vehicle (at launch), the minimum thrust needed to get off the ground would produce at least a 10G acceleration before orbit is achieved, putting an excessive strain on the structure and payload. Instead one must reduce the thrust by either shutting engines off, thottling down or using multiple stages.
A number of technological issues can be pursued independently of a specific design.
First is to design a truly reusable engine. From the V2 onward designs have used LOX and/or LH2 as coolant to keep the engines from melting. However, the thermal stress from this approach is severe.
Turbo-pump design is another area for research. Older rockets used catalyzed hydrogen-peroxide to power the turbo-pumps. The shuttle uses what amounts to two jet engines, one which burns hydrogen in a LOX environment and one which burns oxygen in a LH2 environment. Both produce a mixture of fuel and steam (plus small amounts of water droplets and ice which harm the turbines). The Ares, if I understand it correctly, uses compressed helium. While this is even safer than hydrogen-peroxide, helium is rather expensive.
The best power source for the turbo-pumps is still the rocket fuel itself. The trick is to find a way to handle the resulting temperature extremes. One could even use the waste heat off the rocket engine to power a turbine. Presumably this has not been tried or didn't work out due to the different requirements for power generation and engine cooling.
If there simply isn't enough waste heat produced, one could design a separate rocket/heat exchanger to maximize mechanical power production. Basically this would consist of a number of long pipes carrying rocket exhaust (steam) passing through an LH2 boiler. The exhaust could then be vented into the main combustion chamber where it would be reheated and used for additional reaction mass.
If the problem is insufficient cooling (to prevent the rocket from melting) then the solution is to use better materials. One should expect an increase in weight for a reusable engine.
One way to reduce the thermal stress is to use a closed cycle heat engine with helium as the working fluid (since it is light, non-toxic, non-flammable, and remains a gas over then entire temperature range). The high pressure gas would be heated by the rocket and the low pressure gas would be cooled by the LOX and LH2 (in parallel). The turbine could then drive the turbo-pumps and provide electric power (it already has/needs a starter motor).
One of the best locations for the LOX and LH2 heat exchangers is inside the turbo-pumps where the motion of the fluid would reduce the needed surface area. In addition, one could use centrifugal force to separate gas from liquid, since the liquid is both colder and much more dense than the gas.
Another area for research is in the use of composite materials for fuel tanks. Instead of foam insulation could one use a true Dewer with either aerogel or a geodesic framework in the vacuum gap.
The greater bulk of fuel is consumed simply getting the vehicle out of the atmosphere, before the velocity reaches the point where heat shielding is needed for re-entry. This also puts the re-entry point somewhere over the Atlantic Ocean, making a winged landing somewhat pointless. (Parachutes would mass less anyway.) One simply needs a large enough sea vessel (possibly a catamaran) to recover and return the used boosters.
It should be noted that every launch vehicle in our inventory now uses strap-on solid fuel boosters. While they lack specific impulse as compared to a liquid fueled boosters, they produce a large amount of thrust at the time it is most needed. It turns out that the cost for reusing solid fuel boosters is about the same as the cost of replacing them, but they represent a cheap and easy way to increase the payload capability of any launch system.
A purely solid fuel first stage would cut off long before one would need to based purely on re-entry requirements, placing a greater percentage of the needed delta V on the second stage. The optimal trade-off between solid and liquid fueled boosters will vary with technology and payload.
Because the second stage reaches (near) orbital velocity, it must either be discarded or require extensive heat sheilding and possibly wings for re-entry. In either case it would be better to carry the payload externally in order to minimize the size of the vehicle. For a manned payload, this also improves safety (especially when including a solid fuel escape motor).
The simplest approach would be to use the Ares I second stage. However, the ultimate goal would be a reusable design which would re-enter after one orbit, somewhere over the Gulf of Mexico, and hopefully fly all the way back to the Cape.
One approach would be to replace the outer skin after each mission. A blunt heat shield on the nose and an inflated cone of thermal blankets at the rear would suffer the brunt of the heat and drag. The final landing would use parachutes.
NASA once experimented with magnetic shielding for re-entry vehicles but never really pursued it. One interesting design uses two magnetic booms/probes on the ends of relatively stubby wings (more of a lifting body). These probes would extend beyond the front of the stage. The tips of the probes are magnetic poles, and the ions from the shock wavefront will be deflected away from the probes themselves. The body of the stage would ride behind the shock waves from the probes.
In addition, by directing positive ions downward and electrons upward there is a net lift generated (due to the difference in mass). The skin of the vehicle needs to be able to store an electric charge, and this only works during the early portion of the re-entry. This design would also need parachutes.
Of course one could use a design similar to the existing shuttle (replacing the crew and cargo with fuel tanks). I'm partial to a design which uses a downward dihedral delta wing. At high speed it traps air between the wings, generating compression lift. When the flaps are lowered it would turn into a ground-effect wing. The goal is to remain in the thinnest part of the atmosphere until well below Mach 5. The main problem is the asymetrical mass and drag during launch.
The only reasonable location for the payload is on the nose of the second stage. (The Soviet Buran design makes no sense whatsoever, except to look like the shuttle.) It makes control easier and will avoid damage from ice debris. There will also be less damage if the first or second stages fail. The only problem is that one may need a lengthy adaptor between the payload and the nose of the second stage.
For a manned payload, it should be noted that the tradeoff between a winged re-entry vehicle and a blunt capsule depends on the size of the vehicle. To justify a shuttle-like design would require combining cargo and crew (or a LOT of passengers), whereas from a safety perspective or mass limit it might be better to use separate missions for cargo and personel.