How do you think the sound quality of musical instruments like say, the piano where differentiated and classed back in the day? or take it one step back even to how they were designed and constructed. No blind tests, no room measurement software - it was completely done by ear.
Broadly, musical instrument improvements have come through necessitated advancements to fix mechanical problems and issues, especially on the piano, as much as anything else. With the exception of the very latest integration of digital capabilities piano construction has changed very little since around 1900. Some experimentation has happened of course but fundamentally what are generally regarded as the very best pianos today have changed little in over 100 years.
But musical instruments and RC in amps (or amps in general) are completely different things and are aimed at a fundamentally different purpose. Musical instruments are designed with the purpose of creating a unique and different musical timbre and are subject to a huge number of input variables. This means that subjective listening becomes far more important and relevant as you are not looking for a Bossendorfer to sound exactly like anything else you're trying to stamp your own unique sound on it by design. It then does absolutely come down to subjective preference whether you like Bossendorfer over Steinway because you're not re-creating sounds, you're creating them from scratch.
RC and amps are designed (or are they - this is a another discussion altogether) to replicate or re-create a recording of something like a piano. You don't generate new sound, you replicate sounds generated on something else so the input never changes. By that I mean a recording doesn't change from play to play and the goal most people aim for is for it to sound exactly how the artist intended it to sound (which in theory is exactly what is down on the recording which is never changing). So you're looking to exactly replicate what that Bossendorfer sounded like in the studio, you're not trying to make the Bossendorfer sound like a Steinway.
So the outcome is a professional performer on a Bossendorfer SHOULD sound different to the same professional playing the exact same piece on a Steinway. Conversley a Miles Davis recording played on a MkIntosh should sound exactly the same as on a Lyndorf all other things being equal. Their job is to faithfully replicate the engineers/producers/performers recording without colouring or changing the sound. Now I appreciate that this isn't what everyone believes. Some people like their equipment to colour or "flavour" the sound but as a whole I think most people are aiming for transparency in their amps/systems.
The result is blind testing for musical instruments would offer little benefit as they're designed to be different and within reason it comes down to taste, especially within the same class of instrument (mostly instruments are classed as either student, intermediate or professional with a price tag to match). Sure you might benefit from blind testing if your purpose is to be more objective but as they've not designed to all sound the same (or rather to exactly reproduce the exact same unchanging thing) it becomes less important.
But even in musical instruments, the ears can still deceive. In the hands of a professional, few if any can truly differentiate between a Stradivarius and a much cheaper modern violin if they don't know which is which ahead of time. So even though conventional wisdom is that a Strad is way better than a modern top end violin in reality few, if any can tell the difference. To illustrate that there was a violinist a few years ago that went out on stage in a solo concert and performed absolutely wonderfully with a fantastic full rich sound. The audience gave him a standing ovation and the consensus was it was one of the best performances ever. At the end of the applause he smashed up the violin (what the audience and orchestra thought was a multi million pound instrument) only to find that the instrument was a student model worth a few hundred pounds he'd bought the day before. His point was more to highlight the hypochracy and snobbery in music of course but the point still stands that conventional wisdom counts for little and can negatively taint our perception and opinion.
Many would argue that stereo testing is a much harsher critic of any RC as it's a more focused test, even mono and single speaker mono testing has been used in blind tests. What kind of objective proof are you looking for exactly?
Do you mean a graph? If so, how do you intend to interpret this - you're going to have to take the word of someone saying 'because I say it is' because it can be open to interpretation as not all things that appear bad or good are actually bad or good, it's just not as simple as that - much to the dismay of many Arcam users.
It depends on what you want to test.
In this instance, I would be looking for comparative testing that pits x number of systems against each other to find out which sounds better to those listening. In order for the tests to give a fair and true result, the listeners and testers can't know what they're listening (double blind) and the systems would ideally be configured by the manufacturers to optimise each system to their strength. Theres a lot more than just this but that gives you a flavour.
With all that in place if ARC truly is "better" than DIRAC then it should be clearly statistically apparent that more people prefer it. Over time and with a few more similar tests you'll get an indication of what sounds generally better. That's all there is to it.
Now I do think the test above was certainly indicative and certainly part of the answer but there are a number of problems with that which might have made the results less definitive. (which was touched on by one of the listeners)
- Listener fatigue (it was a long day and listener fatigue was not accounted for)
- Product setup (this is in no way at all suggesting they were not optimally configured but there was clearly unfamiliarity/difficulty with some of the kit)
- Listener bias (some of the listeners already had pre-conceived opinions that were not mitigated through double blind testing)
- Material choice (I don't know what was chosen but clip familiarity and personal taste plays a part)
I'm not in any way criticising what was done - it was never meant to be accurate or scientific in its goals. It was clearly a great day and a bit of fun where some effort was made to reduce the more obvious bias. But because of the issues you also can't put much weight behind the results as there's just too many things that could have tainted or influenced the results.
G