We can divide the analyst community between statistical analysts, like many of those that work for IDC, gathering quantitative information about which products or services are being used and where, and technical analysts whose role is to understand the technology.

I’ve never worked on the statistical side (although I have been trained in statistics so I could do that kind of work at a push). I have gathered user information, but only during quality research, which involves long interviews with end users. So the research I’m best qualified to discuss is technical research.

Technical Research

Research is usually done with a specific goal in mind. You are going to produce a report on IT Security, so you need to do the research before putting pen to paper. You are doing product selection work for some customer, so you need to be up-to-date in that product area. You have to regularly produce a Magic Quadrant, etc.

In such circumstances the research that you do is goal-directed and the activity corresponds accordingly. You talk to the vendors and you talk to the users - but you don’t talk to large numbers of users. If you can, you talk to technical users. They usually know what they’re talking about and they are reliable.

Something that I regard as critically important to technical research is that an analyst has experience at the coal face - even if it was quite a while ago. (there’s a great deal of talk about technology change, but mostly everything out there is like it always was, except it’s wearing a new hat and coat and it carries a new umbrella). If an analyst doesn’t have such a background then they will never truly understand the technology they are supposed to be experts on.

Such analysts are simply reporters, only ever capable of reporting what has been told to them. And they may be very good reporters, taking great care to aggregate all the information they’ve gathered and weight it in a meaningful way, but they are reporters nonetheless.

Architectural Modeling

Those with technical experience can arrive at their own conclusions and they are in a position to be able to identify poor technology from a technical perspective. I remember once being asked by a German IT user, who had a comparative database report I’d written in his hand, how much testing of the products I’d done. I told him the truth; none. (No point in lying). I’d talked to users of the products, but actually only a little. But I knew the products very well.

Here’s how:

It simply is not possible to do a detailed trial of multiple products, because you need a team and it costs an absolute fortune. Even doing a limited trial of a small number of products costs a lot more than you’d ever get from publishing the results. And even if you could say with certainty that one product was, let us say, faster than another, that won’t necessarily be true when the next release of both products appears. And you wouldn’t necessarily spot that there were serious bugs in one or two of the products, because such bugs usually emerge in production rather than benchmark environments.

But you can analyze products from an engineering perspective, by talking to the product developers in depth about how the product works. You can achieve surprising accuracy by doing that - if you understand software engineering. You can also determine whether something that a vendor is presenting as a great advance is simply marketecture.

That’s how I do technical research. I have a set of conceptual models that I’ve build and tested over the years that are based on first principle views of the way IT works. How do I test the models? When I’m with a skilled engineer, I draw diagrams and see if the engineer agrees that the picture matches the technical reality. When I get disagreement I revisit the model and refine it if I can. That’s how I came up with the diagram on which “Service Oriented Architecture for Dummies” was based.

Incidentally, I don’t attempt to keep these models secret. I use them in presentations to users and sometimes in white papers or in reports. I will be publishing all these models over time on this blog. I’m not the only analyst that works this way, but I may be unusual in the number of technical areas that I cover that way. I’ve been told that Gideon Gartner worked that way when he was more involved in the analyst business.

Note: This posting is one in a series of postings that deals with the topic of dealing with analysts. Click here for links to other postings in the series.

  Subscribe to HaveMacWillBlog in a reader