Dear Junior
There are many complex things that need to be overcome to create a system, but in my experience there is one thing that overshadow them all: to gather and create enough knowledge to create a system that not only "works (kind of)", but actually works - i e help the enterprise to get things done in an unobtrusive and natural way.
In most systems I have worked on this, this aspect is far more challenging than technical issues like getting quick enough response time for persistence, or capacity to support enough "simultaneous users" (whatever that is). In essence, to really understand the problem we are solving is the critical complexity to solve for success. Of course, this is basically what Domain Driven Design is about.
The goal is a really tough one, to create a system that will handle any case that comes in its way, without manual involvement. Well, some cases might be delegated for manual handling, but still - the decision to delegate a specific to a human must be done automatically.
I like to compare with the challenge of riding a bike. What we are trying to do is not so much to ride a bike. We are trying to construct a bike-riding robot.
If the goal is a tough one, the starting point is the weaker. The people around are business people, who know not much about system development; and system developers who not know much about the business at hand. So, there are no single person with enough competence to solve the problem. Hence, we need to talk.
Even worse, not even if we gather the total competence of everybody involved will we have covered the competence we need. Again compare with bike riding. Get a pro BMX-rider and a robot engineer into a room; not even together they know how to build a bike-riding robot. Hence, it is not enough to talk. We have to depart onto a knowledge creating journey together.
I have mainly been working on information systems (in contrast to e g games or automation). In these systems, what we need to do is to create a model of the domain. Of course, not the entire domain, but a part of it that constitutes the essence of the tasks we want to support. But the model of that limited domain must be very precise. In fact it must be free from all the ambiguities and vagueness that is acceptable if we "just" work in the domain - it must be precise enough to construct a computer program.
To succeed , there are three types of knowledge that we need to improve, by learning or investigation. . One is well known and obvious, the second is less well known and often overlooked, and the third is almost never discussed explicitly.
First the programmers need to learn about the domain. This is pretty well understood and I have often heard claimed that "the programmers do not understand the business". This is totally correct, as most programmers have been working in other fields and thus do not know the ins and outs of this precise problem. Compare with the robot expert - she will not necessarily know much about bicycling. The good part is that in learning the basics and overview of the domain, they can learn a lot by talking to the domain experts. The bad part ... well, I will get back to that part.
Second, and often overlooked, is that the domain experts need to learn some modeling. This is basically for communication. When modeling the problem we need to get precise - dead to the point precise. Unfortunately domain experts are not used to this, and "works for most cases" or "more or less" are not phrases that helps very much when creating automatic systems. Compare with the BMX pro, they will probably need to learn some things about robots. Do not get me wrong - I do not want to make modelers out of domain experts. I just want them to understand a little about modeling. I guess I am after the same difference as between reading and writing. The good part is that there are expert modelers around from whom they can learn. The bad part ... well, I will get back to that.
The third and final part of the missing knowledge jigsaw puzzle pieces, is unfortunately the combination of the two "bad parts". This is the knowledge that neither part have when starting out. This is the knowledge that we cannot learn from someone else. This is the knowledge we have to create together. This knowledge is the precise modeling of our domain problem.
Again comparing with the robot team: this is the knowledge that enable us to make a robot to make a left turn without falling flat to the ground. On one hand, the robot engineers knows a lot about fast-feedback-systems, but not about bike-turning. On the other hand, the BMX pros can make a bike under very harsh conditions, but they probably do this on an intuitive level without knowing the precise mechanical details. But the depth of knowledge needed is not at a "do"-level ("apply" in Blooms taxonomy), it is needed at "analyze"-level. And at that level, most people do not know how to ride a bike. Telling the rider to "follow the bike" might help a human, but will certainly not help when constructing a robot.
The tough journey is that the robot engineers and BMX pros have to figure out what exactly happens when you make a turn on a bike. They will probably film bike-riders in different situations, try to make some explanation and theory; from that theory build some simple prototype, test it, discard it, start over, make a new theory, fail, simplify, prototype, build, succeed, trying a harder case, fail, back down, redo, try again etc in iterative investigation to build some model of what happens when turing. Finally they might have made a model that holds for having the robot ride a bike through a simple slalom path on flat surface. From there on, they can try doing something a little more complicated, like driving over a small bump.
In the same way programmers and domain experts must interact to find a simple case to start with, and try to model it sufficiently well. Once they have managed that, they can expand to cover interesting cases the first model could not handle. While doing this they will probably have to revise, or trash, parts or all of the first model. But iteratively, and incrementally they will learn (or invent) new ways of thinking about the domain. These are ways of thinking that neither the domain experts nor the programmers had thought about before they started working together.
This is very far from "lets ask the expert", as the domain experts do not have a "ready-baked" model inside their heads. It is also very far from "lets just build what we think they want", as the critical problem is not technical - it is about creating knowledge.
The bottom line is that the most critical activity is to make the domain experts and developers work in cooperation to create the model. To do this we need to mutually learn from each other. But also, and more important, we must create new knowledge as a joint effort, knowledge that did not exist when we started. If that knowledge journey fail, we do not end up with a system that actually works and is fit for purpose.
Facilitating that joint knowledge journey is the key factor to create systems.
Yours
Dan