A starting point here is how many steps do we routinely find in one process - would 1,000 be a practical limit?
If you then say each step is likely to have 50 data points across a few forms but that only a few of the 1,000 steps are current at the same time, we have manageable data flow.
However, one process can have hundreds/thousands of active instances, but no one user wants more than 50-100 pending tasks in their "today" InTray.
But, a hospital or an insurance claim processing or a manufacturing entity can easily have 200 processes.
No matter . . . - with a good run-time environment, data appends to an MS SQL data base are fast and an IIS front end with reasonable horsepower can handle many simultaneous users.
In addition to logged-in userss, you can have hundreds of local and remote systems and apps posting data to a data exchanger with an engine posting the incoming data to the right relational dbms records.
We are handling business transactions here, right, not live data from a satellite launch?
Does anyone have better numbers than the above?
If the BPM is implemented following a strategic approach (what objectives? what processes? which data?), then data and other elements will not be too much nor too less but will be exactly what the system requires. Otherwise, if the BPM is implemented at the operational level without a proper alignment with the business strategy, then this probably will lead to redundant and unnecessary information.
Start with focusing on effectiveness first (doing the right things) and only thereafter on efficiency (doing the things right). That way you avoid creating stuff you shouldn't create in the first place... On top of that, think twice about what is *really* needed; it is nowadays just too easy to incorporate everything (which in itself is actually a cause of broken processes in the first place; KPI's, roles/means and structure for instance should always be derived from processes and not the other way around).
Well scope is one way to control it, but there is a bigger issue here. Like IoT which will get overwhelmed as well, there is so much data needed to figure patterns, contexts, proper decisions and proper actions. Without embracing cognitive assists, both people and software (in this case BPM software) will get overwhelmed. Just my two cents
The problem of process users being overwhelmed by too much data is a both an technical problem and an economic problem.
cheap sensors => cheap data => information overload
The answer is to fund and perform the work of business analysis required to define important events and to filter signals all for the purpose of driving decisions.
There is a huge governance issue here though; look at the persistence of the alarm fatigue problem, especially in health care. There's a use case for managing alarm fatigue, but the business case is harder to make -- and no one wants to make an investment.
[Note: the focus here is primarily on the IoT ("Internet of Things"). And it's true that BPM technology-managed processes are just part of the whole information overload ecosystem. But insofar as BPM + rules + analytics are the primary tools of instantiation of the work of business analysis, the juxtaposition of BPM and information overload is appropriate.]
Here are previous BPM.com items on this topic:
New business process and the Internet of Things -- Alarm fatigue, healthcare, IoT, decisioning
Mistake to avoid when automating with BPM -- Economics of information overload
Technologies making biggest impact on BPM in future -- IoT and semantic overload
And my LinkedIn post on alarm fatigue, from 2014:
I like Walter's take - it's about the goals we set ourselves during the process. This is what we should be measuring and refining.
I guess by now everyone knows how much I believe in the cognitive impairment theory, how this is a big limit in everything we humans do, speaking from an evolutionary perspective:
1/ our neocortex is a small, very young area of our brain. This small portion can only handle small buffers of information, as short-term memory is usually under considerable strain when processing info. Multi-tasking is a myth, and it is actually single tasking done relatively fast.
2/ our archaic limbic amygdala is in charge with paying attention to bad news all the time, readily interrupting our concentration and focus.
The combination of the two cannot make for a very consistent rational set of behaviors in humans.
I'd actually like to extend my cognitive impairment examples to analytics visuals (as this is a current area of concern as we're debating internally to which extent should we grow our in-house analytics engine). I look at all the embeddable BI suites out there and I literally see hundreds of possible graphs readily available. Seriously? Who uses all this crap? Who has the time to even understand the data preparation requirements of them?
I honestly cannot think of business scenarios that can be more conveniently served than by the use of line charts, scatterplots, histograms, bullet charts and... that's just it. No pie charts, no gauge charts, no stock hi-lo's etc. The purpose of graphs is to quickly make a point - if you have to first understand what the hell have you just represented there... you lost.
I think that, before we even start thinking about using cognitive assists, we should focus on making clear points in our existing graphs and tables.
Another point that maybe we should discuss (but I'm way over my quota already) is what do we do with irrelevant, obsolete data. This is directly related to fixing our cognitive limitations: what is relevant towards the decision-making process? who gets to call the "D" in CRUD? why that data, why then?
I believe the best way to not be overwhelemed by the amount of data when thinking about process is to embrace "separation of concern" and decouple the two concepts.
When you decouple the definition of your data from the definition of your process, you can more easily adapt them. Modeling the data (defining the attributes about whatever you are trying to process) becomes very easy when you are not bogged down by the process that will be applied to it.
Think about how you could track your Customers in a process. The customer attributes (name, address, company, etc.) have little to do with the actual process that you want to apply to the Customer. You may have a correspondence process to manage Customers that are a part of your sales pipeline. If that process is decoupled, you are able to refine it without affecting the attributes on the Customer. With that level of separation, you could even have multiple processes linked to the same Customer Model.
So, my recommendation is to decouple the definitions of your data and process and it won't matter if you have 10 records or 10 million records. That allows you to stay focused on building the most efficient process possible for your business.
Down here in the depths of the forum page, where the the accumulated weight of wisdom and punditry of the earlier posts produces a sustained ambient pressure that can be hard on the ears, we sometimes think a little differently. To wit:
If BPM is overloaded with data, it's because BPM's purpose is not to process that data, but rather to act on it.
IoT, Big Data, whatever: all of it boils down to waves of data threatening to swamp your application. I respectfully submit that the problem of analyzing, correlating, and interpreting that data is, in fact, distinct from the set of problems that BPM is meant to address. Lots of smart people are creating lots of amazing tools for channeling torrents of data into manageable streams of actionable information.
BPM is at its best when provided with meaningful and timely information, which it then uses to drive decisions based on the rules and goals set by your organization. Perhaps it would be best if we left the job of distilling that information to a technology better suited for that task.
I am with Jim "Without embracing cognitive assists, both people and software (in this case BPM software) will get overwhelmed." Fortunately, the solution is straighforward - the process template (which is a formal plan for the coordination of activities) is such a "cognitive assist". It should be possible to change the template, use the collected data to simulate the behavior of the process (i.e. achievement of its goals) and take an objective decision about usefulness of that change.
Any experiment in the high energy physics is based on a model (which is an approximation of the laws of nature) which are checked vs data collected by the experiment. Sometimes the model fits the data, sometimes the model can't describe the data and sometimes collected data are wrong (e.g. because of some errors in the acquisition system).
Fortunately, in BPM, the model (i.e. process template) is explicit thus can be simulated and optimized without rocket science.
The focus of BPM will result in delivery of one version of the truth with elimination of duplication as applied to day to day work. However as machines created large amounts of new data there is a need to somehow structure into being useable by the business. I would suggest that BPM could be applied to addressing this issue by applying automation to segment the data as required. Supporting software for BPM must include application of rules and ability to manipulate data including algorithms if required. The game is to ensure delivery of the right data to users for their specific purpose in useable format. Where data falls outside expected parameters alerts escalation etc will trigger automatically. Such an approach should avoid data overwhelming the system....?
I somehow take issue with the accredited opinion that process and data should be modeled with different tools, by differently qualified people. And yes, everybody still does that, including me. Although I try as much as possible to bring a friendly data model in front of the customer, this is painful. People get tears in their eyes when they start hearing about cardinality relationships.
Aren't we supposed to solve business problems? Is any normal business person (i.e. not Fortune 500 CIO's with tens of years of technology background) able to separate the data-driven problems from workflow-driven problems?
IMHO, this pedantic separation is one of the key roadblocks to BPM adoption. I believe further adoption of BPM lays in the fringes of the business world, in the small, unassuming companies that cannot afford to even think architecturally but still want to get involved in the design of their operations. Will there be a robust hybrid architectural approach that will blur this religious divide between data and process?
Maybe the Bene Gesserits of the BPM Empire have realized their full potential and we must turn our attention to the Honored Matres lurking out there in the BPM Scattering?