A New Framework
In the May 2012 issue of JAMIA, the authors McCoy, Waitman, Lewis et al propose a framework for evaluating the appropriateness of CDS alerts and the provider responses to those alerts. The type of evaluation they propose is labor-intensive because it requires experts to judge not only whether each alert displayed was appropriate but also whether the provider’s actual response to the alert was appropriate. In this model, alerts are considered appropriate when they are clinically relevant. Provider responses are considered appropriate when an expert reviewer judges the provider’s action to have potential clinical benefit. Note that overriding an alert may or may not be an appropriate provider response, depending on whether the alert itself was appropriate or inappropriate in the first place.

This latter point is quite important. We often hear about high alert override rates, but overrides are not always the appropriate provider response. In the authors’ model, overrides are considered justifiable when the alert is inappropriate, but overrides are considered provider non-adherence when the alert is an appropriate one.

The authors suggest using this 2×2 table as a framework:

The authors then define the alert override rate as the sum of provider non-adherence (B) and justifiable overrides (C) over total alerts, or (B) + (C) / (A) + (B) + (C) + (D).

Validation Example
In a validation example the authors provided, 400 out of 487 alerts (82%) were initially overridden, although after a more detailed review of the events, including looking at actions that took place after the immediate ordering session, the number of overrides was reduced to 162 out of 487 (33%). This finding supports the notion that alerts may be useful even if they are initially overridden because the management change may occur after the immediate ordering session ends. For example, the provider may consult a colleague or a reference before deciding to take some action, but the alert, having at least raised a good question, was therefore still useful.

Also of note in the validation example was that of the 162 overrides that occurred even after looking at actions that took place following the immediate ordering session, 74 (46%) resulted from provider non-adherence. In other words, only about half the overrides were in response to inappropriate alerts; the other half were actually in response to alerts that were judged to be appropriate. This level of provider non-adherence illustrates the problem we have with alert fatigue – that is, providers are in fact overriding a fair number of appropriate alerts. The challenge therefore is to come up with ways to reduce the number of inappropriate alerts such that when providers receive an alert, they can have greater confidence in its appropriateness, and therefore, they may be more likely to adhere to its recommendations.

 Not only are accounting association conferences a way to catch up on the latest trends in the industry and gather CPE, but they are a way to meet new people and develop your professional network with a handshake and a power lunch instead of on LinkedIn.

For accountants who have small firms or who work at home, having a support network is vital. According the The Business Insider the home office is becoming the norm rather than an exception.

Typical topics this year are technology, the cloud, mobility and social media. Instead of getting the nuts and bolts information you can research for yourself, you can discover how firms like your own are implementing solutions and gaining new clients and realization.

Most associations have a conference or several conferences. Take for example BDO , The Leading Edge Alliance or CPA America. These associations have been formed specifically to share resources and encourage collaboration amongst their members, so networking is the main purpose.

Some associations are highly specialized, such as the CPA Auto Dealer Consultants Association (CADCA) and the Healthcare Financial Management Association (HFMA). The CCH User Conference gives software users a chance to exchange tips and tricks. Your state society usually has a conference which is great to meet and network with local accountants, even if it’s not in an exotic location.

When attending an out-of-state conference, share and collaborate in real-time; then continue with your contacts via Skype, Facetime, Twitter and good old-fashioned email.

Here’s what I think a hybrid content-software product is. It is a product in which software and content are both equal inputs. Predictable processes with unpredictable outcomes, depending on the input, will be encoded into software. Content relevant to each workflow step will be a mere click away for the customer to access for help in determining what input to apply to give that workflow step. Content will help a customer enter an amount (maybe a tax rate, a stock price) answer true or false, yes or no to a question, or even select (a), (b) , (c) or (d) or even “all of the above” in a multiple choice selection. Getting to the content from the software program will not require opening a separate browser window and/or using ALT+TAB to move from window to window. Instead, the content will be fully integrated – as an organic input – into the software experience. This experience goes far beyond mere linking of content and software.
There will likely be a single brand name for the entire hybrid content-software experience. And there will probably be a single price for the organic experience. Content will automatically be updated – accessible from the cloud, of course. Though in a mobile device, there likely will be contextually relevant content stored on the device itself. At first, the content will likely be highly relevant but repurposed legacy content assets – those primary sources of law from government websites and secondary sources that provide robust explanations of their interrelationships. But I foresee a day when the content is written expressly for the software experience.

Professional customers’ needs for efficiency are increasing. Clients demand higher volumes of work at lower cost yet they expect the same levels of quality. There will be ever greater pressure to solve problems through software supplemented by content. Questions with less predictable answers than tax calculations – for which there can be a “likely” or “probably right” answer – will increasingly be expressed in software. And professionally edited content that can help the professional get ever closer to the “probably right” answer will be an essential component of that software experience. Again, it’s a hybrid content-software experience.

Of course, standards, procedures, tests, rates, calculations and other processes designated by regulations, legislation and cases change over time. Hybrid content-software products will need to be built on a flexible architecture to enable rapid updating of the software code to enable rapid updating of the software. The need for classic research in a Web browser – Boolean and natural-language queries run across terabytes of digitized print with robust hypertext linking will likely never go away for tax, legal and regulatory professionals. There always will be complicated factual situations and highly nuanced tax and legal theories that cannot be encoded into software. But where code is possible, I foresee it happening.

These are some of my thoughts. What do you think?

Expert reviews (sometimes called heuristic reviews) are a discount user centered design method. One or more reviewers exercise a user interface (alone, at first) in order to note usability issues that are likely to occur. They may do this with a set of heuristics or principles in mind, it may involve a checklist, and often persona-based tasks or specific sections of the interface are assigned to provide a consistent way for reviewers to experience the interface.

Reviewer input is then combined into a list of issues, questions, and bugs. Each is resolved and prioritized with the larger design & development team, and a holistic solution considering all issues is visualized through wire frame iterations or other simulation method.

But expert reviews are much more than that.

Expert reviews are a way for teams to learn together, and while doing so, become better teams. They are a way for “new bees” to get up to speed on a product quickly and with focus, for veterans to share experience and vision, and a way for project managers to communicate design decisions across projects and groups. They offer a chance to look at the interface as a whole, rather than a feature set or group of change requests.

During a review, problems are noted when the system characteristics violate constraints known to influence usability. What qualifies as a violation, and more importantly, what qualifies as a rule is often at the heart of any discussion about usability.

Expert reviews facilitate this continuous discussion of standards and guidelines against which violations are judged. They provide a history of design decisions, a common language to discuss them, a process for disagreement and resolution, and they can easily be time-boxed. In addition, they identify good design decisions and patterns to repeat, and allow us to see things from different points of view. For many UX teams, they are an important on boarding and training tool.

UX inspection methods like expert reviews increase testing efficiency by identifying “low-hanging fruit issues”, and by focusing research on difficult problem areas. In truth, they are never really “complete”, but they increase the confidence level in the solution architecture, and can be performed at any time during the design process by many different roles, led by UX.

Often the real answers to the problems can only be determined by observing customers while they use a product. However, expert reviews complement user testing, and help team members see things not only from the users’ point of view, but also from each other’s.

According to a recently published Wolters Kluwer Health Law Blog article by Harold Bishop, The Infectious Disease Society of America (IDSA), the European Centre for Disease Prevention and Control (ECDC), the World Health Organization (WHO), and the U.S. Congress believe there is a global infectious disease crisis currently underway. In the United States Congress is considering bipartisan legislation known as the Generating Antibiotic Incentives Now Act, or the GAIN Act, which would streamline the FDA review process and incentivize antibiotic development by extending the exclusivity period for a new “qualified infectious disease product” by five years and another six months if the antibiotic manufacturer identifies a companion diagnostic test for the qualifying pathogen.

As recently as March 23, 2011, Reuters Health Information, as reported by Medscape Today, stated that the WHO predicts more than 2 million people will contract a drug-resistant form of tuberculosis (TB) by 2015. According to the Annual Epidemiological Report 2011 from the European Centre for Disease Prevention and Control (ECDC), TB remains a common infection in Europe, with nearly 80,000 cases annually.

These concerning statistics have the entire world concerned about antiobitic use and development. According the article by Mr. Bishop, nanotechnoly might be the answer:

For some time, the Defense Advance Research Projects Agency (DARPA) and their partners have been trying to develop nanomachines that combat bacteria and all pathogens like viruses and other potential biological weapons. As reported by Steven Salzberg in Genomics, Evolution, and PsuedoScience, a group of scientists published results in The Lancet of their success in treating 4 rhesus monkeys infected with a dose of Ebola 30,000 times the normal fatal dose.

According to Salzberg, Thomas Geisbert of Boston University and scientists from Tekmira Pharmaceuticals and the U.S. Army Medical Research Institute for Infectious Diseases (USAMRIID) designed and synthesized “small interfering RNA” (siRNA) that would stick to the Ebola genes, but not human genes. The siRNAs were delivered by insertion into nanoparticles that were only 81-85 nanometers across. The siRNA treatments were injected 30 minutes after infecting the monkeys, and again each day for 6 days. All the monkeys survived without long-term effects.

The future of nanotechnology in medicine is an interesting one, scientists are already using it to detect harmful bacteria in humans while a recent Chinese study indicates that nanotechnolgy use in water filtration is leading to the current antibiotic resistance crisis. Whether nanotechnology plays a part in solving this crisis remains to be seen, but the possibilities seem as endless as the superbugs that are emerging around the globe.

The conference widened the scope this year and invited quite a number of speakers from the US, Canada and the UK, which was very fruitful for the discussions going on.

I think that three different areas were in the center of the presentations and workshops:

• The general transformation process of publishing houses with regard to the rapidly changing user behavior and the user expectations (“Digital natives”)
• The transformation process from a (print) product centric view to a content centric view and the accompanying challenges around metadata and content enrichment, context and discoverability of content
• The rapid growth of mobile applications, mainly in the area of tablet PCs, but also around smartphones

The first topic was very nicely presented by the opening keynote speaker Brian O’Leary. It clearly showed how fundamental the change process within our organizations must be. Brian also pointed out the importance of context on different levels, which we must preserve and take into account in all our electronic products.

The second main topic had a technical focus, expressed e.g. in the presentation of Andrew Jordan from Thomson Reuters, where he explained that the experience and data that was collected from Open Calais in the past is now used also internally to solve complicated metadata issues like the disambiguation of people or events. For me, the main message in this area of interest was that the technology has in the meantime reached the required level of maturity, so that it can tremendously support the content transformation process we envision today.

Concerning the third topic, I had the impression that the focus was still on convincing people that the challenges around mobile applications will not go away. The presentation of Gregor Waller was full of facts and figures, which proved that we are in the middle of a digital storm and that newspaper publishers were the first to feel it – often in a very painful way. I hope that next year we will more talk about solutions and not so much about whether it will affect us or not – because it will definitely affect us.

I was a bit surprised that a lot of publishing houses, especially fiction publishers, are not really prepared for the challenges we are already facing now. Real innovative products were hardly there; Jurion, which I presented, was one of the very few exceptions. Therefore we had a very lively and intense discussion during the workshop, which followed my presentation at the plenary.

But researchers increasingly conduct searches and browse content not only in professional research products but also on government websites and public search engines. Tax, legal and regulatory professionals increasingly use software applications to respond to their clients’ needs. Software and free and for-fee content are all part of professionals’ workflows. Shouldn’t research folders not follow that workflow?

Today’s professional workflows involve teams. Law firms do work on behalf of corporate legal departments. Lawyers and accountants from different practice groups - labor, tax, securities, etc., – work together on behalf on a single client. They share research. Of course, their research activities involve the Web. And they use software. Wouldn’t it be nice if the research folder also became a collaboration folder?

I think it’s time to think about transforming the traditional research folder into a collaboration folder. It would have elements of social bookmarking, traditional research folders, SharePoint, Twitter, Facebook and iCloud - but more customizable to today’s professional workflows. Professionals would be able to store, organize and annotate links. They could selectively grant access to colleagues. Regardless of the proprietary research system to which a link in a folder is stored, colleagues (inside and outside their organizations) would be able to access the content. The folder would be searchable. It would be possible to set up alerts on the links so that when any content stored in the folder is updated, particularly laws and regulations, folder members would know through emails or text messages. It would be possible to access the folder via a mobile device. Invitees would be able to add notes and forward the links. There would be multi-channel access – desktop, cloud, mobile, etc.

There are, of course, several challenges to such a collaboration folder, including confidentiality, the need to delete out-of-date content, the need to manage permissions as folder members’ roles change, etc. Intellectual property issues arise. Who owns the content in the folder? If a folder member annotates content in the folder, does that member own the annotation? It can get complicated.

Regardless of the challenges, I do believe that traditional research folders are evolving towards a collaboration folder. Any thoughts?

As those of us interested in this area look into future product tools and designs that can translate into more effective e-learning courseware and supporting systems, there is an inherent tension in approaching this area that one rarely sees discussed, how much data is enough and where should it come from? Collecting large amounts of data is not the problem per se as much as it is to creating meaningful translation of the data to refine to actionable insight.

Said another way, while it certainly is possible to extract large amounts of behavioral and performance data from online learning environments and even potentially supplement that data with social and mobile sources, turning this data into an actionable framework is where this becomes challenging

If not enough data is collected and assessments are not detailed enough, at best you are generating recommendations that are not particularly predictive. In fact, very few data points definitively point to specific recommendations for a student – whether behavioral or performance based, rather it is the evaluation of multiple data points often evaluated over time that can begin to be framed into a predictive model. For example, studying the lags between student logins to e-learning systems while an indicator of level of student engagement, it in itself is not determinative of engagement level.

If data and assessments are overly detailed, the development of the analytic framework becomes increasingly expensive and potentially a process that needs to be continuously refined based upon future behavior and while a complex data analysis may create a more highly predictive model, if the model is to complex and not sufficiently transparent, it runs the risk as to whether end users will be less inclined to rely on it – potentially creating a “big brother” backlash.

Finding that appropriate balance between the right amount of data and analytical framework in product design in educational settings will require increasing collaboration between a number of professions including instructional designers, educators and database architects to name but a few.

Continuous improvement process

Continuous improvement can be defined as a designed, structured, and methodical process of continued and incremental evolution (see also John Barker’s post about kaizen). Based on the Deming Cycle (aka plan–do–check–adjust), it consists of 4 phases:

1. assessment of the as-is situation and acquisition of enough data to give suggestions for improvement;
2. fine-tuning and implementation of the selected proposals;
3. verification that the proposals in place are bringing the expected results;
4. implementation and standardization of the proposals with any needed adjustments.

Enablers and Inhibitors

Enablers are factors in a company for which the presence or absence of may offer a catalyzing effect, thereby promoting the development of continuous improvement. Conversely, barriers impede or dissipate a firm’s continuous improvement.

Here is a list of some of them:

• Management involvement and strategy: it is fundamental, to allow concentrating on those activities linked to improvement
• Setting objectives and the need for metrics: the ability to measure is key to successful and sustainable change
• Leadership management: managers as change agents
• Worker involvement: they offer ideas, execute the proposals and sometimes implement the improvements
• Resources: human, time and financial
• Roles enlargement and creation of structures: teams should be transversal, made up of members from across each of the different departments contributing in the improvement project
• Methods for establishing and expanding continuous improvement
• Selection of continuous improvement projects
• Cultural aspects (resistance to change)
• Training and abilities

I’m sure you see the positive or negative effects of these while improving your products and maybe you also experimented with some. Next time I will speak about possible levels in the evolution of continuous improvement programs within companies.