An Upgrade to Viewing Chronicles Data
One of the strengths of a good R&D leader is that they’ll let you explore (and build) tools that should save company time and resources. Viewing Chronicles database records in a developer friendly format was a usability challenge. Chronicles isn’t a relational DB. Nor as some might think a NoSQL database. It’s …, well, an amalgamation of a variety of systems, in many ways taking the best of each and mashing them together into a very adaptable solution. (In fact, it did things in 1993 that many modern Database systems still can’t do efficiently, features that would be of benefit to many developers).
A primary reason that there wasn’t a tool that made viewing a full record straightforward is due to the way Chronicles records stored into the MUMPS global (data) systems at the time (the storage has not changed much since my early days at Epic, primarily it’s only been extended, and the classic MUMPS global still remains the only storage). Essentially, there are different storage layouts for different types of Chronicles items in a database. Chronicles uses a sparse storage layout which significantly improves performance when compared to most RDBMSs and (generally) reduces disk storage requirements as well.
Back up for a second though. Epic has used non-common words for various systems for decades. It’s been a sore spot when discussions with more common database system consumers occur.
Here’s the basic breakdown:
| Chronicles Term | Industry Term | Notes |
|---|---|---|
| Database | Collection of related Tables | Too bad Epic didn’t change this term decades ago |
| . | ||
| Dictionary | Table Structure, Schema | I don’t remember why “dictionary” was used |
| . | ||
| Item | Column | |
| . | ||
| Master File | Static Table | Build data, static things (ex., medications) |
| . | ||
| Category List | Pick List | Table as a list, usually static, but with only an identifier and a title and limited other meta-data; ex.: Country |
| . | ||
| Multiple Response Item | One-to-Many table | Feature of Chronicles to not require a secondary table to store multiple values |
| . | ||
| Related Group | One-to-Many table | A collection of multiple response items |
| . | ||
| Over-time | DB Row storing value and timestamp | Definitely an Epic differentiator for reporting and storage efficiency |
| . | ||
The .1 | Primary Key | Probably the table’s primary key, but sometimes like a ROWID in some DB systems |
| . | ||
| Database Initials | Schema Name | Due to the way Chronicles stores data in MUMPS, Databases have 3 letter codes, like EMP for Employee |
| . | ||
| Networked Item | Foreign Key | Item that points at the ID of another Database |
| . | ||
| Global | Storage | See this article for now |
Those are some of the highlights. Chronicles has support for a wide variety of index types as well. There’s a lot more about Chronicles that isn’t relevant here. I wished we’d had this cheat-sheet guide to hand out to non-Epic staff back then.
As Chronicles is very much an internal Epic-built creation that evolved over the decades, it did not integrate with a set of “off-the-shelf” tools without Epic putting resources to provide APIs enabling tools to access an Epic system. (Not too surprising as Epic is a commercial proprietary system.)
One of the challenges I kept running into during my tenure on Cohort was that I often wanted a holistic view of a Chronicles record (and in particular a recurring need for a specific project I’ll talk about later). Chronicles has built in reporting tools (Report Generator), but they weren’t targeted at developers and they weren’t great for ad-hoc developer needs either. So, one afternoon I built a little experiment that allowed me to see all of the data for a Cohort Database, I’m pretty sure it was OVR, which was Cohort’s storage for lab results. There were lots of items in the database and finding the data quickly was painful. Decoding categories and networked items …, it was cumbersome.
I showed the horribly basic results to my TL and she did not discourage me from continuing. 😁 Over the next few weeks on and off working extra hours I built a nice little viewer. But I hadn’t stopped at the basics. I reverse engineered the ANSI escape codes for VT100+, scoured what few technical books I could find (I found ONE BOOK!), and built a reusable scrollable viewer for this new tool I’d built. REMEMBER: No Internet for research.
The basic steps to use the tool were super developer friendly (I was doing DX before DX became a loved and unloved term!):
- Pick the database by typing in the 3 character code.
- Tool would respond with the friendly DB name
- Select the records using some code that existing to allow record ranges, etc.
- Finally, select the Chronicles items to view.
The tool then would brute force it’s way through the various Chronicles items and display results in a 128 column viewport (if the terminal supported it), allowing scrolling up and down. Now in 1993/1994, this wasn’t a common user experience, especially for a developer tool. The tool wasn’t efficient because of how Chronicles was structured. It would look at the database dictionary, and then go on a hunt for the items that may be available.
My TL and others on my team started using the tool and I was encouraged to contribute not only the code for the tool but some of the screen code to the Foundations team so that everyone could use it (and start to build more DX friendly Chronicles/Foundation tools).
After a few weeks of more work getting things moved over, documented, waiting for release cycles to connect, the tool was born:
EAVIEWID
I didn’t use a lot of my normal 40-hours working on this initially, but the little bit of encouragement from my TL and team inspired me to create not just a one-off tool, but an actual utility that was used for decades (and then thankfully got rewritten after discovering that it was being used at customers — it was not intended for production use given how it worked).
No SQL
(And if you’re wondering, … there was no SQL available at the time for Chronicles data.)
Opportunities
Have you built something like this for your employer? Have your managers been encouraging? Why?