One of the strengths of a good R&D leader is that they’ll let you explore (and build) tools that should save company time and resources. Viewing Chronicles database records in a developer friendly format was a usability challenge. Chronicles isn’t a relational DB. Nor as some might think a NoSQL database. It’s …, well, an amalgamation of a variety of systems, in many ways taking the best of each and mashing them together into a very adaptable solution. (In fact, it did things in 1993 that many modern Database systems still can’t do efficiently, features that would be of benefit to many developers).

A primary reason that there wasn’t a tool that made viewing a full record straightforward is due to the way Chronicles records stored into the MUMPS global (data) systems at the time (the storage has not changed much since my early days at Epic, primarily it’s only been extended, and the classic MUMPS global still remains the only storage). Essentially, there are different storage layouts for different types of Chronicles items in a database. Chronicles uses a sparse storage layout which significantly improves performance when compared to most RDBMSs and (generally) reduces disk storage requirements as well.

Back up for a second though. Epic has used non-common words for various systems for decades. It’s been a sore spot when discussions with more common database system consumers occur.

Here’s the basic breakdown:

Chronicles Term	Industry Term	Notes
Database	Collection of related Tables	Too bad Epic didn’t change this term decades ago
		.
Dictionary	Table Structure, Schema	I don’t remember why “dictionary” was used
		.
Item	Column
		.
Master File	Static Table	Build data, static things (ex., medications)
		.
Category List	Pick List	Table as a list, usually static, but with only an identifier and a title and limited other meta-data; ex.: Country
		.
Multiple Response Item	One-to-Many table	Feature of Chronicles to not require a secondary table to store multiple values
		.
Related Group	One-to-Many table	A collection of multiple response items
		.
Over-time	DB Row storing value and timestamp	Definitely an Epic differentiator for reporting and storage efficiency
		.
The .1	Primary Key	Probably the table’s primary key, but sometimes like a ROWID in some DB systems
		.
Database Initials	Schema Name	Due to the way Chronicles stores data in MUMPS, Databases have 3 letter codes, like EMP for Employee
		.
Networked Item	Foreign Key	Item that points at the ID of another Database
		.
Global	Storage	See this article for now

Those are some of the highlights. Chronicles has support for a wide variety of index types as well. There’s a lot more about Chronicles that isn’t relevant here. I wished we’d had this cheat-sheet guide to hand out to non-Epic staff back then.

As Chronicles is very much an internal Epic-built creation that evolved over the decades, it did not integrate with a set of “off-the-shelf” tools without Epic putting resources to provide APIs enabling tools to access an Epic system. (Not too surprising as Epic is a commercial proprietary system.)

One of the challenges I kept running into during my tenure on Cohort was that I often wanted a holistic view of a Chronicles record (and in particular a recurring need for a specific project I’ll talk about later). Chronicles has built in reporting tools (Report Generator), but they weren’t targeted at developers and they weren’t great for ad-hoc developer needs either. So, one afternoon I built a little experiment that allowed me to see all of the data for a Cohort Database, I’m pretty sure it was OVR, which was Cohort’s storage for lab results. There were lots of items in the database and finding the data quickly was painful. Decoding categories and networked items …, it was cumbersome.

I showed the horribly basic results to my TL and she did not discourage me from continuing. 😁 Over the next few weeks on and off working extra hours I built a nice little viewer. But I hadn’t stopped at the basics. I reverse engineered the ANSI escape codes for VT100+, scoured what few technical books I could find (I found ONE BOOK!), and built a reusable scrollable viewer for this new tool I’d built. REMEMBER: No Internet for research.

The basic steps to use the tool were super developer friendly (I was doing DX before DX became a loved and unloved term!):

• Pick the database by typing in the 3 character code.
• Tool would respond with the friendly DB name
• Select the records using some code that existing to allow record ranges, etc.
• Finally, select the Chronicles items to view.

The tool then would brute force it’s way through the various Chronicles items and display results in a 128 column viewport (if the terminal supported it), allowing scrolling up and down. Now in 1993/1994, this wasn’t a common user experience, especially for a developer tool. The tool wasn’t efficient because of how Chronicles was structured. It would look at the database dictionary, and then go on a hunt for the items that may be available.

My TL and others on my team started using the tool and I was encouraged to contribute not only the code for the tool but some of the screen code to the Foundations team so that everyone could use it (and start to build more DX friendly Chronicles/Foundation tools).

After a few weeks of more work getting things moved over, documented, waiting for release cycles to connect, the tool was born:

EAVIEWID

I didn’t use a lot of my normal 40-hours working on this initially, but the little bit of encouragement from my TL and team inspired me to create not just a one-off tool, but an actual utility that was used for decades (and then thankfully got rewritten after discovering that it was being used at customers — it was not intended for production use given how it worked).

No SQL

(And if you’re wondering, … there was no SQL available at the time for Chronicles data.)

Opportunities

Have you built something like this for your employer? Have your managers been encouraging? Why?

My blog post planned for today was to announce that I had a subscription email set up so you could have my blog posts delivered directly to your email inbox.

Instead of spending the time writing the post, I worked on setup instead. And then … it went longer and longer, and it seemed like the settings and choices never ended and … it’s not going to happen today.

So, this post is about a thing that is coming soon.

I think I got caught up a bit in the “look how shiny and easy this is… marketing fluff and enthusiastic user testimonials,” and didn’t see that getting things off the ground wasn’t so straightforward.

I’m very likely to use the services from buttondown, but the onboarding has been a little clunky, and I’ve lost some data due to some frustrating user interface choices. It’s probably super easy to get something ugly and basic together using buttondown, but I’m not happy with that.

Till next time …, when I plan to be actually done with the setup.

I mentioned in a previous post that early Epic employees, like in many startups, had many responsibilities.

While making coffee and cleaning coffee pots didn’t specifically improve a skill I’ve needed since then (I still don’t drink coffee), we did much more day to day.

So many Hats

One thing really missing from the modern Epic employee experience, and frankly, from most software developer employment careers is taking on work that would normally be done by other roles/staff.

The primary differentiator is that the team I was on was self-supportive. The four of us had to handle everything that was happening as it related to our software application.

Everything.

Of course, this included software development. It included all quality assurance from both code review to end-user quality assurance. There were dependencies on other team’s products and their lifecycle and release schedules (back in the 90s Epic releases were done product by product on their own schedules and weren’t always coordinated cross-team). We had to package our own releases AND either deliver them to customers and walk them through the upgrade, or perform the software upgrades.

Sure, I can hear you saying, “just like a startup.” Hold on.

In addition to creating the software, we wrote about the software as well, both from a technical perspective and a end-user perspective. While our writing skills may not have been worthy of an International Award in Technical Writing, our output was as good as other software companies at the time. Just for clarification and to put this in perspective for my readers, there were no “screenshots” as this was all terminal development. If we wanted a screenshot of something to be included in documentation, we had to draw it by hand using ASCII art (our documentation was using a fixed-width typeface at the time in a very rudimentary Epic-built text editor—not something like Microsoft Word). Think Linux man page in terms of what was possible.

With the exception of being able to use modern tools to write documentation, you’re still thinking, “sounds like a lot of small startups.”

The Phone

When I look back at those years though, the most interesting and I think useful role though was that we did hands-on customer support on the telephone. There wasn’t a layer of phone support (tier 1, 2, 3): it was just us. If a customer would call with an issue, they’d speak directly to one of us (after going through the main Epic number to get routed to us). We had a rotating duty of doing Monday through Friday support, but also carried a pager for after hours emergencies (thankfully, those were rare at that time given the nature of the usage of the software we were selling).

I’m an introvert on most days, so getting me to take unexpected calls and have a meaningful interaction took some practice. I wasn’t thrown to the wolves though as the TL made sure that I had back-up available so that A) customers would get the highest quality service and B) so that I wouldn’t have a support-breakdown. After a few calls though, and getting to know the folks on the other end of the phone, it became second nature and a bit of challenging fun to work through the issues they were having.

As expected, I don’t remember most of the calls as they all blur together. I know sometimes they’d call very unhappy about something that had happened. In fact, there were more than a few times there was yelling unfortunately, so being able to listen and respond calmly regardless of the target of their frustrations proved to be invaluable. I recall one time when a customer called very angry about something that was happening with their system. He was yelling and extraordinarily unhappy. He was loud enough that I remember someone else from the team coming over to see if they could help (and this wasn’t on a speaker phone!). After 10-15 minutes of yelling and walking through the problem, and addressing the concerns, he was laughing and joking and we were talking about what plans we had for the weekend (and he did apologize for his demeanor earlier in the call).

I learned a better sense of empathy for what it’s like to be a customer for the products we were building that you can’t get through surveys or through an immersion trip (I definitely have a post planned about Epic’s immersion trips). When a customer was calling, it was very unlikely they were calling to see how we were doing (on rare occasions, it did seem like they were mostly bored and just wanted to chat). Far more likely was that they were calling because they needed a helping hand. It could have been something they caused, or something the software was doing (or hardware or … who knows!). I know they appreciated being able to talk with the developers that made the product rather than going through a support-tree.

The direct customer interaction when things weren’t going well — it’s had a life long impact on the way I’ve built and thought about software. It helped me develop troubleshooting techniques that a lot of people could use.

A few years after I started, it was common that teams had at least one person in a full-time support role, so software developers stopped taking direct calls. It’s a shame too, as a lot of developers at Epic would have benefited from the experience, even if it was only for a year or two.

If you’re saying to yourself, “but I do support!” Great! Is it direct from the first interaction? That’s what can really help you learn about your own software and the perception others have. If it’s being filtered by other support staff, you’re not seeing the whole picture. And, if you’re a manager who thinks reading summaries or even details about support issues is a substitute for actually doing the work: LOL.

Try it yourself!

And while in a world of software that sells internationally makes a phone call direct to an engineer extremely complex when dealing with language and time zones, I’d urge new startups and existing companies to consider how they could get their software developers on the front lines of support occasionally, especially if it can be personal, on the phone/video, and not only through email, or chat. The ability to think on your feet, maintain composure, troubleshoot, empathize, organize, prioritize, and solve problems is a skill that can help every developer level-up.

If you’re in an environment where you would like this opportunity, talk to your manager and see if there’s some way you could be more involved with customers directly. You won’t regret it…, most of the time. 😁

Have you done direct customer support over the phone?

Discount Ends Soon! (April 5, 2024)

And one more thing, there’s only one more week left for a discount on my résumé/LinkedIn profile review service. Don’t miss out on the savings!

Cohort, if you recall from earlier posts was Epic’s public health laboratory management system. RIP. As it was the first product I worked on, I was interested in its success even though the product wasn’t compelling to me. One function was designed for rapid data entry as lab techs were doing hundreds of tests and then inputting the results manually in many cases.

While some lab machine interfaced directly and stored results into the Epic system, there were many that required manual input. Data input was frequently completed in a 80x25 character table like layout. Fields often needed validation at both a field level and a “row” level as some inputs could only be validated with other values in the row being present.

A simple example might be a type where depending on the type selected, the range of acceptable values for a numeric result might change. Or, the user might type in a number, like 65, then choose the unit of measure like mg/dL. If the lab result were for an HDL test (for cholesterol), then those results would make sense. But, if the user had instead typed in 450, it’s very possible that Cohort would have alerted the user that a value was likely out of range. Or, if the user had typed 65 but selected mg/mL for the unit of measure, again, it would have been flagged.

(I’ll fully admit that I don’t remember if the unit was automatically selected when resulting a test like HDL preventing that type of error. There were definitely connected fields however with more complexity).

As my Cohort knowledge expanded, I had little specific awareness regarding what was going on with Epic’s Legacy (later renamed to EpicCare) at the time other than they were building the application with a tool called Visual Basic (version 2 when I first started!). I didn’t have a license for Visual Basic at work or at home.

But, what I did have was a license for Microsoft Visual C++. It had been released in February of 1993 with another version, 1.5 released in December of 1993. (Honestly, I’m not sure why I owned a copy, it may have been through an educational discount).

During the majority of my time at Epic, I spent many hours at home every day learning new technologies that I could potentially apply to my job, both to satisfy my desire to learn and try new things and to try to make Epic a better place.

At some point during my Cohort time, I decided to make a graphical Windows 3.11 Cohort app. Yep. It sounds a bit ridiculous but as you’ll learn as I continue my Epic experience on this blog, that’s not uncommon.

I dove into learning Windows programming and learned what I could about the Microsoft Foundation Class Library. A big shout-out to Charles Petzold’s “Programming Windows” book at the time which helped me immensely.

After a few weeks, I showed my progress to my TL and she was impressed. I admitted that I had no idea how to make it into something more, but that exploring the opportunities was compelling. In some ways, I knew that I was showing initiative and also that my day job wasn’t challenging enough for me. I wanted more. I asked my former TL what she’d done after showing her the demo and while she didn’t remember the specific time, she is confident that she would have relayed on my success and interests to Carl who was her TL at the time.

The application I created wasn’t comprehensive — it supported data entry, saving, etc. But, by no means was it even 2% of the functionality of a Cohort deployed system. But, for me, it was an exciting journey to learning something new and having an opportunity to show it off at work.

Over the years of working at Epic, many people seemed to believe that I had access to some secret sauce that helped me succeed. I didn’t consider it a secret and have been willing to tell folks that it was at its core: hard work and lots of hours. I learn by doing not by talking or even just reading about tech. There’s not a secret shortcut or “one easy step.”

During building this little demo application, I started to learn about how the Windows operating system that would be the focus of much of my professional development career worked internally. That early awareness grew into broad and often deep knowledge that helped me make better design and development choices for decades.

When I started at Epic, we all had a lot of responsibilities. One of the responsibilities was rotating “kitchen” duty. We didn’t have to clean, but we did need to stock the kitchen, and if the coffee pot was empty when arriving, you’d need to start the pot so that others were properly caffeinated. Someone would fetch a large supply of canned (non-alcoholic of course!) drinks. Kitchen duty included stocking the refrigerator.

WE HAD SUGARED DRINKS

This was the era before “sparkling mineral water” and other drinks replaced the sugared soda. We had the usual suspects for software developers from Coke to Mountain Dew.

But, what I hated the most wasn’t stocking the refrigerator: it was end of day coffee pot cleanup. I don’t drink coffee and never have. So, it was not a favorite duty of mine to clean the pot at the end of the day. I likely did a very poor job. The modern Epic campus has nice coffee makers — easy to fill and go (and they make a very large amount at once). Young Epic had the old-school glass coffee pots, nothing fancy.

Eventually, these tasks fell to someone else and the rotation stopped.

I’ll talk more in a later post about the less unusual responsibilities I had during the era when employees wore a lot of different hats to keep the company operating.

What unexpected tasks have you had to do as part of your job?