Cooked and Cored Apples Anyone?

I’m hardly a fan of a traditional “American Apple Pie”. I’ll eat them, but I’ll look for a better dessert option before I select an apple pie (and most every pie for that matter). I like a good dessert, and if I’m going to eat a lot of calories, I’ll choose something else. In fact, my wife enjoys baking and made a Skillet Pan Apple Cake that was way better to me. But, this isn’t going to be thousands of words about some nonsense like ad farms recipe web site followed by 30 advertisements and finally the recipe. Well, there won’t be advertisements at least. 😏

On EpicCare, on Cadence, and on Tapestry!

The third Epic graphical user interface application launched was Tapestry, the Managed Care solution. The team started shortly after Cadence GUI got off the ground. As I was on the Cadence team at the time, we did quite a bit of initial code sharing—hello copy & paste—with them so that they could get going and save the step of trying to start from the far larger codebase of EpicCare ambulatory at the time. There wasn’t much code sharing at the time beyond the initial dupe and drop though.

I’ll clarify: when there was code sharing it was via copying a file (and whatever specific functionality was necessary). While every team had storage space on a shared networked drive, there absolutely was no real source control system in place. No Git. No SVN. No CVS. There were no Git forks. Some of you may be horrified. 🤯

Long Live the Mapped Drive

Just a folder on a mapped drive (for non-Windows folks, it just meant that Windows PCs had a remote drive available as a specific drive letter, like M — we had M\cadence for example. That location presented as if it were local to our workstations).

That’s all we had and all that was used. It was organized chaos 🤢 at best. We were STRONGLY discouraged from doing file searches on the network storage (as it wickedly thrashed the server drives), so if we wanted to take a sneak peak at another team’s codebase, we’d make a local copy.

I’ll loosely admit the shared folder “worked” because there were only a few of us using any given team’s source code.

Later a developer on EpicCare created a small front end to file copying and introduced the idea of a “locked” file so you could tag a file as being in use and expecting changes. Manual merges of source code were 🤬, high drama, significant pain, and unwelcome events.

Visual Basic 4 and earlier versions weren’t focused on or designed for monstrously large code bases (yet all these applications were rushing headlong into that very situation unfortunately). Further, Visual Basic wasn’t then targeting code sharing either. Sharing meant making a copy and adding a dash of luck and a pinch of hope.

If you knew that someone had snagged a copy of your code and you enhanced it or fixed an issue, the Midwest Nice thing to do was to notify them of what you’d done. Honestly, there wasn’t much of that. The code diverged quickly. Think two children, a box of crayons, and the same drawing to color. The end results may be … dissimilar.

We were young and foolish in many ways, and of course sought to improve and toss our opinionated coding styles into “our” copy of the code generally neglecting to share at all. There wasn’t time we justified (and there was absolute truth to that unfortunately).

As the code bases grew, it became clear that the lack of sharing was becoming a maintenance issue. The Cadence GUI changed the styles of editable controls. Images and icons. Database code. It was all moving along and the only real blessing at the time was that it still wasn’t common for users of one app to need to use another app. So, they wouldn’t see the differences.

We knew though. Oh we knew.

Independence did not mean the code was improving. My favorite recollection of how bad things had become is a highlight for the cultural and technical problem that we had collectively created.

The MUMPS programming language has a function that I am confident is used literally billions and billions of times every day across all Epic customers to this day. It’s fundamental to string parsing in MUMPS. It’s the spice of MUMPS that makes the universe code flow.

It works like this (in a commonly used form):

PIECE(SOURCE,DELIMITER,PIECE_NUMBER)

SOURCE (string) - Target string to be parsed for substrings based on DELIMITER

DELIMITER (string) - One or more characters to use to identify the substrings of the SOURCE string

PIECE_NUMBER (integer) - Given the SOURCE and the DELIMITER, this value represents the specific substring string based on a one-based index.

A few simple examples:

piece("this^is^the^way","^",3) = "the"
piece("this^is^the^way","-",1) = "this^is^the^way"

It’s extraordinarily common to reach for the piece function when storing data in MUMPS globals or even when passing data to functions or when doing RPCs to the server from the GUI application. It’s a useful way to pack data into a single value.

All the GUI applications needed a piece function to extract substrings that had been lovingly hand-crafted in MUMPS.

The Application Kitchens

Because we all knew better than our predecessors, each GUI team had their OWN piece function. You’d think something so fundamental would be shared. Mmm. No.

There was a reason other than hubris why Cadence GUI didn’t share the piece implementation that was used by EpicCare at the time. At first we shared like cousins once removed (“here — take this, I’ll see you at the next reunion!”). However, during a dark day of debugging an issue of code that seemed like it should be working from reading gobs of application code, I debugged into the piece function, baffled by what was happening. Step. Step. St…

WHAT THE FUNC IS HAPPENING???

The EpicCare piece function that we’d been using included a bug. When requesting the last substring from a string in several cases, it would improperly add a empty space character! Like finding a bug in an apple pie, it was just as unwanted in this code.

I rewrote the piece function to exclude the functionality of adding an extra space and tweaked some of the logic so that it would do fewer string concatenation operations leading to a measurable performance boost (the old 386SX/486 Intel chips along with the slow RAM of the day weren’t great at hiding mindless copies that I see far too frequently today in code).

Yes, I did make a journey over to the EpicCare team to discuss the discovery. They were interested and investigated, but … postponed a fix their version indefinitely. They had built a unfortunately surprising amount of logic that depended on the bug so fixing the issue would be far more impactful than desired. They stuck to their recipes.

By postponing the fix, they of course made the impact larger over time. They may have decided to include the new bug-free and faster version as a Piece2 — I can’t remember for sure. Eventually, they fixed their function once tracking down all the locations dependent on the incorrect functionality. To be clear, the piece bug didn’t cause a bug in the application, as their code expected the bug. There was no end-user impact.

Tapestry learned of the bug and rather than using the version we’d created and tested on Cadence GUI, … they wrote their own. I don’t know why. I imagine a developer thought they could further improve the performance of the piece function for some edge cases.

I am skeptical that the amount of extra time spent to attempt further optimizations was realized as a benefit to end users especially given the cost of and loss of time to make and test the changes (and run comparisons, etc.).

In case you’d forgotten, Visual Basic did not have a unit-test or profiling infrastructure at all. If you wanted “test code”, you’d add it, and then remove it (or comment it out if it could be valuable in the future). There were many developers that used their watch or a stopwatch to do performance analysis back then. It was elementary at best.

I wish I could honestly say I’ve never spent time over-optimizing a function. Those who know me well also know that I later practiced and evangelized:

“optimize when you need to optimize and no earlier.” -A More Mature Me.

Or …

”Sometimes you’ve got to know when to hold back optimizations, and other times, know when to fold them in.” - Definitely NOT Kenny Rogers

The amount of duplicated yet diverging code grew until their was an upswell of interest in creating a new team to maintain and share application code. After a few meetings, it was a go. Not long after, the team was staffed by one developer full-time with assistance from all the GUI application teams.

It was called…

ApplCore

One of the primary motives of the ApplCore team was to release shared code with the applications which was a different strategy from the “Core” Foundations team. (Get it? Application Core vs Core? Yeah, not original.) ApplCore wasn’t a product as much as it was a “concept.”

The Foundations team was always working on a version ahead of products. Changes in Foundations would be made, tested, and then released on a specific date. Teams would then take that release, do their development during their development cycle, and ship a release.

The rationale was that the Foundations team wanted maximal “baking” time for their code. By releasing a full development cycle ahead, they could be generally assured that their code would be tested at Epic for a full application product development cycle.

But ApplCore would be different. The delay of doing coding a full release cycle ahead didn’t work well for application products. Requiring a massive amount of predictive coding was untenable: “what are we going to need in 18 months”?

Unlike Foundations, the ApplCore code still lived in its own source tree. There was no obligation to take the code at any particular time, but teams were expected to not make changes to a copy of the ApplCore code without contributing back to the original.

I’m going to save the details of why ApplCore was doomed to fail for a later post. Some of you may know. 😉

And now onto the tasty part …

Apple Pie Skillet Cake, Canned Apple Pie Filling version

This is not low calorie, healthy or low fat. This is tasty and straightforward. It’s not fancy. It uses canned apple pie filling. 😋

• 1 10-inch iron skillet
• 1 20oz can of apple pie filling (sweetened)
• 1 cup unsalted butter (2 sticks)
• 1 1/2 cups packed light brown sugar
• 1/2 tsp. ground cinnamon, plus a little more for topping
• 1 tsp. pure vanilla extract
• 2 large eggs
• 2 cups all-purpose flour
• 1 tsp. baking powder
• 1 tsp. kosher salt
• Optional, but say YES to vanilla ice cream (or a cinnamon ice cream is also GREAT)

1. Preheat Oven to 350F
2. (Optionally stir in 1 tsp of cinnamon into the apples in a bowl)
3. Melt 1 cup butter in a microwave safe bowl (or on stove)
4. In large bowl, whisk in the vanilla, sugar, and 1/2 tsp. cinnamon
5. Once well combined, whisk in eggs until smooth
6. Add flour, baking powder and salt until just combined — don’t over-mix
7. Fold approximate 1 1/2 of the can of apple pie filling into the bowl
8. Pour batter into skillet
9. Bake for about 25-30 minutes (check with a toothpick — should come out clean)
10. The pan will be HOT 🔥 (and stay hot)
11. Top with remaining apples and add extra cinnamon if desired

Enjoy!