I mentioned in a previous post that early Epic employees, like in many startups, had many responsibilities.

While making coffee and cleaning coffee pots didn’t specifically improve a skill I’ve needed since then (I still don’t drink coffee), we did much more day to day.

So many Hats

One thing really missing from the modern Epic employee experience, and frankly, from most software developer employment careers is taking on work that would normally be done by other roles/staff.

The primary differentiator is that the team I was on was self-supportive. The four of us had to handle everything that was happening as it related to our software application.

Everything.

Of course, this included software development. It included all quality assurance from both code review to end-user quality assurance. There were dependencies on other team’s products and their lifecycle and release schedules (back in the 90s Epic releases were done product by product on their own schedules and weren’t always coordinated cross-team). We had to package our own releases AND either deliver them to customers and walk them through the upgrade, or perform the software upgrades.

Sure, I can hear you saying, “just like a startup.” Hold on.

In addition to creating the software, we wrote about the software as well, both from a technical perspective and a end-user perspective. While our writing skills may not have been worthy of an International Award in Technical Writing, our output was as good as other software companies at the time. Just for clarification and to put this in perspective for my readers, there were no “screenshots” as this was all terminal development. If we wanted a screenshot of something to be included in documentation, we had to draw it by hand using ASCII art (our documentation was using a fixed-width typeface at the time in a very rudimentary Epic-built text editor—not something like Microsoft Word). Think Linux man page in terms of what was possible.

With the exception of being able to use modern tools to write documentation, you’re still thinking, “sounds like a lot of small startups.”

The Phone

When I look back at those years though, the most interesting and I think useful role though was that we did hands-on customer support on the telephone. There wasn’t a layer of phone support (tier 1, 2, 3): it was just us. If a customer would call with an issue, they’d speak directly to one of us (after going through the main Epic number to get routed to us). We had a rotating duty of doing Monday through Friday support, but also carried a pager for after hours emergencies (thankfully, those were rare at that time given the nature of the usage of the software we were selling).

I’m an introvert on most days, so getting me to take unexpected calls and have a meaningful interaction took some practice. I wasn’t thrown to the wolves though as the TL made sure that I had back-up available so that A) customers would get the highest quality service and B) so that I wouldn’t have a support-breakdown. After a few calls though, and getting to know the folks on the other end of the phone, it became second nature and a bit of challenging fun to work through the issues they were having.

As expected, I don’t remember most of the calls as they all blur together. I know sometimes they’d call very unhappy about something that had happened. In fact, there were more than a few times there was yelling unfortunately, so being able to listen and respond calmly regardless of the target of their frustrations proved to be invaluable. I recall one time when a customer called very angry about something that was happening with their system. He was yelling and extraordinarily unhappy. He was loud enough that I remember someone else from the team coming over to see if they could help (and this wasn’t on a speaker phone!). After 10-15 minutes of yelling and walking through the problem, and addressing the concerns, he was laughing and joking and we were talking about what plans we had for the weekend (and he did apologize for his demeanor earlier in the call).

I learned a better sense of empathy for what it’s like to be a customer for the products we were building that you can’t get through surveys or through an immersion trip (I definitely have a post planned about Epic’s immersion trips). When a customer was calling, it was very unlikely they were calling to see how we were doing (on rare occasions, it did seem like they were mostly bored and just wanted to chat). Far more likely was that they were calling because they needed a helping hand. It could have been something they caused, or something the software was doing (or hardware or … who knows!). I know they appreciated being able to talk with the developers that made the product rather than going through a support-tree.

The direct customer interaction when things weren’t going well — it’s had a life long impact on the way I’ve built and thought about software. It helped me develop troubleshooting techniques that a lot of people could use.

A few years after I started, it was common that teams had at least one person in a full-time support role, so software developers stopped taking direct calls. It’s a shame too, as a lot of developers at Epic would have benefited from the experience, even if it was only for a year or two.

If you’re saying to yourself, “but I do support!” Great! Is it direct from the first interaction? That’s what can really help you learn about your own software and the perception others have. If it’s being filtered by other support staff, you’re not seeing the whole picture. And, if you’re a manager who thinks reading summaries or even details about support issues is a substitute for actually doing the work: LOL.

Try it yourself!

And while in a world of software that sells internationally makes a phone call direct to an engineer extremely complex when dealing with language and time zones, I’d urge new startups and existing companies to consider how they could get their software developers on the front lines of support occasionally, especially if it can be personal, on the phone/video, and not only through email, or chat. The ability to think on your feet, maintain composure, troubleshoot, empathize, organize, prioritize, and solve problems is a skill that can help every developer level-up.

If you’re in an environment where you would like this opportunity, talk to your manager and see if there’s some way you could be more involved with customers directly. You won’t regret it…, most of the time. 😁

Have you done direct customer support over the phone?

Discount Ends Soon! (April 5, 2024)

And one more thing, there’s only one more week left for a discount on my résumé/LinkedIn profile review service. Don’t miss out on the savings!

Cohort, if you recall from earlier posts was Epic’s public health laboratory management system. RIP. As it was the first product I worked on, I was interested in its success even though the product wasn’t compelling to me. One function was designed for rapid data entry as lab techs were doing hundreds of tests and then inputting the results manually in many cases.

While some lab machine interfaced directly and stored results into the Epic system, there were many that required manual input. Data input was frequently completed in a 80x25 character table like layout. Fields often needed validation at both a field level and a “row” level as some inputs could only be validated with other values in the row being present.

A simple example might be a type where depending on the type selected, the range of acceptable values for a numeric result might change. Or, the user might type in a number, like 65, then choose the unit of measure like mg/dL. If the lab result were for an HDL test (for cholesterol), then those results would make sense. But, if the user had instead typed in 450, it’s very possible that Cohort would have alerted the user that a value was likely out of range. Or, if the user had typed 65 but selected mg/mL for the unit of measure, again, it would have been flagged.

(I’ll fully admit that I don’t remember if the unit was automatically selected when resulting a test like HDL preventing that type of error. There were definitely connected fields however with more complexity).

As my Cohort knowledge expanded, I had little specific awareness regarding what was going on with Epic’s Legacy (later renamed to EpicCare) at the time other than they were building the application with a tool called Visual Basic (version 2 when I first started!). I didn’t have a license for Visual Basic at work or at home.

But, what I did have was a license for Microsoft Visual C++. It had been released in February of 1993 with another version, 1.5 released in December of 1993. (Honestly, I’m not sure why I owned a copy, it may have been through an educational discount).

During the majority of my time at Epic, I spent many hours at home every day learning new technologies that I could potentially apply to my job, both to satisfy my desire to learn and try new things and to try to make Epic a better place.

At some point during my Cohort time, I decided to make a graphical Windows 3.11 Cohort app. Yep. It sounds a bit ridiculous but as you’ll learn as I continue my Epic experience on this blog, that’s not uncommon.

I dove into learning Windows programming and learned what I could about the Microsoft Foundation Class Library. A big shout-out to Charles Petzold’s “Programming Windows” book at the time which helped me immensely.

After a few weeks, I showed my progress to my TL and she was impressed. I admitted that I had no idea how to make it into something more, but that exploring the opportunities was compelling. In some ways, I knew that I was showing initiative and also that my day job wasn’t challenging enough for me. I wanted more. I asked my former TL what she’d done after showing her the demo and while she didn’t remember the specific time, she is confident that she would have relayed on my success and interests to Carl who was her TL at the time.

The application I created wasn’t comprehensive — it supported data entry, saving, etc. But, by no means was it even 2% of the functionality of a Cohort deployed system. But, for me, it was an exciting journey to learning something new and having an opportunity to show it off at work.

Over the years of working at Epic, many people seemed to believe that I had access to some secret sauce that helped me succeed. I didn’t consider it a secret and have been willing to tell folks that it was at its core: hard work and lots of hours. I learn by doing not by talking or even just reading about tech. There’s not a secret shortcut or “one easy step.”

During building this little demo application, I started to learn about how the Windows operating system that would be the focus of much of my professional development career worked internally. That early awareness grew into broad and often deep knowledge that helped me make better design and development choices for decades.

When I started at Epic, we all had a lot of responsibilities. One of the responsibilities was rotating “kitchen” duty. We didn’t have to clean, but we did need to stock the kitchen, and if the coffee pot was empty when arriving, you’d need to start the pot so that others were properly caffeinated. Someone would fetch a large supply of canned (non-alcoholic of course!) drinks. Kitchen duty included stocking the refrigerator.

WE HAD SUGARED DRINKS

This was the era before “sparkling mineral water” and other drinks replaced the sugared soda. We had the usual suspects for software developers from Coke to Mountain Dew.

But, what I hated the most wasn’t stocking the refrigerator: it was end of day coffee pot cleanup. I don’t drink coffee and never have. So, it was not a favorite duty of mine to clean the pot at the end of the day. I likely did a very poor job. The modern Epic campus has nice coffee makers — easy to fill and go (and they make a very large amount at once). Young Epic had the old-school glass coffee pots, nothing fancy.

Eventually, these tasks fell to someone else and the rotation stopped.

I’ll talk more in a later post about the less unusual responsibilities I had during the era when employees wore a lot of different hats to keep the company operating.

What unexpected tasks have you had to do as part of your job?

Epic’s User Group Meeting. For many recent years, UGM has been covered extensively by Madison local newspapers and trade press from around the United States and internationally. It’s quite a spectacle.

It wasn’t always that way. In fact, when I started it was nearly the polar opposite to what someone might experience today if they were to attend.

The first UGM I attended in 1993 was held in the The Madison Concourse Hotel conference center. I’d not been in a hotel conference center before, so I had zero specific expectations. My memories of the facility is that it was fine, very bland, very corporate, very practical. Epic had reserved most of the rooms for various events and sessions that would take place. There were a few broad general sessions with teams parading on stage to discuss upcoming application functionality. They were long sessions. In the era before mobile devices, wifi, laptops, this meant you were either captivated by the content, or lost in your thoughts (or an occasional napper).

The theme that year was…oh, this was before the annual theming became a thing. So, it was themed as Epic UGM 1993. The attendees were quite varied. Remember that this was before EpicCare was a thing with sales. A lot of content was focused on application functionality, but there was a healthy dose of technical content too. Many earlier Epic customers had IT and developers extending and modifying the core functionality at the time. There were courses on Epic’s database for example, Chronicles.

Leading up to the event, Epic employees set up a ton of terminals in the Concourse conference center so that “live training” could take place. I attended most of the “programmer” focused training sessions as a way for me to get up to speed on Epic’s technology stack. My only recollection really is that they were well done courses for the customers. I found them to be very slow paced for me, but I wasn’t the training target.

UGM was absolutely an all hands experience. I don’t recall what I did leading up to the event (I was quite overwhelmed by new employment, a conference, customers, MUMPS, etc.). The entire company was laser focused on UGM during the weeks leading up to the event. We had a number of all staff meetings and there were many many preparation meetings. Practically speaking, very little other work was done leading up to UGM in the month prior. If we weren’t doing UGM prep in some way, you’d likely get a stink eye or two. 🦨👁️

Modern UGM attendees are spoiled by the multimedia extravaganza and spectacle. Remember, this was before fancy digital projectors and before PowerPoint was nearly on every business computer. We had transparencies and overhead projectors and microphones. There was a video projector as far as I recall in the main session room where the Epic applications were shown in their Sunday Finest.

I’ve had to socialize/mingle over the years at many events, but nothing like UGM. I’m an introvert by default, so when my team leader said we’d all been assigned a specific person (a customer’s employee) to shadow and be sure that they were always “doing well” and was having a productive and useful experience, I was more than a bit terrified. This was a person I’d never met nor talked to and I was somehow supposed to stealthily monitor their happiness during UGM. 😬😳

One of Cohort’s customers was the state of Texas public health lab (Cohort = Epic’s Public Health Lab software). My assignment was one of their employees, Ron. I was assured they were all friendly. Not surprisingly, that doesn’t make this introvert feel any better. (“Are some of them not friendly?“)

In any case, when UGM started, my TL introduced me and I proceeded to rather poorly I think try to interact with Ron over the few days of UGM. What’s amusing I suppose is that they knew this was happening too and just played along.

If you wear a suit routinely because you have to, I am sorry. I thankfully donated my last suit when I left Epic. Back in 1993, Everyone had to wear Business Formal. Including customers. It was just the expectation that this was a formal event (the UGM invitations for many years said as much). It was in such stark contrast to Epic on a day to day where anything went, shorts, t-shirts, etc. (And in August in Southern Wisconsin can have some brutally hot and humid days). I didn’t have a suit, so I had to buy one for the “once a year” need. I had to relearn from my father on how to tie a tie the easy way. I’ve never learned anything but that basic knot.

Epic’s UGMs have for decades had a dinner event and in 1993 it was held at a local private event space in the Madison Club. I suppose it wasn’t held in the hotel just to give attendees a break from being in the hotel (but there may have been more to it than that). My recollection is that the dinner was held at this same location the following year. So, I may be confusing the two events, but all I remember was that the facility was essentially at capacity for attendance and some of us, well, less-important staff and products were relegated to an off-room of some sort. For those of you with big families who have attended an event where the kids were in a different room at a table, this had a similar feel. But, what I remember the most was that it was FREAKISHLY HOT. Like a sauna. In my suit.

It was awful. There were too many humans and an overcapacity cooling system that hadn’t paid attention during the UGM prep meetings to keep customers comfortable.

Epic has had a formal way of customers directing development directions over the years. One of course is money (and sales prospects). But, in absence of an extra spend on software functionality, customers have had the opportunity to vote on what was most important to their organization. In advance of UGM, current customers would receive a list of options with some details about the functionality. Each Epic product team would hold a special session going through more details of future development directions and then would step through the suggested options for functionality that could be scheduled. Each customer would then be able to cast a certain number of votes (or rank) helping Epic decide what was most important during the next long development cycle for a product. What was interesting about the process is that it appeared to be a transparent democracy in many ways as these items were discussed. Customers would be able to see what was important for other customers in addition to their own requests. The votes/ranks were completed by the end of the session. The full results however were not given back to customers and were generally only a team guide/influence. Unfortunately, there wasn’t a level of accountability during this process, but very often the ranked/voted functionality was scheduled and worked on. Most importantly, large customers had no more votes than a small customer.

As customers were often suggesting features during a development cycle, this process helped them focus their interests on functionality that was most important to them. There were definitely healthy discussions during the voting sessions too. When Epic was smaller, these meetings had fewer attendees too and I know that one customer’s enthusiasm and need for a feature could transfer to another customer who would end up deciding that they too would benefit from a feature. That type of immediacy and interaction is missing in many software planning discussions unfortunately. A forum post or a thumbs-up 👍 on a feature request isn’t the same. Of course, even then there would be some vocal and opinionated attendees that could derail an otherwise positive discussion (not unlike in a online forum today).

I’ll leave you with this …, downtown Madison not being on a North-South / East-West and having the Capital be in the center with roads wrapping around …, has never never clicked with me. I’ve not had a useful mental model of the streets or layout. It’s broken in my head. I’d not been in the downtown more than a few times ever in my entire life, so nothing was familiar.

So, one day when I left the parking ramp on the way back to my apartment, I was attempting to navigate the streets, discovering more and more places that I didn’t recognize. I didn’t have a map of downtown, only a general map of Wisconsin. With a weird mix of extreme unhappiness and delight, it wasn’t until I drove by the street that eventually goes to the Dane County Airport that I could find my location on the map (the street is called International Lane, which is funny as Dane County Regional airport doesn’t have international flights ✈️).

I was 45 minutes from my apartment. I’d been lost for about 30 minutes. So, what should have been a 15 minute trip turned into more than an hour. Ugh.

My first day at Epic was August 23, 1993. A Monday. I don’t know why I’ve remembered that date so well over the years — it was my first full time job and I suppose that’s locked the date in my brain. Do others remember their start dates too?

It was a remarkable Monday at Epic. And not because I was starting. 😎 Never before had more than 2 people started on the same day. If you’ve seen Epic hiring days during the busy hiring years when 100+ would start on a single day, then 2 probably sounds ridiculous, downright impossible. But, in a company that had fewer than 40 people at the time, hiring 4 people was worthy of a big Note. I remember the names of the others that started still to this day even though I’ve not seen them in decades (I stayed a lot longer). Now that I think about it, the staff hired back in the 90s was quite different from a more modern Epic, and it wasn’t just because of the grunge style popular then.

On that day, I was the only new employee that had no full-time job experience prior to Epic. I was fresh out of college and still had that new-college smell. One of the new hires was placed into a customer facing support role, one was software development, and the other had a few roles that crossed areas. Epic hired software developers often fresh out of college, but not nearly as frequent in the 1990s. New college grads became more common at Epic as the number of job openings increased. In fact, it became rare to encounter someone with prior working experience in software development joining Epic. I do have a topic queued up to talk about the pros and cons of that hiring practice.

The first few hours of Day 1 are a bit hazy but I know there was some paperwork to sign, but not too much: healthcare, confidentiality agreements, employment agreements, etc. We didn’t have ID cards or anything of the sort. Weirdly, I was hoping for a badge as that was going to be a “right of passage” for me. I can’t recall whether we had actual keys to get in … I think we were given keys. The basement of the building had a dentist’s office and a secondary door which was locked, preventing patients of the dentist from making an unexpected visit. The front door was always open during business hours.

Donuts 🍩

Even back in 1993, Epic had the concept of a team mentor that would provide some guidance during the orientation period of employment. I’m sure I met my mentor early in the day — and then we went to gather a donut. One of the long traditions at Epic through the early 2010s or so was on an official hiring day that donuts would be supplied for all employees. The idea was that the donut feeding area would be a gathering place for the new hires to meet existing existing staff and start the day off with some sugared goodness. We hung around in the kitchen for 15-20 minutes and met some staff that way while I enjoyed my (probably) fruit or chocolate cake donut (my first choice when available).

Admittedly in practice, even back then, there wasn’t much donut eating & mingling going on, but the idea was sound. Usually people headed to the kitchen and grabbed a donut and ran back to their offices. No one complained loudly about free donuts. Seriously. (I think there were occasionally bagels too for the non-donut lovers).

Meet My Team

I started on Cohort. Cohort is one of the few Epic applications that was officially put out to the Midwest pasture (AKA sunset or retired). Some portion of its code base lived on in Epic Lab, later to become known as Epic Beaker. Cohort however had an unusually limited market, as it was specifically designed and built for (USA/Canada) public health laboratory systems (of which there was one in each state of the USA and provinces of Canada). So, maximum number of customers: 60 +/-. At the time I joined, there were 4 customers. And, now there were 4 employees on Cohort. There had recently been two departures from the team, so I and a customer facing employee were the shiny replacements.

My TL by the way was the same Epic employee that I’d told: “I just want a job.”

I do wish I had a screen shot of a workflow of Cohort, but given the era it was created and the privacy controls Epic has around it’s software and screenshots for decades, I’m not surprised I cannot dig one up. You’ll need to use your imagination — 80 characters wide x 25 characters tall. Lots of text, some ASCII art lines, and a square blinking cursor. It was a terminal based application. For those of you that occasionally dabble using modern terminal based applications, you should have a good sense of what it was.

Not Much for Day 1

Epic did have a basic training plan for new employees and it shared some content with customer facing technical training. My training would start later due to scheduling issues with the upcoming Epic User Group Meeting that was going to be held in about 2 weeks from my start date. I’ll talk more about that later.

Epic employees that started before the Verona campus did not have a cafeteria available. So, what had been tradition even back in the early nineties was that each day someone would pick a restaurant from a folder of places that could deliver and employees would write down what they wanted and leave the money. Someone would call it in later and announce when the food arrived. Epic paid for my first day’s meal — I have no recollection of what it was that we ate that day. Regardless, it was a smart way to keep staff in the building working more and offer a variety of food options during the week.

The Afternoon

My TL gave me the ABCs of MUMPS programming that I mentioned in a prior post and said: “read and learn this.” My mentor and TL would review some coding exercises (named appropriately: A, B, and C). Additionally, she scheduled some time to start to teach some of what a ‘public health lab system’ was and how the software worked both from a programming perspective, but also from the end user perspective.

My Terminal

Few Epic employees started BEFORE everyone had a personal computer or laptop. My first “Epic” computer was a dumb character-oriented terminal (I don’t remember the models, maybe an ex-Epic from that era remembers and could provide that detail). It wasn’t actually a green screen as far as I recall, it was amber. But, you get the idea. Glowing CRT text on a dark screen. Although, it was one of the fancier terminals as it had TWO connections it could maintain to distinct server connections. Two! The licenses and the server resources were limited though, so if you weren’t using a connection to a server, you’d release it.

I realize that many of you have never experienced first-hand a CRT terminal. While you may have used a LCD terminal, or even a CRT monitor displaying a terminal, it’s just not quite the same as low resolution CRT computer terminal and its glowing text (and the ). I sit here typing this on a 4K monitor display — and I don’t miss the glowing text. 😉

This Introvert’s Nightmare: The Overhead Page

If you needed a server resource and didn’t want to wait … ugh. An introvert’s nightmare…all-building paging system and say, I kid you not:: You’d grab your desk phone’s handset, tap a few keys to activate the

“Paging a COMPUTER_NAME line, paging a COMPUTER_NAME line.”

For example, one of the early servers that Cohort developers would use was called Quark. So, I’d need to announce: “Paging a Quark line.” Everyone’s phone would make the announcement immediately. It was so nerve wracking and I hated doing it. I did not grow more comfortable with that task and was very thankful for the era where that became uncommon and unnecessary.

The Simple Script

While there was a basic new employee orientation script for teams to follow, it wasn’t complicated or lengthy:

• Learn MUMPS/M
• Learn Chronicles (Epic database system)
• Learn your application
• Learn other responsibilities needed by your team (I’ll talk more about how broad this was back then)
• Learn “Epic” through a few courses taught by various staff. Judy taught one called “Philosophy of Epic”

I’ll talk more about these though in later posts as this was just Day 1 with Donuts.

Donut Science

Oh my the Science section of Wikipedia’s article on Donuts is thorough.

Questions for you

What was your first day like at your first full time job in your current career? What’s your favorite donut / doughnut? Write me! I want to know! 🍩💖