My blog post planned for today was to announce that I had a subscription email set up so you could have my blog posts delivered directly to your email inbox.
Instead of spending the time writing the post, I worked on setup instead. And then … it went longer and longer, and it seemed like the settings and choices never ended and … it’s not going to happen today.
So, this post is about a thing that is coming soon.
I think I got caught up a bit in the “look how shiny and easy this is… marketing fluff and enthusiastic user testimonials,” and didn’t see that getting things off the ground wasn’t so straightforward.
I’m very likely to use the services from buttondown, but the onboarding has been a little clunky, and I’ve lost some data due to some frustrating user interface choices. It’s probably super easy to get something ugly and basic together using buttondown, but I’m not happy with that.
Till next time …, when I plan to be actually done with the setup.
I mentioned in a previous post that early Epic employees, like in many startups, had many responsibilities.
While making coffee and cleaning coffee pots didn’t specifically improve a skill I’ve needed since then (I still don’t drink coffee), we did much more day to day.
So many Hats
One thing really missing from the modern Epic employee experience, and frankly, from most software developer employment careers is taking on work that would normally be done by other roles/staff.
The primary differentiator is that the team I was on was self-supportive. The four of us had to handle everything that was happening as it related to our software application.
Everything.
Of course, this included software development. It included all quality assurance from both code review to end-user quality assurance. There were dependencies on other team’s products and their lifecycle and release schedules (back in the 90s Epic releases were done product by product on their own schedules and weren’t always coordinated cross-team). We had to package our own releases AND either deliver them to customers and walk them through the upgrade, or perform the software upgrades.
Sure, I can hear you saying, “just like a startup.” Hold on.
In addition to creating the software, we wrote about the software as well, both from a technical perspective and a end-user perspective. While our writing skills may not have been worthy of an International Award in Technical Writing, our output was as good as other software companies at the time. Just for clarification and to put this in perspective for my readers, there were no “screenshots” as this was all terminal development. If we wanted a screenshot of something to be included in documentation, we had to draw it by hand using ASCII art (our documentation was using a fixed-width typeface at the time in a very rudimentary Epic-built text editor—not something like Microsoft Word). Think Linux man page in terms of what was possible.
With the exception of being able to use modern tools to write documentation, you’re still thinking, “sounds like a lot of small startups.”
The Phone
When I look back at those years though, the most interesting and I think useful role though was that we did hands-on customer support on the telephone. There wasn’t a layer of phone support (tier 1, 2, 3): it was just us. If a customer would call with an issue, they’d speak directly to one of us (after going through the main Epic number to get routed to us). We had a rotating duty of doing Monday through Friday support, but also carried a pager for after hours emergencies (thankfully, those were rare at that time given the nature of the usage of the software we were selling).
I’m an introvert on most days, so getting me to take unexpected calls and have a meaningful interaction took some practice. I wasn’t thrown to the wolves though as the TL made sure that I had back-up available so that A) customers would get the highest quality service and B) so that I wouldn’t have a support-breakdown. After a few calls though, and getting to know the folks on the other end of the phone, it became second nature and a bit of challenging fun to work through the issues they were having.
As expected, I don’t remember most of the calls as they all blur together. I know sometimes they’d call very unhappy about something that had happened. In fact, there were more than a few times there was yelling unfortunately, so being able to listen and respond calmly regardless of the target of their frustrations proved to be invaluable. I recall one time when a customer called very angry about something that was happening with their system. He was yelling and extraordinarily unhappy. He was loud enough that I remember someone else from the team coming over to see if they could help (and this wasn’t on a speaker phone!). After 10-15 minutes of yelling and walking through the problem, and addressing the concerns, he was laughing and joking and we were talking about what plans we had for the weekend (and he did apologize for his demeanor earlier in the call).
I learned a better sense of empathy for what it’s like to be a customer for the products we were building that you can’t get through surveys or through an immersion trip (I definitely have a post planned about Epic’s immersion trips). When a customer was calling, it was very unlikely they were calling to see how we were doing (on rare occasions, it did seem like they were mostly bored and just wanted to chat). Far more likely was that they were calling because they needed a helping hand. It could have been something they caused, or something the software was doing (or hardware or … who knows!). I know they appreciated being able to talk with the developers that made the product rather than going through a support-tree.
The direct customer interaction when things weren’t going well — it’s had a life long impact on the way I’ve built and thought about software. It helped me develop troubleshooting techniques that a lot of people could use.
A few years after I started, it was common that teams had at least one person in a full-time support role, so software developers stopped taking direct calls. It’s a shame too, as a lot of developers at Epic would have benefited from the experience, even if it was only for a year or two.
If you’re saying to yourself, “but I do support!” Great! Is it direct from the first interaction? That’s what can really help you learn about your own software and the perception others have. If it’s being filtered by other support staff, you’re not seeing the whole picture. And, if you’re a manager who thinks reading summaries or even details about support issues is a substitute for actually doing the work: LOL.
Try it yourself!
And while in a world of software that sells internationally makes a phone call direct to an engineer extremely complex when dealing with language and time zones, I’d urge new startups and existing companies to consider how they could get their software developers on the front lines of support occasionally, especially if it can be personal, on the phone/video, and not only through email, or chat. The ability to think on your feet, maintain composure, troubleshoot, empathize, organize, prioritize, and solve problems is a skill that can help every developer level-up.
If you’re in an environment where you would like this opportunity, talk to your manager and see if there’s some way you could be more involved with customers directly. You won’t regret it…, most of the time. 😁
Have you done direct customer support over the phone?
Cohort, if you recall from earlier posts was Epic’s public health laboratory management system. RIP. As it was the first product I worked on, I was interested in its success even though the product wasn’t compelling to me. One function was designed for rapid data entry as lab techs were doing hundreds of tests and then inputting the results manually in many cases.
While some lab machine interfaced directly and stored results into the Epic system, there were many that required manual input. Data input was frequently completed in a 80x25 character table like layout. Fields often needed validation at both a field level and a “row” level as some inputs could only be validated with other values in the row being present.
A simple example might be a type where depending on the type selected, the range of acceptable values for a numeric result might change. Or, the user might type in a number, like 65, then choose the unit of measure like mg/dL. If the lab result were for an HDL test (for cholesterol), then those results would make sense. But, if the user had instead typed in 450, it’s very possible that Cohort would have alerted the user that a value was likely out of range. Or, if the user had typed 65 but selected mg/mL for the unit of measure, again, it would have been flagged.
(I’ll fully admit that I don’t remember if the unit was automatically selected when resulting a test like HDL preventing that type of error. There were definitely connected fields however with more complexity).
As my Cohort knowledge expanded, I had little specific awareness regarding what was going on with Epic’s Legacy (later renamed to EpicCare) at the time other than they were building the application with a tool called Visual Basic (version 2 when I first started!). I didn’t have a license for Visual Basic at work or at home.
But, what I did have was a license for Microsoft Visual C++. It had been released in February of 1993 with another version, 1.5 released in December of 1993. (Honestly, I’m not sure why I owned a copy, it may have been through an educational discount).
During the majority of my time at Epic, I spent many hours at home every day learning new technologies that I could potentially apply to my job, both to satisfy my desire to learn and try new things and to try to make Epic a better place.
At some point during my Cohort time, I decided to make a graphical Windows 3.11 Cohort app. Yep. It sounds a bit ridiculous but as you’ll learn as I continue my Epic experience on this blog, that’s not uncommon.
After a few weeks, I showed my progress to my TL and she was impressed. I admitted that I had no idea how to make it into something more, but that exploring the opportunities was compelling. In some ways, I knew that I was showing initiative and also that my day job wasn’t challenging enough for me. I wanted more. I asked my former TL what she’d done after showing her the demo and while she didn’t remember the specific time, she is confident that she would have relayed on my success and interests to Carl who was her TL at the time.
The application I created wasn’t comprehensive — it supported data entry, saving, etc. But, by no means was it even 2% of the functionality of a Cohort deployed system. But, for me, it was an exciting journey to learning something new and having an opportunity to show it off at work.
Over the years of working at Epic, many people seemed to believe that I had access to some secret sauce that helped me succeed. I didn’t consider it a secret and have been willing to tell folks that it was at its core: hard work and lots of hours. I learn by doing not by talking or even just reading about tech. There’s not a secret shortcut or “one easy step.”
During building this little demo application, I started to learn about how the Windows operating system that would be the focus of much of my professional development career worked internally. That early awareness grew into broad and often deep knowledge that helped me make better design and development choices for decades.
When I started at Epic, we all had a lot of responsibilities. One of the responsibilities was rotating “kitchen” duty. We didn’t have to clean, but we did need to stock the kitchen, and if the coffee pot was empty when arriving, you’d need to start the pot so that others were properly caffeinated. Someone would fetch a large supply of canned (non-alcoholic of course!) drinks. Kitchen duty included stocking the refrigerator.
WE HAD SUGARED DRINKS
This was the era before “sparkling mineral water” and other drinks replaced the sugared soda. We had the usual suspects for software developers from Coke to Mountain Dew.
But, what I hated the most wasn’t stocking the refrigerator: it was end of day coffee pot cleanup. I don’t drink coffee and never have. So, it was not a favorite duty of mine to clean the pot at the end of the day. I likely did a very poor job. The modern Epic campus has nice coffee makers — easy to fill and go (and they make a very large amount at once). Young Epic had the old-school glass coffee pots, nothing fancy.
Eventually, these tasks fell to someone else and the rotation stopped.
I’ll talk more in a later post about the less unusual responsibilities I had during the era when employees wore a lot of different hats to keep the company operating.
What unexpected tasks have you had to do as part of your job?
Epic’s User Group Meeting. For many recent years, UGM has been covered extensively by Madison local newspapers and trade press from around the United States and internationally. It’s quite a spectacle.
It wasn’t always that way. In fact, when I started it was nearly the polar opposite to what someone might experience today if they were to attend.
The first UGM I attended in 1993 was held in the The Madison Concourse Hotel conference center. I’d not been in a hotel conference center before, so I had zero specific expectations. My memories of the facility is that it was fine, very bland, very corporate, very practical. Epic had reserved most of the rooms for various events and sessions that would take place. There were a few broad general sessions with teams parading on stage to discuss upcoming application functionality. They were long sessions. In the era before mobile devices, wifi, laptops, this meant you were either captivated by the content, or lost in your thoughts (or an occasional napper).
The theme that year was…oh, this was before the annual theming became a thing. So, it was themed as Epic UGM 1993. The attendees were quite varied. Remember that this was before EpicCare was a thing with sales. A lot of content was focused on application functionality, but there was a healthy dose of technical content too. Many earlier Epic customers had IT and developers extending and modifying the core functionality at the time. There were courses on Epic’s database for example, Chronicles.
Leading up to the event, Epic employees set up a ton of terminals in the Concourse conference center so that “live training” could take place. I attended most of the “programmer” focused training sessions as a way for me to get up to speed on Epic’s technology stack. My only recollection really is that they were well done courses for the customers. I found them to be very slow paced for me, but I wasn’t the training target.
UGM was absolutely an all hands experience. I don’t recall what I did leading up to the event (I was quite overwhelmed by new employment, a conference, customers, MUMPS, etc.). The entire company was laser focused on UGM during the weeks leading up to the event. We had a number of all staff meetings and there were many many preparation meetings. Practically speaking, very little other work was done leading up to UGM in the month prior. If we weren’t doing UGM prep in some way, you’d likely get a stink eye or two. 🦨👁️
Modern UGM attendees are spoiled by the multimedia extravaganza and spectacle. Remember, this was before fancy digital projectors and before PowerPoint was nearly on every business computer. We had transparencies and overhead projectors and microphones. There was a video projector as far as I recall in the main session room where the Epic applications were shown in their Sunday Finest.
I’ve had to socialize/mingle over the years at many events, but nothing like UGM. I’m an introvert by default, so when my team leader said we’d all been assigned a specific person (a customer’s employee) to shadow and be sure that they were always “doing well” and was having a productive and useful experience, I was more than a bit terrified. This was a person I’d never met nor talked to and I was somehow supposed to stealthily monitor their happiness during UGM. 😬😳
One of Cohort’s customers was the state of Texas public health lab (Cohort = Epic’s Public Health Lab software). My assignment was one of their employees, Ron. I was assured they were all friendly. Not surprisingly, that doesn’t make this introvert feel any better. (“Are some of them not friendly?“)
In any case, when UGM started, my TL introduced me and I proceeded to rather poorly I think try to interact with Ron over the few days of UGM. What’s amusing I suppose is that they knew this was happening too and just played along.
If you wear a suit routinely because you have to, I am sorry. I thankfully donated my last suit when I left Epic. Back in 1993, Everyone had to wear Business Formal. Including customers. It was just the expectation that this was a formal event (the UGM invitations for many years said as much). It was in such stark contrast to Epic on a day to day where anything went, shorts, t-shirts, etc. (And in August in Southern Wisconsin can have some brutally hot and humid days). I didn’t have a suit, so I had to buy one for the “once a year” need. I had to relearn from my father on how to tie a tie the easy way. I’ve never learned anything but that basic knot.
Epic’s UGMs have for decades had a dinner event and in 1993 it was held at a local private event space in the Madison Club. I suppose it wasn’t held in the hotel just to give attendees a break from being in the hotel (but there may have been more to it than that). My recollection is that the dinner was held at this same location the following year. So, I may be confusing the two events, but all I remember was that the facility was essentially at capacity for attendance and some of us, well, less-important staff and products were relegated to an off-room of some sort. For those of you with big families who have attended an event where the kids were in a different room at a table, this had a similar feel. But, what I remember the most was that it was FREAKISHLY HOT. Like a sauna. In my suit.
It was awful. There were too many humans and an overcapacity cooling system that hadn’t paid attention during the UGM prep meetings to keep customers comfortable.
Epic has had a formal way of customers directing development directions over the years. One of course is money (and sales prospects). But, in absence of an extra spend on software functionality, customers have had the opportunity to vote on what was most important to their organization. In advance of UGM, current customers would receive a list of options with some details about the functionality. Each Epic product team would hold a special session going through more details of future development directions and then would step through the suggested options for functionality that could be scheduled. Each customer would then be able to cast a certain number of votes (or rank) helping Epic decide what was most important during the next long development cycle for a product. What was interesting about the process is that it appeared to be a transparent democracy in many ways as these items were discussed. Customers would be able to see what was important for other customers in addition to their own requests. The votes/ranks were completed by the end of the session. The full results however were not given back to customers and were generally only a team guide/influence. Unfortunately, there wasn’t a level of accountability during this process, but very often the ranked/voted functionality was scheduled and worked on. Most importantly, large customers had no more votes than a small customer.
As customers were often suggesting features during a development cycle, this process helped them focus their interests on functionality that was most important to them. There were definitely healthy discussions during the voting sessions too. When Epic was smaller, these meetings had fewer attendees too and I know that one customer’s enthusiasm and need for a feature could transfer to another customer who would end up deciding that they too would benefit from a feature. That type of immediacy and interaction is missing in many software planning discussions unfortunately. A forum post or a thumbs-up 👍 on a feature request isn’t the same. Of course, even then there would be some vocal and opinionated attendees that could derail an otherwise positive discussion (not unlike in a online forum today).
I’ll leave you with this …, downtown Madison not being on a North-South / East-West and having the Capital be in the center with roads wrapping around …, has never never clicked with me. I’ve not had a useful mental model of the streets or layout. It’s broken in my head. I’d not been in the downtown more than a few times ever in my entire life, so nothing was familiar.
So, one day when I left the parking ramp on the way back to my apartment, I was attempting to navigate the streets, discovering more and more places that I didn’t recognize. I didn’t have a map of downtown, only a general map of Wisconsin. With a weird mix of extreme unhappiness and delight, it wasn’t until I drove by the street that eventually goes to the Dane County Airport that I could find my location on the map (the street is called International Lane, which is funny as Dane County Regional airport doesn’t have international flights ✈️).
I was 45 minutes from my apartment. I’d been lost for about 30 minutes. So, what should have been a 15 minute trip turned into more than an hour. Ugh.
