Surviving a highly technical interview

Technical interviews can often materialize as a game of Battleship. Each side, lobbing over random coordinates of pop-quiz style knowledge tests looking for a “hit”. Will your armada sink in the next interview?

Board game metaphors aside, a well-done technical interview can be highly insightful into the technical skill of a candidate, where resumes and chit chat may not reveal the true nature of someone’s ability to contribute to teams expecting each person to perform highly technical tasks. Conversely, a poorly done interview only succeeds in demoralizing possibly good candidates, with those who pass only showing that they know what you *think* is important – a risky proposition that threatens to create an environment with low diversity of thought and skill.

So what does a good technical interview process look like?

  • Explain Everything
    • A candidate is in an unfamiliar place with people they do not know, evaluating them in all manners. Go out of your way to explain who you are, the interview process, what is expected, where the bathrooms are, who will come next to talk, introduce them, so on. If a candidate is too nervous from trivial things, they will not be able to give you good readings on the things you care about. Help them out a bit.
  • Measure Strengths
    • A good technical interview should only explore a candidate’s claimed strengths. This is so important. Do not waste valuable time making candidates feel small in things they already said they don’t know well. Move on.
  • Let them talk
    • Avoid standardized testing style questions with exactly one answer. Instead, use questions which let candidates answer in their own words, encouraging them to volunteer their level of comfort in the subject.
    • Positive example: Explain polymorphism to me in your own words.
    • Negative example: If I want to define a class that cannot be instantiated, but define some of the behavior for subclasses, what keyword goes in the class declaration?
  • Enough sorts
    • Do candidates really need to write sorting routines at your company? Can they just use one that was already written? If so, why are you testing them on writing a sort? Instead, prefer a trivial sample to prove they can code, or instead offer them to provide some examples of advanced algorithms they have written outside the interview. Remembering how to quick sort with someone standing over your shoulder should not be a gauge of anything.
    • Instead of asking for route memorization of algorithms, consider replacing these segments with tests that identify that candidates know which algorithms are a good fit for what problems. It is likely this is more important to you.
  • Know it too
    • When questioning candidates about technical subjects, it is important that the interviewer also discretely know the subjects being investigated. If this expertise does not exist at your company, solicit a peer or trusted third party to help you.
  • Be respectful
    • Even if candidates do not make it through your process, make sure they are treated well. If your hiring pipeline has gaps, give the candidate your personal contact information so you can follow up with getting them the outcome of their interview. Press your coworkers to decide quickly about each candidate so they can get the news of what comes next (if anything). You would appreciate the same treatment.

Given these guidelines, most folks should be able to cobble together a reasonable process to evaluate candidates.

How do candidates maintain their composure during this sort of intense question session? A few tips have helped me in the past.

  • Get basic context
    • What job role are you applying for? Is this question abnormal for the role? Ask why this question applies to your role if you do not think it does. It may be a sign you will be doing a different job than you thought.
  • Remember your value
    • The reason most people get hired is because they offer a value greater than the resources they consume (or so it seems). If you do not know the answer, follow up with how you can likely get the answer on your own, or how you would identify who to ask for help. If it looks like this sort of problem would leave you stuck, it is a sign your value generation would stop here.
  • Talk through it
    • There is no problem with thinking a bit before you answer questions, but preface those awkward pauses with a simple statement, “Let me think about that for a bit.” Letting the interviewer know you are thinking, or what you are thinking as often as you are able prevents you from looking like your brain just shutdown.
  • Ask questions
    • Despite being the interviewee, ask questions of your own. Failure to get a single job does not mean you should get nothing from the process. Clarify every question you did not know the answer to, solicit feedback on your interview, ask how the company solves problems, about their culture, about your manager, teammates, break room, everything. At the end of the interview, ask how long it will take to receive an update on the process.
  • Follow up
    • Sometimes companies do their own Human Resources work, but are lousy at it. Help them out by following up immediately after your interview, thanking them for the opportunity to meet. Include a friendly reminder that you look forward to hearing about the result of your interview. If no such contact is made for 3-4 business days, send a short note asking if a decision has been made.
  • Get contacts
    • Networking is a huge value during the job search process. Be sure to continue expanding your network as you interview.
  • Relax
    • If it does not work out, keep trying. If you continue to improve yourself, apply yourself with careful thought towards advancing to the next step, you will achieve your goals.

Hopefully the next time you participate in a technical interview, one or more of these points will help you.

Agree or disagree, drop me a comment.

Dear Recruiters, I cannot hear you. You do not speak my language.

Dear Recruiters,

I cannot hear you. You do not speak my language.

TL;DR; – why I spambox your messages

A while ago, I learned that many business folks speak a different language from me as an engineer. When talking to them about cool technologies, algorithms, and making a legacy built on software, they are trying to figure out how to map that to their world – revenue and expenses. Without talking in their terms, I was losing them in mere seconds.

I bring this up, because it seems most recruiters have an upside down view of how I seek a new full time opportunity.

“Urgent opportunity!”, “Software engineer position in Nowhere, ID”, and “great comprehensive packages” are not what I am looking for.

If you want to speak my language, tell me:

  • How does this role matter?
  • What kind of people will I work with?
  • Does management have a clear vision of the kind of future they want to pursue?
  • Does the business have sufficient resources to reasonably tackle these problems?
  • How far along in the process are they?
  • What challenges still remain?

In short, if I work on achieving someone else’s dreams, it must also fit my own. My dream is that I change the world for the better in meaningful ways for as many people as possible. I specifically want to reduce their suffering and increase their quality of life.

I end with:

“How is your message helping me achieve my dreams?”

John Goodwin

Answering this question cut 30 lbs of body fat in 4 months

Recently, I completed the Ironman NC 2016 (Oct 2016). The race would have been a full 140.6, but was cut short by 56 miles due to flooding in the area. A race image can be seen nearby (185lbs). Next to it, you can see me 4 months later, nearly 30lbs less body fat (155lbs).

Since 2010, I have been doing triathlons from sprint distances up until Ironman NC 2016 for the longest distance to date. Along the way, I expected that I would be not only healthier, but look healthier. Sadly, this did not evidence true. Year after year, my training left me hovering between 180-185lbs with rare exceptions. I felt like I carried a winter coat everywhere I went.

As I completed the Ironman NC, I found myself getting injured during the run – typical of all my long distance runs. This was disheartening, since it meant I was going to run/walk and finish after dark. As an added bonus, I get more hunger and more time to ponder what I have been doing wrong.

Post race, I decided I had enough of simply increasing my threshold for distance, but press my pace – aggressively. Settling on a 3 hour marathon pace as my long term goal, I knew this could not be done by simply improving all my bad habits. I needed to get serious about ditching any detractors. This is when I asked the question.

“How is what I eat accelerating achievement of my goals?”

It was that simple. Once I realized that many of my biggest body fat issues were not fixable by out working the other side of the equation (exercise), the fat literally melted off. Within 8 weeks, I had lost 25 lbs.

I cut out:

  • Added sugar
  • Added sugar alternatives
  • Preservatives
  • Most processed carbs (bread/pasta)
  • Oversized meat portions
  • Meals > 500 cals
  • Most GMO/non-organic foods

The simple version of my diet is now:

  • When waking up
    • Unflavored, grass fed whey protein concentrate or isolate mixed with water, or half serving with other half Orgain chocolate protein powder
  • Post workout
    • repeat shake/water mix
  • Lunch
    • 500 or so cals of organic brown rice, organic black beans, chicken breast, and vegetable
  • Second lunch 3 hrs later
    • repeat lunch
  • Dinner
    • repeat lunch
  • Snacks
    • When hungry, eat a small handful of nuts, or eat a single piece of fruit
    • Sometimes at night, eat unflavored greek yogurt with some sliced fruit, a few raisins, sliced almonds, or similar toppings.
  • Water
    • Drink lots
  • Before bed
    • repeat shake/water mix

My workouts

  • 1-2hrs / day of endurance or strength or mixed training 5 days/week
  • 4hrs long set 1 day a week

Some might say that with my level of exercise, it should be a no brainer to lose weight. To those people, I say, I tried that for years, without any real results. Maybe younger people can get away with that, but at 38 this year, I don’t think I will be pulling that off.

If you find these insights useful, leave a comment.

May you achieve your goals – every last one! 🙂

How I went from a lousy swimmer to passable

When training for my first triathlon, my swimming was so bad, it looked more like a distressed swimmer simulation. The harder I tried, the worse I got – until finally, a breakthrough.

When you watch some people swim, I think their brains naturally figure it out so easily, I don’t even think they know how to explain to someone who stinks *SO* bad.  You see, not only did I struggle with the stroke, form, flexibility, posture, breathing, head position, power delivery or anything else. I had the most critical issue of every problem anyone can ever have. I had no idea how to improve other than to just keep trying.

This piece is the key to making real and steady improvements.  When it comes to training in repetitive motion sports, it is important to build good habits early so that all that training can pay off.  Otherwise, it is like making minimum payments on a credit card.  You keep paying, but nothing actually gets paid off.

If I could go back, the first thing I would tell myself is, “Fix your breathing”. The microscopic gains that can be made without fixing breathing are just not worth grinding out. Face it.  No one wants drink half the pool by the end of the workout. (Consider yourself spared from all the puns I considered here.)

I have an unnaturally low sense of fear and danger. This is not to say I am a thrill seeker. It is more to say I did not breathe well, but did not worry I would drown.  That, however does not reduce the biological reaction to being out of breath. The urgency to breathe with the normal desire to look forward caused many young swimmers, including myself to swim more like dog paddling meets freestyle than just freestyle.

To fix this, I worked on drills designed only to increase my awareness of breathing, my comfort in breathing, environmental acuity in knowing where the top of the water was in relation to my mouth and related notions. Combined with this, I abandoned the preconceived notion that I would swim perfectly according to some image in my head and just breathe on the same side if it evened out my effort and rhythm.

Much of this advice came by way of various coaches who have instructed me until it finally sunk in.  Pat Rohner and Kendal Smith were among those who kept urging me to get better with small targeted instructions.

I still struggle to swim well, but breaking through these terrible blockers:

  • Not understanding what I am trying to do
  • Not knowing what to improve
  • Trying to fix too much at once

Has allowed me to move into a phase of improvement that allows for more gains with less work.

So, for anyone working on breathing or past it, my advice is simply this.  Find a good coach to help you fix one thing at a time, while helping you to understand not just what motion to perform, but what the underlying motivations are driving that instruction.  Then, proceed to measure your wonderful gains.

Good luck to all the other struggling swimmers out there. See you on the other side of the buoys 🙂

Lessons from finishing last in my first sprint triathlon

In my first sprint triathlon (in 2010) I wanted to believe that I could do it like most things in my life – might not be the best, but would not be the worst. I could not have been more wrong.

For anyone who reads even the first chapters of “The 7 Habits of Highly Effective People” by Steven R. Covey knows that you cannot cheat a natural system. A computer programmer who does not actively participate in sports may not do that well “cramming” for his first race. That is exactly what happened and the results were predictable. Of all the men who finished, I finished dead last. Every grandmother and grandfather who finished all passed me. Even the ones riding cruisers rocking the streamers on the handlebars.

At this point, I faced two choices:

  • Acceptance
  • Change

In some ways, I still don’t entirely understand all the forces that make me do the things I do. Ego, soul seeking, or maybe just a rock head. I chose to make a change. Like most changes, the decision is far easier than the execution.

My wife asked me a few times why I would keep at this. Each struggle just somehow makes me more determined to win.

I found out:

  • I suck at swimming, biking and running
  • I have exercise induced asthma
  • Sitting is the new smoking
  • You cannot out exercise your diet
  • Failure to measure destroys steady gains
  • Finishing is about determination
  • Performance is your training plan

In the coming weeks, I hope to break down my discoveries into discrete lessons. In the process, I hope to better my own understanding as well as help those who may happen upon these posts.


Why the Apple Watch is Doomed to Fail

April 24th, 2015, the Apple Watch is scheduled to be released.  It will fail.  I’ll tell you why.

Initial Reaction

Initially, all the Apple fans will buy their obligitory new device to prove devotion.  Everyone else will still be trying to understand why they need an expensive watch that’s even more delicate than a normal one.  Everyone else will still be trying to figure out what that watch does for them that their iPhone doesn’t already do.

Deal Breakers

Battery life

When someone traditionally thinks about wearing a watch, they want something reliable to tell time.  If you look at the Citizen Eco watches, it helps keep the watch going without winding.  Even without this, a traditional watch with watch battery is going to last years.  One that uses a side winding many people do once a day, but it just takes a second and can be done anywhere.  Automatic winding watches can be worn daily without much winding just due to the normal motion of your arms.  The Apple Watch by contrast will have a rated 18 hour battery life.  Post release, we will find out if that’s screen on time or not. In addition, Apple watch is charged using yet another special USB cable.


Another reason people often wear a watch is that some of them are downright beatiful works of art.  Engineering wise Apple Watch is a work of art, fashion wise, not so much.


Considering the cost of the watch (something like $1k), you expect this thing to do something really neat that has never been done.  The truth is, except for the fitness applications, I can’t really see the value in any of the other features to make people want yet another thing to charge, another thing to carry, and another thing to spend their hard earned money on.


In short, I think the Apple Watch is solving an aggregation problem that doesn’t exist, will fail to do it at a valuable price point, and will fail to do it well enough.  The only possible benefit I can think of is for people who regret buying their iPhone 6+, so they want a small device again.

Agree? Disagree?  Consider posting a tactful comment.

Google Glass Review

I had a chance to get a Google Glass. Being the sucker for trying new technology that I am, I gave it a shot. Here’s what I learned.


  • Navigation app gives simple instructions, with high contrast drawings of the road.
  • Taking pictures is dirt easy
  • Taking videos is dirt easy
  • Touch pad on the side gives an alternative for constantly talking to the phone.


  • Battery life sucks – “one day of use”.
    • Realistically a few hours of intensive use, and then it’s over.  For example, I drove from Myrtle Beach, SC back to Raleigh, NC using it for GPS and had to switch back to my phone midway.
    • Related to this, the device is inoperable while charging.  This means it would not be possible to rig a shirt pocket battery with micro usb extension to it.
    • I put a 9300mah battery on my phone for a reason.  5 days of normal use seems a lot more usable than one typical day.  In the example above, I used my phone for gps going to Myrtle Beach using the phone at full bright the entire time.  It drained about a third.  Glass turns off its prism after a few seconds.  It still only went maybe 2/3 of the way before it was saying it had low battery.
  • Asymmetrical design isn’t really for me.
    • Partial fix for this and above would be to include a battery on both sides instead of just one.
  • Huge thing sticking out of the top right of your head
    • Fix for this would be to pull back the prism and entire unit going back towards the ear.  Then directly put the prism into the side of the lens, getting rid of the glass-less glass.
  • Had to get contacts – not compatible with my existing glasses.
    • I actually wear glasses.  Using google glasses with my glasses was a no go since the camera area collides with the prescription frames.  You also end up with two nose pieces.
    • As a side note, prescription glasses for this thing will probably be crazy expensive.
    • Even with contacts, I only got the tinted non-prescription lenses free.  The clear ones were another $75.  Very expensive for clear glass or rather plastic.
  • Common workflows needs help.
    • Take a picture with glass, autobackup will put it on your google account at some random time.  Generally soon if you are on wifi.  Go to google photos and deal with them.  Your phone will have a notification about the photos.  You’ll also still have a copy of the photos on the google glass, which appear if you plug them into your computer which wants to import them like a camera.
    • There needs to be a lot more work on single instancing the data across devices and work to process it.
    • Notifications make a slight buzz on the bone conductor thing.  It will make a slight sound and end up tickling your ear.
    • I am a bit of a mindshare snob.  I don’t allow emails to chime when they come in.  IMs don’t beep, bloop, ding, chirp, bounce, wave, flash, or jump to focus.  I send people to voicemail if I don’t know them.  My concentration is important.  I can’t be randomly disturbed because an email from some random person came in.  If Google provides a way to reverse this, only notifying you with things you whitelist, then I would be much happier.  Until then, just give me a way to turn all of that off.
  • Google Glass is outside the glasses.
    • Tinted glasses will dim your Glass Screen.
  • Mono ear piece kept falling out of my ear.  Completely unusable.
  • Not really a fan of the visible screws (to someone talking to you) right between the eyes.
  • No easy way to “put it away”.
    • Device is flexible not collapsible.  If you go to do something with it, besides wear it, the only thing you can do is put it above your eyes like a woman’s hair hoop, but less secure.
  • You’re encouraged to talk to it.
    • No offense Google, but I’m not talking to my phone/glasses in public.
  • Head Position sensors
    • You can wake glass by tilting your head up.  Really?!  Can I have some lawsuit with that strained neck?  Besides being silly, this just seems like a bad idea.
  • Navigation app doesn’t do adhoc changes.
    • I was driving somewhere.  My wife asks me if we can stop at the next rest stop on our route.  Asking Glass to help out greeted me with the option to look at the overhead map or stop the route.
  • Starting up gives no feedback for several seconds.
    • If you push the button to turn it on, then put it on your head, the delay is long enough, you end up taking them off to verify that it is indeed booting/starting.  You can tell by looking at the back of the power button.  By then, you’ll see something is finally shining in the eyepiece.
  • Setup oddly confusing.
    • Maybe the setup was simple for the guy who invented it, but as a professional computer programmer for 16 years for everything from C64 basic to .NET apps, network apps, server apps, DB apps, and mobile apps, I couldn’t easily figure out what they wanted to setup the device.
    • Glass App seems confusing too.  I would expect to be able to explore the device more, but it seems like a stub service application with just a few UI bits on it.  Some things I would have expected to be able to do:  Browse the files, view logs, get battery level, projected run time, and so on.

Other Interesting Bits

  • Social Acceptance
    • I thought I was going to have more problems with people wearing it than I had.  I think that was helped by the fact that I used the tinted glasses rather than go without a lens, am a fairly big guy, and wasn’t obnoxious about wearing glass.
    • I actually had more people ask me about my Galaxy Note II (with extended battery) inside an arm band case on my arm.  I don’t know if they were really curious, or just trying to tell me I had a phablet on my arm 😉
    • Only time will tell if it will end up outlawed everywhere.  If it is, will it be outlawed if it is off?
  • Not ready for prime time
    • If Google is just happy to get early adopters when they go full release, then they will probably get it.
    • If Google wants general consumption.  I think it’s not ready for that.  Just too many important things in the Con list.
  • Limited use
    • While this device seems like it should have a lot of possibilities, I’m struggling to identify the practical applications.  I’m sure as more applications come out, people smarter than me will hopefully figure out how to do non-perverted, non-orwellian, and life enriching applications.  If I had to guess, I’d say that camera/gps based recognition of objects/places will end up being the top practical feature.  Ease of photos probably the best impulse recreation use.  If they can get the battery life up, probably navigation will add into the practical uses.

Understanding What Is Happening to the PC Market

There have been a lot of claims about the rise of tablet sales, the decline of PC sales spelling the death of the PC. In this article, I’ll not only explain why that’s exaggerated, but also what is actually happening.

The Core Markets Are The Same

If you follow the sales of PCs, you will see how the early units were by today’s standards, very expensive in the buying power at the time they were released, while the capabilities were somewhat basic. Work, write a letter/essay, or play games.  Not really much else.

So, who bought them?  People who believed that what it did was worth what it cost.

Those were people who:

  • Believed in new technology
  • Needed it for work
  • Tinkering types who could come up with the money
  • Rich people

For clarity, in the rest of the article, I’ll refer to the list above as the Professional market and anyone else as the Consumer market.

As the price of computers fell, the format available to the Consumer market primarily were desktops and laptops.  Once the internet started to catch on, people started using it to explore and communicate with others digitally.  The Consumer market was interested generally with the cheapest machines which fulfilled their desire to:

  • Browse the web
  • Email
  • Play games
  • Consume media

The only machines available to fill that need were desktops and laptops.  Desktops for a long time were just cheaper and more reliable than laptops.

At the point that laptops really started to catch up in delivering a comparable value to desktops, shortly after, the tablet market entered.  The commoditization of cheap computers made by Apple and all the Android based vendors allowed for computers which really did everything that the consumer wanted.

As a consequence, consumers did the most natural thing.  Bought the devices which fit their needs.

Why Not Tablets For All?

In a Professional market, the need for better computers, more reliable computers, faster data entry, customization, bigger screens, exploration of technology will continually drive a path for a niche of devices not necessarily the cheapest thing that gets you on the internet and plays games.

In A Nutshell

Professionals will still need what they originally needed and keep buying it.  Consumers only needed what they needed and will keep buying appropriate devices.  PC-like devices will continue to sell as long as people continue to need them.

Understanding why Windows RT failed

Let’s face it. Windows RT failed.

Rather than hide our heads in the sand, let’s understand why.

It might seem easy to dogpile on Microsoft regarding their RT project, but there were some key indicators to not only allow us to see retrospectively why it failed, but would have allowed Microsoft to see why it was at enormous risk before it even started the project.

A short history lesson on Microsoft Hardware

Microsoft has an expensive and dismal record when it comes to hardware.

Anyone remember Microsoft Phone?  Not talking cell phones here.  Before that they had a cordless phone.  Still don’t remember?  For good reason.  It died just after they purchased it.

Next up is XBox.  Have they ever made money?  If you roll them up into the entire entertainment section of Microsoft, then maybe it looks good, but just the Xbox portion isn’t too good.

Microsoft Kin.  Maybe you were one of the 503 Kin owners Verizon managed to sell?  This device flopped so hard, the apathy created a singularity somewhere in the universe.

Microsoft Surface RT and to a lesser degree Surface Pro tablets have failed to capture our imaginations.  You can tell Microsoft is desperate, because every commercial compares them to someone else better received.

So, why did all these fail?

In the end, I believe it comes down to several factors:

  • At it’s core, Microsoft innovates software platforms.
    Look at every success Microsoft has had at both capturing the hearts and imaginations of their audience.  They did it by building software platforms that other people were expected to extend.
  • Microsoft has lost their vision
    Where are they going with each one of their hardware experiments?  I think they failed to deliver on something amazing because they had no idea where they are going with this.  Take the last product.  Surface RT and Pro.  Did they really believe people wouldn’t be confused?  RT looks cheaper, until you realize, it won’t run any of your regular windows apps, and oh, by the way software developers, it only has a subset of the programming platform we’ve been selling you on using for over a decade.  Have fun!
  • Microsoft spent too long staring at the backs of their competitors.
    Microsoft will never be able to get their vision back as long as they let their view fill up with the the ground that’s already been paved by competition.
  • Decisions need to become less political.
    It’s very clear (at least to me) that too many projects are taken to completion at Microsoft, only because someone can’t affort to lose face.  Some projects fail.  Some ideas are bad.  Let’s just accept that.  Once we have, we can start making decisions to stop pursuing bad ones earlier.

What can they do about it, now?

Microsoft needs to get back to their roots.

Instead of making XBoxes, they should be making XNA frameworks, and encouraging hardware vendors to make the hardware (or one specific one).  Instead of making Surface RT, they needed to build up their platforms which allow developers not only to target battery powered devices with touch screens, but do it in a way where their applications behave, perform as expected, and easily. Instead of looking for the next, “Me too!” project that competitors have already hit, and hit well, they need to think about where they can be the next leap ahead – where they can win back the confidence of their partners. Microsoft needs more guys in basements to believe that they can innovate with their platforms again.

Last, but not least, Microsoft needs to either replace Steve Ballmer, or he needs to have a big change of heart.  Steve shouldn’t be afraid to surround himself with men greater than himself.  He should embrace the need for greatness in his inner circle.

If you agree or disagree with this article, let me know.

Reducing a 3 to 6 hour SSIS job to 2 minutes

In computer programming, the OCD inside many computer programmers wants to write the most efficient program possible in order to accomplish the task at hand.  This of course has to be balanced with time to completion, understanding of requirements, complexity, and generally how many other tasks are also waiting to be completed.

Sometimes, you make due with “get ‘er done” mainly because the system completes the job in an acceptable amount of time.  This can be especially true of a scheduled job – like SSIS tasks.

In our specific example, we had an SSIS task which would take anywhere from 3 to 6 hours to complete, and was set to run hourly.   Obviously a job like this cannot safely run hourly.  Luckily the SQL Agent scheduler doesn’t overlap the job executions and instead skips any repeats.

Sample Job Flow (before):

  1. Download remote data
  2. Extract source data and place into temp database
  3. Transform data
  4. Load data into target database

This might sound pretty normal, but the problem was that the remote data came as the entire dataset every time.  Steps 1-3 didn’t take too long because of this, but step 4 took a *long* time because of remoting overhead.  Also, the check for duplicate data was happening at the target database instead of during our SSIS job.

To fix this, I came up with the following idea:

  1. Save transformed data with the addition of 3 new dates: ChangeDate, InsertDate, UpdateDate
  2. Compare subsequent job runs with the saved data.  Then, we only use the delta for step 4 Load data into target database

To implement this magic, I use the T-SQL command EXCEPT, which works similar to a UNION, but instead of appending, it subtracts any exact matches from the first dataset, ignoring any data only existing in the second dataset.

You now have a dataset which is only the changed, or inserted records.  For my own benefit, I actually created a temp table to hold this dataset, so I could handle the new and changed records separately.

SELECT [fields] FROM [New Entire Transformed Dataset]
SELECT [fields] FROM [Saved Transformed Dataset]

Now, to check which ones are new:

WHERE [primarykey] NOT IN (SELECT [primarykey] FROM [Saved Transformed Dataset])

To see which ones are updated, it is just the opposite:

WHERE [primarykey] IN (SELECT [primarykey] FROM [Saved Transformed Dataset])

Using this, I was able to make much smaller change sets for step 4 Load data into target database.  This change set ended up being just a few dozen records normally, instead of tens of thousands.

The inserts, I add into the [Saved Transformed Dataset] with no InsertDate, and the updates, I set each field equal to the new field, and ChangeDate = GetDate()

Now, after each step 4 Load data into target database was completed, I ran one more statement to mark the inserted items as inserted, and updated as updated.

For Inserts:

UPDATE [Saved Transformed Dataset]
SET InsertDate = GetDate()

For Updates:

UPDATE [Saved Transformed Dataset]
SET UpdateDate = GetDate()
WHERE (UpdateDate IS NULL AND InsertDate < ChangeDate) OR (UpdateDate < ChangeDate)

The job execution time was brought down 99%.  Success!